Container and coding formats

There are two main competing streaming container formats in WebM and MP4. Both have advantages and disadvantages directly tied to the nature of the containers as well as the coding formats they support.

WebM
Based on a subset of Matroska (MKV), specifically designed for internet streaming. Open source.

Has major advantages in terms of container metadata for streaming using Media Source Extensions.

This format has compatibility issues stemming from Apple's refusal to ship an open and royalty free codec in their products. Generally, Apple devices using their native browser engines cannot play WebM, and don't support VP8 or VP9 playback. The user must install a 3rd party browser (e.g. Firefox or Chrome) in order to play these files. Furthermore, Apple mobile devices (including iPad and iPhone) can't play these files back regardless of what browser they install on that device.

WebM supports these formats:


 * VP8, VP9, AV1 for video streams
 * Vorbis and Opus for audio streams

MPEG-4 Part 14 (MP4)
Older container based on the QuickTime file format. Supports more codecs than will be described here, as most of them are not suitable for streaming due to legal or technical limitations.

MP4 can still be used with MSE but each file needs to be specifically prepared to enable this, which boils down to remuxing the stream in question into a properly formatted container without any transcoding. There are no issues with traditional HTTP "streaming" and playback.

The main advantage of MP4 is its widespread support, as well as excellent support for some of the coding formats it can contain. Note that the browser/player must still support the codecs used in the container. For this reason, it is best to use the combination of H.264 for video and AAC for audio to maximize compatibility across all devices.

For modern purposes, MP4 supports these formats.


 * MPEG-2, MPEG-4 Part 2, AVC (H.264), HEVC (H.265) for video streams
 * MP3 and AAC for audio streams

Coding formats
Only coding formats relevant to streaming content will be listed here. Formats that aren't listed are either stillborn due to excessively greedy patenting issues (e.g. HEVC) or not suitable for internet streaming (e.g. MPEG-2).

All audio formats listed are lossy.

MPEG-3 (MP3)
MP3 is probably the most widely supported format for distributing music. However, it's generally worse than AAC for files of the same size. This means that an AAC file of the same bitrate will be higher quality than MP3, and an AAC file of the same quality will have a smaller file.

Don't use this.

Advanced Audio Coding (AAC)
AAC is currently the best available audio codec when using MP4 for streaming. It offers better compression and better quality than MP3, and has a similar compatibility spread. There are some web stores that distribute music as AAC instead of MP3 (e.g. Apple Music, Recochoku).

Note that the built in FFmpeg AAC encoder is quite poor. If encoding AAC, make sure to use either  (Apple's proprietary codec, bundled with their OS and iTunes) or   (non-free, requires you to build FFmpeg from source.)

Use this if your container is MP4.

Vorbis
Open-source format intended to compete with MP3 due to licensing fees. Not as well supported as MP3. Deprecated, superseded by Opus.

Don't use this.

Opus
Open-source format intended to compete with AAC. The reference implementation of the codec is the only one worth using. Has aforementioned compatibility issues with Apple products, but otherwise well supported.

Use this if your container is WebM.

Advanced Video Coding (AVC, H.264)
The most commonly used video format on the internet. Nearly all large streaming websites use this to distribute video content. Has nigh-universal support. Most devices have hardware decoding for AVC, resulting in lower processor usage and battery drain.

The successor to AVC, HEVC (H.265), is useless for streaming due to patenting issues. This means that AVC is essentially an older-generation codec currently used as a broadcast standard, leading to widespread adoption. AVC does not suffer from the same legal issues as HEVC as Cisco paid for licensing fees to develop an open source encoder for AVC, and serving AVC as unpaid content is also free to use under the MPEG patent. This means that Cisco foots the bill to allow the rest of the world to encode AVC.

Encoding in AVC is extremely fast even without hardware acceleration (which should not be used during encoding anyway), resulting in files that are generally 1.0-1.5x the size of the equivalent quality HEVC/VP9 file, but take 10-20x less time to encode.

Use this if your container is MP4.

VP8
Open-source format created by Google's acquisition of On2, intended to compete with AVC due to licensing fees in the past. Superseded by VP9.

Don't use this.

VP9
Open-source format developed by Google, intended to compete with HEVC. As HEVC is entirely worthless for streaming, this is the only option for video streaming of that generation. Free usage as long as you don't sue Google. Good support outside of Apple products.

Encode times and quality are comparable to HEVC.

Use this if your container is WebM.

AV1
Successor to VP9. As of 2022, extremely slow to encode on typical consumer hardware to the point where usage is discouraged for this project. Open and royalty-free. Poorly supported as of this writing.

These strikes against AV1 may change in the future. However, for the time being don't listen to AV1 shills. They're living in a fantasy world where everyone has ultra high-end hardware and nobody uses older software or Apple products.

Don't use this until the format matures.

Conclusions
For WebM, use VP9 and Opus. WebM formats are included in broadcast standards, i.e. not supported ubiquitously.

For MP4, use AVC and AAC. These formats are part of ATSC, ITU, and EBU broadcast standards, meaning almost all hardware supports them.