Into the Depths: The Technical Details Behind AV1

AV1, the next generation royalty-free video codec from the Alliance for Open Media, is making waves in the broadcasting industry.

Firefox, Coney & Mozilla's AV1 team at IBC 2018

Since AOMedia officially cemented the AV1 v1.0.0 specification earlier this year, we’ve seen increasing interest from the broadcasting industry. Starting with the NAB Show (National Association of Broadcasters) in Las Vegas earlier this year, and gaining momentum through IBC (International Broadcasting Convention) in Amsterdam, and more recently the NAB East Show in New York, AV1 keeps picking up steam. Each of these industry events attract over 100,000 media professionals. Mozilla attended these shows to demonstrate AV1 playback in Firefox, and showed that AV1 is well on its way to being broadly adopted in web browsers.

Continuing to advocate for AV1 in the broadcast space, Nathan Egge from Mozilla dives into the depths of AV1 at the Mile High Video Workshop in Denver, sponsored by Comcast.

AV1 leapfrogs the performance of VP9 and HEVC, making it a next-generation codec. The AV1 format is and will always be royalty-free with a permissive FOSS license.

The post Into the Depths: The Technical Details Behind AV1 appeared first on Mozilla Hacks – the Web developer blog.

Original Link

The Video Wars of 2027

Author’s Note: This post imagines a dystopian future for web video, if we continue to rely on patented codecs to transmit media files. What if one company had a perpetual monopoly on those patents? How could it limit our access to media and culture? The premise of this cautionary tale is grounded in fact. However, the future scenario is fiction, and the entities and events portrayed are not intended to represent real people, companies, or events.

Illustration by James Dybvig

The year is 2029. It’s been two years since the start of the Video Wars, and there’s no end in sight. It’s hard to believe how deranged things have become on earth. People are going crazy because they can’t afford web video fees – and there’s not much else to do. The world’s media giants have irrevocably twisted laws and governments to protect their incredibly lucrative franchise: the right to own their intellectual property for all time.

It all started decades ago, with an arcane compression technology and a cartoon mouse. As if we needed any more proof that truth is stranger than fiction.

Adulteration of the U.S. Legal System

In 1998, the U.S. Congress passed the Sonny Bono Copyright Term Extension Act. This new law extended copyrights on corporate works to the author’s lifetime plus 95 years. The effort was driven by the Walt Disney Company, to protect its lucrative retail franchise around the animated character Mickey Mouse. Without this extension, Mickey would have entered the public domain, meaning anyone could create new cartoons and merchandise without fear of being sued by Disney. When the extension passed, it gave Disney another 20 years to profit from Mickey. The news sparked outrage from lawyers and academics at the time, but it was a dull and complex topic that most people didn’t understand or care about.

In 2020, Disney again lobbied to extend the law, so its copyright would last for 10,000 years. Its monopoly on our culture was complete. No art, music, video, or story would pass into the public domain for millennia. All copyrighted ideas would remain the private property of corporations. The quiet strangulation of our collective creativity had begun.

A small but powerful corporate collective called MalCorp took note of Disney’s success. Backed by deep-pocketed investors, MalCorp had quietly started buying the technology patents that made video streaming work over the internet. It revealed itself in 2021 as a protector of innovation. But its true goal was to create a monopoly on video streaming technology that would last forever, to shunt profits to its already wealthy investors. It was purely an instrument of greed.

Better Compression for Free

Now, there were some good guys in this story. As early as 2007, prescient tech companies wanted the web platform to remain free and open to all – especially for video. Companies like Cisco, Mozilla, Google, and others worked on new video codecs that could replace the patented, ubiquitous H.264 codec. They even combined their efforts in 2015 to create a royalty-free codec called AV1 that anyone could use free of charge.

AV1 was notable in that it offered better compression, and therefore better video quality, than any other codec of its time. But just as the free contender was getting off the ground, the video streaming industry was thrown into turmoil. Browser companies backed different codecs, and the market fragmented. Adoption stalled, and for years the streaming industry continued paying licensing fees for subpar codecs, even though better options were available.

The End of Shared Innovation

Meanwhile MalCorp found a way to tweak the law so its patents would never expire. It proposed a special amendment, just for patent pools, that said: Any time any part of any patent changes, the entire pool is treated as a new invention under U.S. law. With its deep pockets, MalCorp was able to buy the votes needed to get its law passed.

MalCorp’s patents would not expire. Not in 20 years. Not ever. And because patent law is about as interesting as copyright law, few protested the change.

Things went downhill quickly for advocates of the open web. MalCorp’s patents became broader, vaguer, ever-changing. With billions in its war chest, MalCorp was able to sue royalty-free codecs like AV1 out of existence. MalCorp had won. It had a monopoly on web streaming technology. It began, slowly at first, to raise licensing fees.

Gorgeous Video, Crushing Fees

For those who could afford it, web video got much better. MalCorp’s newest high-efficiency video codecs brought pixel-perfect 32K-Strato-Def images and 3D sound into people’s homes. Video and audio were clear and rich – better than real life. Downloads were fast. Images were crisp and spectacular. Fees were high.

Without access to any competing technologies, streaming companies had to pay billions instead of millions a year to MalCorp. Streaming services had to 100x their prices to cover their costs. Monthly fees rose to $4,500. Even students had to pay $50 a minute to watch a lecture on YouTube. Gradually, the world began to wake up to what MalCorp had done.

Life Indoors

By the mid-twenties, the Robotic Age had put most people out of work. The lucky ones lived on fixed incomes, paid by their governments. Humans were only needed for specialized service jobs, like nursery school teachers and style consultants. Even doctors were automated, using up-to-the-minute, crowd-sourced data to diagnose disease and track trends and outbreaks.

People were idle. Discontent was rising. Where once a retired workforce might have traveled or pursued hobbies, growing environmental problems rendered the outside world mostly uninhabitable. People hiked at home with their headsets on, enjoying stereoscopic birdsong and the idea of a fresh breeze. We lived indoors, in front of screens.

Locked In, Locked Out

It didn’t take long for MalCorp to become the most powerful corporation in the world. When video and mixed reality files made up 90 percent of all internet traffic, MalCorp was collecting on every transmission. Still, its greed kept growing.

Fed up with workarounds like piracy sites and peer-to-peer networks, MalCorp dismantled all legacy codecs. The slow, furry, lousy videos that were vaguely affordable ceased to function on modern networks and devices. People noticed when the signal went dark. Sure, there was still television and solid state media, but it wasn’t the same. Soon enough, all hell broke loose.

The Wars Begin

During Super Bowl LXII, football fans firebombed police stations in 70 cities, because listening to the game on radio just didn’t cut it. Thousands died in the riots and, later, in the crackdowns. Protesters picketed Disneyland, because the people had finally figured out what had happened to their democracy, and how it got started.

For the first time in years, people began to organize. They joined chat rooms and formed political parties like VidPeace and YouStream, vying for a majority. They had one demand: Give us back free video on the open web. They put banners on their vid-free Facebook feeds, advocating for the liberation of web video from greedy patent holders. They rallied around an inalienable right, once taken for granted, to be able to make and watch and share their own family movies, without paying MalCorp’s fees.

But it was too late. The opportunity to influence the chain of events had ended years before. Some say the tipping point was in 2019. Others blame the apathy and naiveté of early web users, who assumed tech companies and governments would always make decisions that served the common good. That capitalism would deliver the best services, in spite of powerful profit motives. And that the internet would always be free.

Original Link

DASH playback of AV1 video in Firefox

Bitmovin and Mozilla partner to enable HTML5 AV1 Playback

Bitmovin and Mozilla, both members of the Alliance for Open Media (AOM), are partnering to bring AV1 playback with HTML5 to Firefox as the first browser to play AV1 MPEG-DASH/HLS streams. While the AV1 bitstream is still being finalized, the industry is gearing for fast adoption of the new codec, which promises to be 25-35% more efficient than VP9 and H.265/HEVC.

The AV1 bitstream is set to be finalized by the end of 2017. You may ask – “How does playback work on the bitstream that is not yet finalized?”. Indeed, this is a good question as there are still many things in the bitstream that may change during the current state of the development. However, to make playback possible, we just need to ensure that the encoder and decoder use the same version of the bitstream. Bitmovin and Mozilla agreed on a simple, but for the time being useful, codec string, to ensure compatibility between the version of the bitstream in the Bitmovin AV1 encoder and the AV1 decoder in Mozilla Firefox:

 "av1.experimental.<git hash>"

A test page has been prepared to demonstrate playback of MPEG-DASH test assets encoded in AV1 by the Bitmovin Encoder and played with the Bitmovin HTML5 Player (7.3.0-b7) in the Firefox Nightly browser.

playback demo screenshotAV1 DASH playback demo by Bitmovin and Firefox Nightly. Short film “Tears of Steel” cc-by Blender Foundation.

Visit the demo page at You can download Firefox Nightly here to view it.

Bitmovin AV1 End-to-End

The Bitmovin AV1 encoder is based on the AOM specification and scaled on Bitmovin’s cloud native architecture for faster throughput. Earlier this year, the team wrote about  the world’s first AV1 livestream at broadcast quality, which was demoed during NAB 2017 and brought the company the Best of NAB 2017 Award from Streaming Media.

The current state of the AV1 encoder is still far away from delivering reasonable encoding times without extensive tuning to the code base: e.g. it takes about 150 seconds on an off-the-shelf desktop computer to encode one second of video. For this reason, Bitmovin’s ability to provide complete ABR test assets (multiple qualities and resolutions) of high quality in reasonable times was extremely useful for testing of the MPEG-DASH/HLS playback of AV1 in Firefox. (HLS playback of AV1 is not officially supported by Apple, but technically possible of course.) The fast encoding throughput can be achieved thanks to Bitmovin’s flexible cloud native architecture, which allows massive horizontal scaling of a single VoD asset to multiple nodes, as depicted in the following figure. An additional benefit of the scalable architecture is that quality doesn’t need to be compromised for speed, as is often the case with a typical encoding setup.

block diagram of the encoderBitmovin’s scalable video encoder.

The test assets provided by Bitmovin are segmented WebM outputs that can be used with HLS and MPEG-DASH. For the demo page, we decided to go with MPEG-DASH and encode the assets to the following quality levels:

  • 100 kbps, 480×200
  • 200 kbps, 640×266
  • 500 kbps, 1280×532
  • 800 kbps, 1280×532
  • 1 Mbps, 1920×800
  • 2 Mbps, 1920×800
  • 3 Mbps, 1920×800

We used the royalty-free Opus audio codec and encoded with 32 kbps, which provides for a reasonable quality audio stream.

Mozilla Firefox

Firefox has a long history of pioneering open compression technology for audio and video. We added support for the royalty-free Theora video codec a decade ago in our initial implementation of HTML5 video. WebM support followed a few years later. More recently, we were the first browser to support VP9, Opus, and FLAC in the popular MP4 container.

After the success of the Opus audio codec, our research arm has been investing heavily in a next-generation royalty-free video codec. Mozilla’s Daala project has been a test bed for new ideas, approaching video compression in a totally new way. And we’ve been contributing those ideas to the AV1 codec at the IETF and the Alliance for Open Media.

AV1 is a new video compression standard, developed by many contributors through the IETF standards process. This kind of collaboration was part of what made Opus so successful, with contributions from several organizations and open engineering discussions producing a design that was better than the sum of its parts.

While Opus was adopted as a mandatory format for the WebRTC wire protocol, we don’t have a similar mandate for a video codec. Both the royalty-free VP8 and the non-free H.264 codecs are considered part of the baseline. Consensus was blocked on the one side by the desire for a freely-implementable spec and on the other for hardware-supported video compression, which VP8 didn’t have at the time.

Major hardware vendors have been involved with AV1 from the start, which we expect will result in accelerated support being available much sooner.

In April, Bitmovin demonstrated the first live stream using the new AV1 compression technology.

In June, Bitmovin and Mozilla worked together to demonstrate the first playback of AV1 video in a web page, using Bitmovin’s adaptive bitrate video technology. The demo is available now and works with Firefox Nightly.

The codec work is open source. If you’re interested in testing this, you can compile an encoder yourself. The format is still under development, so it’s important to match the version you’re testing with the decoder version in Firefox Nightly. We’ve extended the MediaSource.isTypeSupported api to take a git commit as a qualifier. You can test for this, e.g.:

var container = ‘video/webm’;
var codec = ‘av1.experimental.e87fb2378f01103d5d6e477a4ef6892dc714e614’;
var mimeType = container + ‘; codecs=”’ + codec + ‘“‘;
var supported = MediaSource.isTypeSupported(mimeType);

Then select an alternate resource or display an error if your encoded resource isn’t supported in that particular browser.

Past commit ids we’ve supported are aadbb0251996 and f5bdeac22930.The currently-supported commit id, built with default configure options, is available here. Once the bitstream is stable we will drop this convention and you can just test for codecs=av1 like any other format.

As an example, running this code inside the current page, we can report:

Since the initial demo, we’ve continued to develop AV1, providing feedback from real-world application testing and periodically updating the version we support to take advantage of ongoing improvements. The compression efficiency continues to improve. We hope to stabilize the new format next year and begin deployment across the internet of this exciting new format for video.

Ralph has contributed to media technology and royalty-free codecs for most of his career. Currently he helps maintain the video playback module in Firefox and supports new work in the Rust programming language. In his spare time he enjoys books and early music.

More articles by Ralph Giles…

Martin is responsible for Bitmovin Encoding product strategy, roadmap and development. His team works to enable complex video encoding workflows for global premium media and technology companies like Red Bull Media House and the New York Times. As one of the first employees, Martin led the development of Bitmovin encoding infrastructure, building the world’s first commercial massively scalable encoding service, capable of achieving 100x speeds over realtime. Currently, Martin oversees further development of the Bitmovin Encoding solution, including integration of new technologies, like AV1.

More articles by Martin Smole…

Original Link