Media stack Mid-Year review

We recently closed the first half of 2019 and with that it is time to look back and do a quick summary of what the media team has achieved during this 6 months period.

Looking at some stats, we merged 87 Pull Requests, we opened 56 issues, we closed 42 issues and we welcomed 13 new amazing contributors to the media stack.

A/V playback

These are some of the selected A/V playback related H1 acomplishments

Media cache and improved seeking

We significally improved the seeking experience of audio and video files by implementing preloading and buffering support and a media cache.

Basic media controls

After a few months of work we got partial support for the Shadow DOM API, which gave us the opportunity to implement our first basic set of media controls.

The UI is not perfect, among other things, because we still have no way to render a progress or volume bar properly, as that depends on the input type="range"> layout, which so far is rendered as a simple text box instead of the usual slider with a thumb.

GStreamer backend for MagicLeap

Another great achievement by Xavier Claessens from Collabora has been the GStreamer backend for Magic Leap. The work is not completely done yet, but as you can see on the animation below, he already managed to paint a full screen video on the Magic Leap device.

Hardware accelerated decoding

One of the most wanted features that we have been working on for almost a year and that has recently landed is hardware accelerated decoding.

Thanks to the excellent and constant work from the Igalian Víctor Jáquez, Servo recently gained support for hardware-accelerated media playback, which means lower CPU usage, better battery life and better thermal behaviour, among other goodies.

We only have support on Linux and Android (EGL and Wayland) so far. Support for other platforms is on the roadmap.

The numbers we are getting are already pretty nice. You might not be able to see it clearly on the video, but the renderer CPU time for the non hardware accelerated playback is ~8ms, compared to the ~1ms of CPU time that we get with the accelerated version.

Improved web compatibility of our media elements implementation

We also got a bunch of other smaller features that significantly improved the web compatibility of our media elements.

• ferjm added support for the HTMLMediaElement poster frame attribute
• swarnimarun implemented support for the HTMLMediaElement loop attribute
• jackxbritton implemented the HTMLMediaElement crossorigin attribute logic.
• Servo got the ability to mute and unmute as well as controlling the volume of audio and video playback thanks to stevesweetney and lucasfantacuci.
• sreeise implemented the AudioTrack, VideoTrack, AudioTrackList and VideoTrackList interfaces.
• georgeroman coded the required changes to allow changing the playback rate of audio and video files.
• georgeroman, again, implemented support for the HTMLMediaElement canPlayType function.
• dlrobertson paved the way for timed text tracks support by implementing the basics of the TextTrack API and the HTMLTrackElement interface.

WebAudio

We also got a few additions on the WebAudio land.

• PurpleHairEngineer implemented the StereoPannerNode backend.
• collares implemented the DOM side of the ChannelSplitterNode.
• Akhilesh1996 implemented the AudioParam setValueCurveAtTime function.
• snarasi6 implemented the deprecated setPosition and setOrientation AudioListener methods.

WebRTC

Thanks to jdm’s and Manishearth’s work, Servo has now the foundations of a WebRTC implementation and it is able to perform a 2-way calling with audio and video playback coming from the getUserMedia API.

Next steps

That’s not all folks! We have exciting plans for the second half of 2019.

A/V playback

On the A/V playback land, we want to:

• Focus on adding hardware accelerated playback on Windows and OSX.
• Add support for fullscreen playback.
• Add support for 360 video.
• Improve the existing media controls by, for instance, implementing a nicer layout for the <input type="range"> element, with a proper slider and a thumb, so we can have progress and volume bars.

WebAudio

For WebAudio there are plans to make some architectural improvements related to the timeline and the graph traversals.

We would also love to work on the MediaElementAudioSourceNode implementation.

WebRTC

For WebRTC, data channels are on the roadmap for the second half.

We currently support the playback of a single stream of audio and video simultaneously, so allowing the playback of multiple simulatenous streams of each type is also something that we would like to get during the following months.

Others

There were also plans to implement support for a global mute feature, and I am happy to say, that khodza already got this done right at the start of the second half.

Finally, we have been trying to get Youtube to work on Servo, but it turned out to be a difficult task because of non-media related issues (i.e. layout or web compatibility issues), so we decided to adjust the goal and focus on embedded Youtube support instead.