Building a Dash-264 Player
- 1. Building a Dash-264 Player Jeff Tapper / Michael Labriola Digital Primates jtapper@digitalprimates.net mlabriola@digitalprimates.net Twitter: @jtapper / @mlabriola
- 2. Jeff Tapper • Senior Consultant at Digital Primates – Building next generation client applications • Built video applications for clients including … • Developing Internet applications for 17 years • Author of 12 books on Internet technologies
- 3. Who am I? • Michael Labriola • Senior Consultant at Digital Primates – Building next generation client applications • Built video applications for clients including …well lots… • Developing Internet applications for 17 years • Author of 5 books on Internet technologies
- 4. Agenda • Video and the Internet today • Understanding HTTP Streaming • What is DASH • Flash as a video platform • The inner workings of DASH • The challenge • Getting video bytes info flash • Introducing the Digital Primates DASH-264 Player • Questions
- 5. Video is dominating the Internet • Desktop: Video makes up 50% of traffic at peak periods – notably 30% from Netflix and 11% from YouTube • Mobile: Video traffic is growing exponentially Fixed Internet Mobile Internet
- 6. Encode each segment HTTP Adaptive Streaming 1 at multiple bitrates Split the video into 2 small segments Client splices together 5 and plays back Make each segment 0010101000 01010 0101010100 01110 0111010001 1010101 3 addressable via a HTTP-URL 0010101000010 10 0101010100011 10 0111010001101 0101 0010101000 0101010100 0111010001 01010 01110 1010101 0010101000 0101010100 0111010001 01010 01110 1010101 Client makes decision Media Capture & Encoding 4 on which segment to download Media Origin Servers HTTP Cache Servers Client Devices
- 7. HTTP Streaming Landscape • Apple’s HTTP Live Streaming (HLS) • Microsoft’s Smooth Streaming • Adobe’s HTTP Dynamic Streaming (HDS) • And many more…
- 8. The challenge • Most agree that HTTP Streaming is the most efficient choice • Different devices support different streaming protocols • No one standard is currently supported ubiquitously • Results in media being served in several different formats to support the broadest range of devices
- 9. What is MPEG-DASH DASH – Dynamic Adaptive Streaming via HTTP International open standard, developed and published by ISO Addresses both simple and advanced use cases Enables highest-quality multiscreen distribution and efficient dynamic adaptive switching Enables reuse of existing content, devices and infrastructure Attempts to unify to a single standard for HTTP Streaming
- 10. DASH and codecs • The DASH specification is codec agnostic • Any existing or future codec can work with DASH • DASH manifest describes which codec is used – Different codecs store the actual video data differently
- 11. DASH264 • H.264 is dominant format today • Many vendors and service providers are committed to supporting/enabling DASH264 • Provides support for today’s requirements such as DRM • H.264 is backed by rigorous testing and conformance
- 12. DASH Industry Forum • Addressing the dramatic growth of broadband video by recommending a universal delivery format that provides end users with the best possible media experience by dynamically adapting to changing network conditions.
- 13. DASH Industry Forum • Objectives: – promote and catalyze market adoption of MPEG-DASH – publish interoperability and deployment guidelines – facilitate interoperability tests – collaborate with standard bodies and industry consortia in aligning ongoing DASH standards development and the use of common profiles across industry organizations • Currently 46 members • Visit http://dashif.org for more information
- 14. Members
- 15. Building a Desktop DASH player • There are many technologies available for playing video in modern web browsers – Flash – SilverLight – HTML5 – Etc. • We focused on Flash for our first DASH player – Largest percentage of users can support it
- 16. Building Video Players in Flash • Flash Player includes a native video player – flash.media.Video • Video accepts a source from a Camera or a NetStream – flash.media.Camera – flash.net.NetStream • Netstream supports: – Streaming FLV file – Manually appending FLV Bytes into its buffer
- 17. Flash Video (continued) • NetStream and Video classes can be manipulated manually • Open Source frameworks are available for easing the process – Adobe’s Open Source Media Framework (OSMF) – Open Video Player (OVP) • We have chosen to build on OSMF
- 18. OSMF • Extensible framework with plugin architecture • Native support for DRM • Contains classes to work with HTTP Streaming technologies – Only HDS is supported natively – Extensible architecture makes it possible to support new technologies
- 19. How to DASH in Flash • Extract Video Bytes from DASH segments • Wrap bytes in format understood by flash (FLV) • Append FLV Bytes into Netstream
- 20. Understanding DASH structure • Three types of files – Manifest (.mpd) • XML file describing the segments – Initialization file • Contains headers needed to decode bytes in segments – Segment Files • Contains playable media • Includes: – 0…many video tracks – 0…many audio tracks
- 21. DASH Manifest • Manifest contains: – Program Info (including title) – 1 or more periods • Periods contain 1 adaptation set per video stream and • Periods contain 1 adaptation set per audio stream • Adaptation Sets contain: – Content Composition nodes (for each video or audio track) – 1 or more Representation node » Each representation describes a single bitrate » Representations contain data on finding the actual segments » Different ways a representation can describe segments
- 22. Describing Representations • SegmentBase – Describes a stream with only a single Segment per bitrate – Can be used for Byte Range Requests • SegmentList – A SegmentList will contain a specific list of each SegmentURL (individual HTTP packet with media data) – Can be used for Byte Range Requests • SegmentTemplate – Defines a known url for the fragment with wildcards resolved at runtime to request a segments (see bbb.mpd) – Alternatively, can specify a list of segments based on duration
- 23. SegmentList <Representation id="h264bl_hd" mimeType="video/mp4" codecs="avc1.42c01f" width="1280" height="720" startWithSAP="1" bandwidth="514864"> <SegmentList timescale="1000" duration="10000"> <Initialization sourceURL="mp4-main-multi-h264bl_hd-.mp4"/> <SegmentURL media="mp4-main-multi-h264bl_hd-1.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-2.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-3.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-4.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-5.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-6.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-7.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-8.m4s"/>
- 24. SegmentTemplate fixed segment duration <AdaptationSet> <ContentComponent id="1" contentType="video"/> <SegmentTemplate initialization="BigBuckBunny_720p_1800kbps_44khz_track1_dash.mp4"/> <Representation id="1" mimeType="video/mp4“ codecs="avc1.64001f" width="1280" height="720“ startWithSAP="1" bandwidth="1809954"> <SegmentTemplate timescale="1000" duration="13809" media="bbb_seg_BigBuckBunny_720p_1800kbps_44khz_track1$Number$.m4s" startNumber="1"/> </Representation> </AdaptationSet>
- 25. SegmentTemplate variable segment duration <AdaptationSet group="2" mimeType="video/mp4" par="16:9“ minBandwidth="475000“ maxBandwidth="6589000" minWidth="176" maxWidth="1680" minHeight="99" maxHeight="944“ segmentAlignment="true“ startWithSAP="1"> <SegmentTemplate timescale="1000" initialization="dash/ateam-video=$Bandwidth$.dash" media="dash/ateam-video=$Bandwidth$-$Time$.dash"> <SegmentTimeline> <S t="0" d="4171" /> <S d="2503" /> <S d="2961" /> <S d="2461" /> <S d="2127" r="2" /> …
- 26. Understanding MP4 – H.264 structure • Each segment is binary, with a series of boxes contained within • First 8 bytes of every box defines – Box type – Number of bytes for the box.
- 27. Initialization Segment • FTYP – file type and compatibility • FREE – free space • MOOV – container of all metadata – MVHD – movie header – 1.* TRAK – container for individual stream data • TKHD – track header • MDIA – container for all media info – Lots of child boxes which contain more detailed stream info
- 28. Content Segments • STYP - segment type • SIDX - segment index • MOOF -movie fragment – MFHD - movie fragment header – TRAF - track fragment • TFHD - track fragment header • TFDT - track fragment decode time • TRUN - track fragment run • MDAT - media data container (actual media data) – Contains one or more samples (smallest described piece of content)
- 29. Pulling it all together • Using OSMF, we wrote a DASH specific IndexHandler and FileHandler • IndexHandler responsible for deciding url of next http request • FileHandler responsible for parsing data from segments and returning FLV Bytes
- 30. DashIndexHandler • When OSMF determines a new fragment needs to be loaded, it calls the DashIndexHandler. • When playing, getNextFile is called • When seeking, getFileForTime is called • Both return an HTTPStreamRequest, which OSMF uses to request the proper segment
- 31. DashFileHandler • When the requested file is returned, it is handed to the DashFileHandler for processing – initializeProcessing – gets decoder – processFileSegment – asks decoder to process data • We are using a Decoder called MP4Decoder • MP4Decoder returns FLVBytes for the data within the MDAT • processFileSegment returns ByteArray (FLVBytes) to OSMF, for adding to the NetStream buffer
- 32. MP4Decoder • processData() – iterates through all boxes in segment – Calls getNextBox() • getNextBox knows how to process each specific box type • When processing MDAT, individual samples are wrapped with FLVBytes • FLVBytes wrap specific binary header around data, as required by NetStream.appendBytes()
- 33. Digital Primates DASH Player • Currently our DASH player is not commercially available • We use it as the basis for DASH players we build for our clients • Initially built as a learning exercise to better understand DASH
- 34. Current DP Dash Player • We currently support SegmentLists, SegmentTemplates and SegmentBase in manifest – Byte Range Requests only currently supported via Akamai • We support Dynamic Adaptive Streaming for VOD Streams • We have tested with streams up to 1080p • We have tested with streams from several different encoders
- 35. DP Dash Player plans • 4Q12 • 1Q13 – Basic Seek Fixes – Initial JavaScript Player – Improved multi-channel – Support for DRM audio support – Support for Enhanced seeking – Support for Live • 2Q13 Streams – Investigate HEVC (h.265) – Continue to test and fix streams from more – Initial Android Player encoding partners • 3Q13 – Initial iOS Player
- 36. Resources • GPAC – http://gpac.wp.mines-telecom.fr – Provides baseline test streams – Provides baseline player • MP4Parser – http://code.google.com/p/mp4parser/ – Open Source java project – Allows for display of contents within boxes • DASH Industry Forum – http://www.dashif.org
- 37. Questions?