WebRTC Real time media P2P, Server, Infrastructure, and Platform
0 likes244 views
Real-time media can use P2P, server-based infrastructure, or a hybrid approach. Originally WebRTC used a direct P2P model but this does not scale well. Server-based approaches using media servers like SFUs and MCUs can scale to millions of users but require transcoding and break end-to-end encryption. A hybrid approach with an end-to-end media layer can provide scalability while maintaining low-latency and encryption by adapting to the weakest link in the network path.
WebRTC Real time media P2P, Server, Infrastructure, and Platform
- 1. Real-Time Media P2P, Server, Infrastructure, and Platform Dr. Alex Gouaillard
- 2. P2P: In the beginning Originally, the webrtc use case was 1-to-1, p2p, a-la facetime, or skype. The media flows directly between the two peers, which have a direct connection and thus can know almost everything about the connection of their respective remote peer. Bandwidth estimation, Encryption, Bandwidth adaptation, are all relatively trivial to support in that setting, and the only thing left to do is the discovery and handshake. The discovery is the process through which peers find each other. The handshake is the negotiation process which precede the media establishment and allow to assess capacity of each peer and exchange needed info like the encryption keys.
- 3. P2P: In the beginning A lot of first generation servers are thus signalling servers, servers which enable the discovery and facilitate the handshake for peers. Most of the &yet portfolio for example is based on p2p connections (simplepeer, talky, …) as well as the first version of tokbox, appear.in and almost any webrtc platform. This design was very appealing to PaaS designers as the cost of the media bandwidth, up to 85% of the operational cost, is fully externalized to the end user. However the main limitation of this design is the scalability. Extending the logic to multiparty chats is trivial, but the design forces e.g. multiple encoding of the same stream, once per remote peer, and uses a lot of CPU. The complexity is then quadratic wrt the # of users. Beyond scalability, features like recording, or inter-connection to other protocols for VoIP or streaming, all require a media server.
- 4. Note on Turn / Stun / ICE ICE is a solution to firewall and NAT traversal. It’s an algorithm based on the usage of either a STUN or a TURN server (purist will tell you a TURN server is actually a STUN server but not the other way around). Because of that TURN and STUN server are usually called ICE servers in implementations, but in commercial offers almost everybody speak about TURN services (twilio, xirsys, callstats.io, …). What is interesting is that TURN servers are always useful, they are independent of your back-end choice (p2p, sfu, infra, platform), they only act on the client side, and it’s the only part of a webrtc solution that can be completely separated from the rest.
- 5. Media Servers Roughly two kinds of media servers exists: SFU and MCU. Roughly the distinction is that MCU processes the media while SFU only relay/route media. NOTE: medooze is a media server toolkit. You have “bricks” at your disposal to put together to end up with a media server with the desired capabilities. It can be an SFU, an MCU, and any variations around. There is no such thing as a Medooze server, per say. Less granular: Janus as a core engine, and some plugins that can be used to add desired functionalities. So you speak about a janus server, with a given plugin. “Video Room” is the plugin which implements SFU functionalities, and the one we have SDKs and native clients for. Not granular: MediaSoup. Absolutely monolithic: Jitsi (red5, ant media).
- 6. Media Servers: MCUs MCUs need de decrypt the stream, reconstruct the encoded frame, decode it to get the raw media frame, apply some transformation to it, and do everything in reverse. ● MCU always remove the encryption are not secure by design, ● This process represents an extra 80% CPU consumption, ● Transcoding at least doubles the media processing latency on the path. ● If the incoming and outgoing codecs are different, it’s called transcoding, ● If the incoming and outgoing protocols are different, it’s called transmuxing, ● It is very versatile, and can reduce the bandwidth usage to a minimum (linear to # of users). MCU are mandatory to use when transcoding RTMP, RTSP, or HLS/MPEG-Dash to/from Webrtc. We are using an MCU configuration of meedoze for the corresponding media Servers. For a very long time, bitrate adaptation for streaming was done by reducing the resolution on the server. That requires transcoding, scaling down the frame in the middle of the process. They call it ABR. ABR is thus intrinsically unsecure, and costly both in terms of CPU footprint, and overall latency. In VoIP, where a single video stream is accepted by default, MCU are used to stitch several stream together and send it to a SIP client.
- 7. Media Servers: SFUs SFUs (or selective forward units), do just that: chose a stream and relay it. ● It is very lightweight since it does not process media, i.e. no encoding or decoding. ● Adding support for a new codec is trivial, since no encoding/decoding takes place. ● It is very well suited for multiparty conferences as it just duplicate the incoming packets, rewrite the destination address, and push it out. ● It is also insecure by default: to be able to send it to multiple peers, it needs to establish a separate encrypted connection with each, each with different keys. End-to-end encryption solves this problem and is implementable with an SFU, while impossible with MCU. Unless you want to lose the “end-to- end encrypted” webrtc claim, you need to add an additional end-to-end encryption layer when using an SFU. The original claim is void and null if using an MCU. ● Recording would require decoding the media stream. It is either implemented as a packet dump (rtp packets themselves are recorded, which allow post mortem network debugging, as well as on- demand media file generation), or in a multi-server environment, delegated to a separate MCU. ● Most SFU have extra reliability features like own per-outgoing stream packet cache, and jitter buffer and can help improve real-time reliability on bad networks. ● They are especially powerful when coupled with simulcast (any codec) and SVC (special codecs)
- 8. Media Server: Recording solution: SFUs + MCUs Keep the main path SFU only (transcoding free) for min latency ! Push the storage and MCU (mixer) out to the outgoing leg that requires it and only that one Keep pcap as the main storage format: allow debugging, contain media AND network info, and can be stored E2E encoded. RTMP ingest, if video only, and if single resolution can be done with an SFU. For other codecs support, and/or audio support, and/or simulcast support, you need an MCU. Here again, keep it off main path.
- 9. ● ABR makes separate parallel encoding of the same media source, and delivers the best one a viewer can consume. By chunking the encoded media, one can adapt the resolution delivered dynamically, with a reaction time equivalent to the chunk size (90s by default with HLS). It a server- side feature for viewer-side optimization. Transcoding is needed, doubling the media latency. It usually requires storages of the chunk prior to delivery, adding to the latency. Media Servers: Bandwidth Adaptation: ABR
- 10. ● Simulcast is exactly the same thing but done on the sender-side. It removes the need to manipulate the media on the server (so one can now use an SFU). It’s a sender-side feature for viewer-side optimization. Its reaction time depends on the time to the next full frame, usually 10s by default, but smart implementations generate a full frame on-demand. It also removes the need for an extra decoding/encoding cycle in the server, reducing the media latency by 2. It does not require storage. It still requires a separate encoder per resolution, which can be problematic on mobile device where only one hardware encoder is present. It works with any codec, even old ones. Media Servers: Bandwidth Adaptation: Simulcast
- 11. ● SVC is achieving the same thing as simulcast: getting multiple resolutions from a single source, but using only one encoder, with all the resolution layered into a single bitstream. The layering also allows for easier and more robust network resilience. The reaction time in the server to switch between one resolution to another one is in the order of one RTP packet read, around 10ms. ● It requires special codec. H.264 (AVC), H.265 (HEVC), VP9 all have an SVC mode. AV1 supports SVC by default. Media Servers: Bandwidth Adaptation: SVC
- 12. Streaming and Media Infrastructure Old protocols, like RTMP, and RTSP, are not encrypted by default, have no capacity to traverse NAT or firewall, and are not adaptive either. Practically it means that: ● anybody with access can watch, ● You need to open a port in your firewall and NAT ahead of time, ● You need to allocate and reserve enough bandwidth. If you go below, the call fails. Streaming protocols solve this by offsetting all the traffic over to HTTP, which is never blocked, leaving the security to the user, and use ABR for adaptability. The storage allows for scalability through CDNs. Using CDNs and HTTP (cache) allowed to achieve then a much lower cost of operation. Each of those solutions address the original problems by adding latency. WebRTC solved those problems in real-time, using respectively ICE, encryption by default, simulcast/SVC, but with a p2p model which does not scale so well. Nowaday, the cost of CDNs, and the cost of bandwidth are in par.
- 13. Streaming and Media Infrastructure Webrtc is much closer to RTMP and RTSP in nature than to HLS and MPEG-DASH. They are media transport protocols that do not use HTTP nor use file format/storage, everything is done in-memory and on-the-wire. Given the limitations of the protocols themselves, the only thing you could do with RTMP and RTSP was to cluster and (auto)-scale. Since there was no firewall / NAT traversal capacity, you would ignore them. Since there was no adaptability, you would not try to probe the bandwidth and to be a good network citizen by applying congestion control to yourself. You would assume x Mbps available and reserved for you at any time, and perfect network conditions. Then scaling becomes a simple matter of connecting servers to one another in chains from source to consumer/viewer of the media. It’s so simple, that free HTTP proxies like NGINX also implement RTMP routing, relaying and recording by default. With nowadays cloud services, providing the capacity to start and stop servers, to distribute the sources and the viewer across servers, and to route media from source to viewer is enough. You can automate all that through what is called “load balancing” and “auto-scaling”.
- 14. Streaming and Media Infrastructure: Stadia
- 15. Streaming and Media Infrastructure: Stadia
- 16. Streaming and Media Infrastructure A - Single, stand-alone server (red5, ant media, medooze) Stand-Alone SFU source viewer Max: 1K viewers viewer viewer viewer viewer Load Balancing Internal routing Scaling ● If webrtc, you still need a signalling server for discovery and handshake, but usually this is integrated in the media server itself (all the server above). Each source, respectively viewer, needs to discover and do a handshake with the media server separately. ● If rtmp or rtsp, you just need to provide the IP and PORT of the target computer, and you’re done. Of course, port needs to be open, network needs to be almost perfect, bandwidth needs to be plenty, and you cannot upgrade codecs (RTMP and RTSP at best support H264). ● Load balancing is the process of distributing the stream across several servers. But in the case of a single stand alone server, there is no need for distribution as both source and viewer will always connect to the same server. It makes things much simpler, but limit your audience to roughly 1,000 viewers. ● Routing, which is the process of connecting the source and the viewers, is trivial and can be done by the media server. ● Scaling, which is adding or removing media servers to a cluster or pool of servers is not needed, since there is only one media server. ● You need end-to-end encryption if you want to be secure. The distribution of the keys if of topic here.
- 17. Streaming and Media Infrastructure B - 2 servers in path, no e2e media layer (red5, ant, phenixRTS) source Ingest SFU egress SFU viewer viewer viewer viewer viewer egress SFU viewer viewer viewer viewer viewer Max: 1M viewers Load Balancing Internal routing N N N Scaling ● Same problems with handshake and discovery: ○ RTMP just connects by IP:PORT ○ Webrtc needs a signaling server for the source and the viewers to connect to the infrastructure ● Everything else being equal, this design can handle 1M viewers. ● The problems starts to arise here as the WebRTC Bandwidth Estimation, and Congestion Control algorithms fail when used with multi-hops media paths. Plus Noisy neighbors effects start to appear. ● The easy solution to that, is to switch back to ABR. It solves all those problems, at the cost of breaking RT bandwidth adaptability, adding latency, disabling end-to-end encryption and basically making webrtc into nothing more than RTMP. Easy to implement, does not “work” worse than RTMP, while providing better codec than RTMP, and Browser native support.
- 18. Streaming and Media Infrastructure B - 2 servers in path, no e2e media layer (red5, ant, phenixRTS) source Ingest SFU egress SFU viewer viewer viewer viewer viewer egress SFU viewer viewer viewer viewer viewer Max: 1M viewers Load Balancing Internal routing N N N Scaling ● Here you suppose that instead of a single ingest SFU, you have a pool of N of them. ● In simple designs (LUXON v1), one simply add or remove instances to the pool manually. In more advanced design (LUXON v2, red5, ant, phenixRTS), the pools auto-scale, i.e. servers are being added or removed on-demand depending e.g. on the load. ● Load balancing manages how source, resp viewers, are being directed to a specific server. In simple designs (LUXON v1), one use a round-robin algorithm which send each new connection to the next server in a list of server (in order), and when reaching the end of the list, resumes at the beginning of the list. It works well when the number of servers is fixed, and when all servers are idempotent. ● Routing becomes more complicated here. In simple designs (LUXON v1), we would just relay the ingest stream to all egress SFUs. We would then be sure viewer can connect to any egress SFU and have any stream available. Obviously, it does not scale well and wastes a lot of bandwidth. In LUXON v2, we have a “director API” which does the routing and the load balancing on-demand. (slides exist)
- 19. Streaming and Media Infrastructure C - 2+ servers in path, no e2e media layer Load Balancing Internal routing source Ingest SFU egress SFU viewer viewer viewer viewer viewer egress SFU viewer viewer viewer viewer viewer Max: 1G viewers Relay SFU Relay SFU N N N Scaling N N ● Having a extra media server in the path allows for extra scalability (up to 1 Trillion theoretically). ● It also allow for smarter routing, and better connections managements, including multi-clouds. ● We have the 3 configurations with Medooze. We have a version of LUXON like this.
- 20. Streaming and Media Infrastructure D - 2 servers in path, with e2e media layer (LUXON) Load Balancing Internal routing Scaling ● The problems starts to arise here as the WebRTC Bandwidth Estimation, and Congestion Control algorithms fail when used with multi-hops media paths. Plus Noisy neighbors effects start to appear (slide). ● The solution, is to take into account the network status and capacity at each server in the media path. ● One such algorithm is called BBR, (Bottleneck Bandwidth and Round-Trip Time). It’s conceptually easy to understand, You should not send more than the weak- -est network in the path can handle. ● This is e.g. what Google uses in Stadia, What is used in Facebook’s Quic impl. ● It requires a full monitoring, interactive E2e Media layer, on top of the media Servers, and in addition to scaling/routing. It cannot be in a single server by design. It’s a media infrastructure level feature.
- 21. Streaming and Media Infrastructure: Stadia full monitoring, Interactive, E2E Media layer, Separated from Media Servers, Separated from scaling/routing.
- 22. Streaming and Media Infrastructure D - 2+ servers in path, with e2e media layer (LUXON) Load Balancing Internal routing source Ingest SFU egress SFU viewer viewer viewer viewer viewer egress SFU viewer viewer viewer viewer viewer Server (Medooze™) & Clients/SDKs layer (Include E2E encryption) (include watermarking) (very differentiating) Relay SFU Relay SFU N N N Scaling LB LB Smart Routing N N Infrastructure scalability layer: “Director API”™ (not differentiating) Bandwidth Bottleneck and RTT + other black magic Infrastructure BWE and CC layer: “RT Media Enabler”™ (UNIQUE) LUXON Media Only
- 23. Media Streaming PaaS - Millicast source Ingest SFU egress SFU viewer viewer viewer viewer viewer egress SFU viewer viewer viewer viewer viewer Server (Medooze™) & Clients/SDKs layer (Include E2E encryption) (include watermarking) (very differentiating as traditionally most vendors are server vendors with no native SDK capacity) Relay SFU Relay SFU N N N LB LB Smart Routing N N Infrastructure scalability/logic layer: “Director API”™ (not differentiating) Bandwidth Bottleneck and RTT + other black magic Infrastructure BWE and CC layer: “RT Media path manager”™ (UNIQUE) EXT: OAuth, Tokens, Analytics, Billing, Payment, All APIs. Platform layer: “Influxis Synapse”™ (little differentiation) Hosting / Cloud Management INT: monitoring, notifications, Recording/storage.
- 24. RTMP Ingest support - in production Core webrtc media path for minimum latency, RTMP first mile WebRTC source Ingest SFU egress SFU viewer viewer viewer viewer viewer egress SFU viewer viewer viewer viewer viewer Relay SFU Relay SFU N N N N N ● NOW: No video transcoding (=> no simulcast, no SVC support)(=> only support H264) ● NOW: Audio transcoding to opus stereo at best (can lose quality if input is 5.1 AAC) ● NOW: added latency from media processing: 1 frame at most (1/30s = 33ms) ● PLANNED: Video Transcoding with simulcast / SVC, and capacity to choose which codec. ● PLANNED: Audio transcoding to multiOpus if input if 5.1 or 6.1. No loss of spatial quality. RTMP / RTSP source Ingest MCU N
- 25. RTMP/HLS/MPEG-DASH egress support Core webrtc media path for minimum latency, WebRTC source Ingest SFU egress SFU viewer viewer viewer viewer viewer egress SFU viewer viewer viewer viewer viewer Relay SFU Relay SFU N N N N N RTMP MCU N HLS MCU N Social Site CDN Facebook, Twitch ● The media part (medooze transcoding MCU) is done. ● The difficult part now is the logic and the routing needed to integrate in LUXON. ● Then, cabling it in millicast with the business logic (accounting, billing, authentification, …) ● Connecting to FB or a CDN could and should be automated as well.
- 26. Server-Side Ad-Insertion Core webrtc media path for minimum latency, ● The media part (medooze transcoding MCU) is done. ● The difficult part now is the logic and the routing needed to integrate in LUXON. ● Then, cabling it in millicast with the business logic (accounting, billing, authentification, …) ● The ad targeting is missing, pending viewer analytics availability. Collaboration project with DataZoom. Source Ad Decision SFU (Medooze) View er request response ad content Transcoder (Medooze) Ad Storage Live Stream W ebRTC Ad Request Ad content HLS/MPEG-DASH Ad content W ebRTC Live Stream + Ad W ebRTC VAST 1 2 3 4 5 6 7
- 27. ANNEX 4: Other Features Core webrtc media path for minimum latency, ● E2EE with SFrame. Ready to integrate. Native Apps only for now. ● multiOpus. Ready to integrate. Web and Native. Hidden in chrome, risk that they remove it. ● Watermarking. Trivial to integrate (client side only). Native Apps only for now. Needs more research on the Watermarking algorithm, currently not robust enough. Needs Ludo time. Slow (R&D). ● AV1: not ready to integrate, wait for SVC from Google. ● webXR with RTP Timestamp based sync. No POC yet.