Multistream in Janus @ CommCon 2019

1. “Sounds like a plan!” Or how I added multistream to Janus using Uniﬁed Lorenzo Miniero @elminiero CommCon 2019 July 8th 2019, Latimer Estate, Buckinghamshire (UK)

2. Couldn’t ﬁnd a good picture of me Lorenzo Miniero • Ph.D @ UniNA • Chairman @ Meetecho • Should probably eat more Contacts and info • lorenzo@meetecho.com • https://twitter.com/elminiero • https://www.slideshare.net/LorenzoMiniero

3. A few words on Meetecho • Co-founded in 2009 as an academic spin-off • University research efforts brought to the market • Completely independent from the University • Focus on real-time multimedia applications • Strong perspective on standardization and open source • Several activities • Consulting services • Commercial support and Janus licenses • Streaming of live events (IETF, ACM, etc.)

4. Proudly brewed in sunny Napoli, Italy!

5. First of all, what is Janus? Janus General purpose, open source WebRTC server • https://github.com/meetecho/janus-gateway • Demos and documentation: https://janus.conf.meetecho.com • Community: https://groups.google.com/forum/#!forum/meetecho-janus

6. Modular architecture • The core only implements the WebRTC stack • JSEP/SDP, ICE, DTLS-SRTP, Data Channels, ... • Plugins expose Janus API over different “transports” • Currently HTTP / WebSockets / RabbitMQ / Unix Sockets / MQTT / Nanomsg • “Application” logic implemented in plugins too • Users attach to plugins via the Janus core • The core handles the WebRTC stuff • Plugins route/manipulate the media/data • Plugins can be combined on client side as “bricks” • Video SFU, Audio MCU, SIP gatewaying, broadcasting, etc.

10. Extensible Architecture and API

11. Extensible Architecture and API

12. A known limitation, though... • Since day one, PeerConnections in Janus had a well-known “limitation” • Only one stream and m-line per media type allowed • PeerConnections limited to 1 audio + 1 video + 1 data channel • Never limited the Janus functionality... • You simply need more PeerConnections if you need more audio/video streams • ... but bundling streams together can be really useful, though • e.g., to reduce networking overhead and number of PeerConnections Why not add it years ago, then? • Missing interoperability between Chrome and Firefox is what stopped us • We really didn’t want to implement both just to drop one later • Decided to focus on features instead (simulcast, SVC, etc.)

16. Separate PeerConnections in the SFU scenario

17. Multistream in WebRTC: so many plans! • Plan B • https://tools.ietf.org/html/draft-uberti-rtcweb-plan-00 • One m-line per media type (msid/SSRCs to identify streams) • Originally implemented by Chrome • Uniﬁed Plan (originally Plan A) • https://tools.ietf.org/html/draft-roach-mmusic-uniﬁed-plan-00 • A separate m-line per media type (mid/rid to identify streams) • Originally implemented by Firefox • No Plan (!) • https://tools.ietf.org/html/draft-ivov-rtcweb-noplan-01 • Attempt to reduce the number of offer/answer exchanges • Never implemented, AFAICT?

20. An example of Plan B [..] a=group:BUNDLE audio video a=msid-semantic: WMS SPpFifgubAElUYAO cOtMlbdcVRU6JHeu m=audio 41535 UDP/TLS/RTP/SAVPF 100 [..] a=ssrc:769372116 cname:SPpFifgubAElUYAO a=ssrc:769372116 msid:SPpFifgubAElUYAO 812931fd-94f4-4eca-b8d7-55789c386642 a=ssrc:769372116 mslabel:SPpFifgubAElUYAO a=ssrc:769372116 label:812931fd-94f4-4eca-b8d7-55789c386642 m=video 41535 UDP/TLS/RTP/SAVPF 101 102 [..] a=ssrc-group:FID 912016579 529821397 a=ssrc:912016579 cname:SPpFifgubAElUYAO a=ssrc:912016579 msid:SPpFifgubAElUYAO f71ba339-22bd-4bbf-b99f-00aaffec36d7 a=ssrc:912016579 mslabel:SPpFifgubAElUYAO a=ssrc:912016579 label:f71ba339-22bd-4bbf-b99f-00aaffec36d7 a=ssrc:529821397 cname:SPpFifgubAElUYAO a=ssrc:529821397 msid:SPpFifgubAElUYAO f71ba339-22bd-4bbf-b99f-00aaffec36d7 a=ssrc:529821397 mslabel:SPpFifgubAElUYAO a=ssrc:529821397 label:f71ba339-22bd-4bbf-b99f-00aaffec36d7 a=ssrc-group:FID 823879851 508644174 a=ssrc:823879851 cname:cOtMlbdcVRU6JHeu a=ssrc:823879851 msid:cOtMlbdcVRU6JHeu 0772619f-ea4d-4e99-8122-92709b2bc295 a=ssrc:823879851 mslabel:cOtMlbdcVRU6JHeu a=ssrc:823879851 label:0772619f-ea4d-4e99-8122-92709b2bc295 a=ssrc:508644174 cname:cOtMlbdcVRU6JHeu a=ssrc:508644174 msid:cOtMlbdcVRU6JHeu 0772619f-ea4d-4e99-8122-92709b2bc295 a=ssrc:508644174 mslabel:cOtMlbdcVRU6JHeu a=ssrc:508644174 label:0772619f-ea4d-4e99-8122-92709b2bc295

25. An example of Uniﬁed Plan [..] a=group:BUNDLE 0 1 2 3 a=msid-semantic: WMS SPpFifgubAElUYAO cOtMlbdcVRU6JHeu m=audio 41535 UDP/TLS/RTP/SAVPF 100 [..] a=mid:0 a=msid:SPpFifgubAElUYAO 812931fd-94f4-4eca-b8d7-55789c386642 a=ssrc:769372116 cname:SPpFifgubAElUYAO m=video 41535 UDP/TLS/RTP/SAVPF 101 102 [..] a=mid:1 a=msid:SPpFifgubAElUYAO f71ba339-22bd-4bbf-b99f-00aaffec36d7 a=ssrc-group:FID 912016579 529821397 a=ssrc:912016579 cname:SPpFifgubAElUYAO a=ssrc:529821397 cname:SPpFifgubAElUYAO [..] m=video 41535 UDP/TLS/RTP/SAVPF 101 102 [..] a=mid:2 a=msid:cOtMlbdcVRU6JHeu 0772619f-ea4d-4e99-8122-92709b2bc295 a=ssrc-group:FID 823879851 508644174 a=ssrc:823879851 cname:cOtMlbdcVRU6JHeu a=ssrc:508644174 cname:cOtMlbdcVRU6JHeu

29. A Plan to Unify them all! • IETF consensus on Unified Plan was reached a long time ago • IETF 87, Summer of 2013! • https://webrtchacks.com/a-hitchhikers-guide-to-webrtc-standardization/ • Firefox was the first to implement it, a couple of years later • https://hacks.mozilla.org/2015/03/webrtc-in-firefox-38-multistream-and-renegotiation/ • Chrome only started implementing it recently, though • https://webrtc.org/web-apis/chrome/unified-plan/ Translated... No more excuses for us, and time to work on it!

33. Thanks to Highﬁve for sponsoring the development! https://github.com/meetecho/janus-gateway/pull/1459

34. How much of a refactoring of the Janus internals? • For our existing PeerConnections, it didn’t matter much • For one audio and one video stream, plans are basically interoperable • ... well, kinda (you still need to be prepared for mid, transceivers, etc.) • Multistream support required a considerable refactoring, though, of: 1 SDP parsing and generation utils 2 Support for multiple streams in the core • Hardcoded references to single audio/video streams • Routing and addressing of individual streams 3 Support for multiple streams in all (most?) plugins 4 And let’s not forget the client side of things!

40. Not an easy task, as you can imagine!

41. SDP parsing and generation • Janus includes “homemade” SDP utils to quickly parse/generate SDPs • Used by the core to handle WebRTC SDP negotiation • Used by plugins as well to negotiate what they want • Particularly helpful when creating new SDPs • janus_sdp_generate_offer() vs. janus_sdp_generate_answer() • Variable length methods with ﬂags to customize output • m-lines and attributes as lists that can then be pruned/extended • Assumption about 1 audio/1 video, though, implied simpler syntax • “I want audio, but not video” • “Use VP9 for video, and add this fmtp attribute”

44. SDP generation before... (offer) janus_sdp *offer = janus_sdp_generate_offer("My session", "127.0.0.1", JANUS_SDP_OA_AUDIO, TRUE, JANUS_SDP_OA_AUDIO_PT, 100, JANUS_SDP_OA_AUDIO_DIRECTION, JANUS_SDP_SENDRECV, JANUS_SDP_OA_AUDIO_CODEC, "opus", JANUS_SDP_OA_VIDEO, TRUE, JANUS_SDP_OA_VIDEO_PT, 96, JANUS_SDP_OA_VIDEO_DIRECTION, JANUS_SDP_SENDONLY, JANUS_SDP_OA_VIDEO_CODEC, "vp8", JANUS_SDP_OA_DATA, FALSE, JANUS_SDP_OA_DONE);

45. SDP generation before... (answer) janus_sdp *answer = janus_sdp_generate_answer(offer, JANUS_SDP_OA_AUDIO, TRUE, JANUS_SDP_OA_AUDIO_DIRECTION, JANUS_SDP_RECVONLY, JANUS_SDP_OA_AUDIO_CODEC, "opus", JANUS_SDP_OA_VIDEO, FALSE, JANUS_SDP_OA_DATA, FALSE, JANUS_SDP_OA_DONE);

46. SDP generation now! (offer) janus_sdp *offer = janus_sdp_generate_offer("My session", "127.0.0.1", JANUS_SDP_OA_MLINE, JANUS_SDP_AUDIO, JANUS_SDP_OA_MID, "audio1", JANUS_SDP_OA_PT, 100, JANUS_SDP_OA_DIRECTION, JANUS_SDP_SENDONLY, JANUS_SDP_OA_CODEC, "opus", JANUS_SDP_OA_MLINE, JANUS_SDP_VIDEO, JANUS_SDP_OA_MID, "video1", JANUS_SDP_OA_PT, 96, JANUS_SDP_OA_DIRECTION, JANUS_SDP_RECVONLY, JANUS_SDP_OA_CODEC, "vp8", JANUS_SDP_OA_MLINE, JANUS_SDP_AUDIO, JANUS_SDP_OA_MID, "audio2", JANUS_SDP_OA_PT, 0, JANUS_SDP_OA_DIRECTION, JANUS_SDP_SENDRECV, JANUS_SDP_OA_CODEC, "pcmu", JANUS_SDP_OA_AUDIO_DTMF, TRUE, JANUS_SDP_OA_MLINE, JANUS_SDP_VIDEO, JANUS_SDP_OA_MID, "video2", JANUS_SDP_OA_ENABLED, FALSE, JANUS_SDP_OA_PT, 102, JANUS_SDP_OA_DIRECTION, JANUS_SDP_RECVONLY, JANUS_SDP_OA_CODEC, "h264", JANUS_SDP_OA_MLINE, JANUS_SDP_APPLICATION, JANUS_SDP_OA_MID, "data", JANUS_SDP_OA_DONE);

47. SDP generation now! (answer) janus_sdp *answer = janus_sdp_generate_answer(offer); GList *temp = offer->m_lines; while(temp) { janus_sdp_mline *m = (janus_sdp_mline *)temp->data; if(m->type == JANUS_SDP_AUDIO) { janus_sdp_generate_answer_mline(offer, answer, m, JANUS_SDP_OA_CODEC, "opus", JANUS_SDP_OA_DIRECTION, JANUS_SDP_RECVONLY, JANUS_SDP_OA_ACCEPT_EXTMAP, JANUS_RTP_EXTMAP_MID, [..] JANUS_SDP_OA_DONE); } } else { [..] } temp = temp->next; }

48. Multistream in the Janus core • Assumptions on PeerConnections limits meant some hardcoded properties • Audio related properties (all assuming a single stream) • Video related properties (all assuming a single stream, and maybe simulcast) • Single datachannel • This had an impact on the media routing as well • Incoming packet could be either audio or video (or data) • Same level of multiplexing when talking to plugins as well • Besides, some “legacy” code meant info was in different places • Old stream/component structure based on ICE concepts • Some properties on the same thing in the former, other in the latter • Made sense initially, when we allowed non-bundle, but not anymore...

51. Structures in Janus before...

52. Structures in Janus now, multistream-aware!

53. Demultiplexing traffic in the core • Addressing traffic was updated as a consequence • Not just "is it audio or video", but "which medium does this belong to?" • Demultiplexing based on different information • RTP vs RTCP vs data (exactly as before) • For RTP/RTCP, mapping between known SSRC and existing medium • For RTP, mapping with mid/rid in extension (if we need to figure out SSRC) • Media (and stats) then internally addressed and routed basing on index or mid Wait, figure out SSRC?! • Yep, it might happen, e.g., with the new Chrome simulcast! • https://www.meetecho.com/blog/simulcast-janus-ssrc/

57. Making plugins multistream-aware • Support for multistream in the core was of course only the ﬁrst step • If plugins don’t use it, it’s worthless! • Main step was updating the media routing API • incoming_rtp() and incoming_rtcp() updated with m-index info • relay_rtp() and relay_rtcp() updated with m-index info as well • Everything else up to plugins themselves • e.g., SDP negotiation, support for managing multiple streams, etc. Decided to only start with a few key plugins • EchoTest • Streaming • VideoRoom

61. A multistream EchoTest plugin • First plugin we updated was obviously the EchoTest • Simple playground for both signalling and media • Every packet on each stream is sent back • Relatively small changes • Updated usage of SDP utils to generate multistream answer • Updated RTP routing methods/callbacks to make them aware of m-indexes Works nicely already, although it still needs some tweaks • Currently doesn’t support simulcast on more than one stream (because I’m lazy) • PLIs should be generated on all video streams as well

64. Multistream EchoTest plugin example

65. Multistream EchoTest plugin example

66. Ok, it works, can we stop now?

67. This is getting ridiculous

68. WebRTC developers have no friends

69. A multistream Streaming plugin • Streaming plugin required some more effort • Same assumptions on single audio/video as in the core • Ports for audio/video/data hardcoded in here as well • Original mountpoint configuration quite rigid as a consequence • Huge refactoring of mountpoint internals • Mountpoints as generic array of configurable streams • Each stream can be audio, video or data, with common properties • Static configuration changed to take advantage of libconfig arrays • Dynamic API updated as well to reflect this new flexibility Much more flexible now! • ... although streams list can’t be modified once created, but whatever

72. Conﬁguring a multistream mountpoint multistream-test: { type = "rtp" id = 123 description = "Multistream test (1 audio, 2 video)" media = ( { type = "audio" mid = "a" label = "Audio stream" port = 5102 pt = 111 rtpmap = "opus/48000/2" }, { type = "video" mid = "v1" label = "Video stream #1" port = 5104 pt = 100 rtpmap = "VP8/90000" }, { type = "video" mid = "v2" label = "Video stream #2" port = 5106 pt = 100 rtpmap = "VP8/90000" }

73. Multistream Streaming plugin example

74. A multistream VideoRoom plugin • VideoRoom plugin was even harder than that... • Not only multistream subscribers, but multistream publishers as well! • First decision was to keep publishers and subscribers separated • Not “one PeerConnection to rule them all”, but two! • All active streams on one PC (publishers) • All passive streams on another PC (subscribers) • Several reasons behind that • Avoiding glare, of course (easier when O/A pattern is always the same) • Smarter management of resources • And most importantly, we’re not the only ones doing that! (e.g., mediasoup) • Of course, old approach still supported • Freedom to distribute publishers and subscribers however you want

79. Aggregating publisher/subscriber streams

83. A multistream VideoRoom plugin • For the rest, several updates required • Refactoring of publishers and subscribers as collection of streams • Refactored media relationships at a stream level • Updated code to use the new SDP utils for crafting SDPs • Existing features now stream-speciﬁc, rather than publisher/subscriber speciﬁc • e.g., in theory possible to send multiple different simulcast video streams • Updated API to allow for dynamic updates on existing PeerConnections • e.g., subscribe/unsubscribe at any time, with metadata on the streams Two demos now available (and interoperable!) 1 videoroomtest.html: legacy approach, same as old one 2 mvideoroomtest.html: multistream version of the demo above

87. “Legacy” VideoRoom plugin example

88. Multistream VideoRoom plugin example

89. They might look the same, but actually...

90. Tone it down, Skeletor, we’re not done yet!

91. Another challenge: data channels! • With separate PeerConnections in the VideoRoom, relaying is easy • If I want data from Alice, I’ll subscribe negotiating datachannels too • Incoming data on that PeerConnection will only come from Alice • What if the same PeerConnection is used for all subscriptions, though? • Only one datachannel is created, and shared for all sources • Who is the incoming data on that PeerConnection from?! Problem solved by starting to use datachannels to their full potential • Added support for multiple streams/labels (instead of just one, as before) • Plugins can associate different labels to different sources • Easy to demultiplex incoming messages from an application perspective

94. Last step: updating the client side (janus.js) • Client-side, the biggest thing to be aware of are transceivers • https://blog.mozilla.org/webrtc/the-evolution-of-webrtc/ • https://webrtc.org/web-apis/chrome/unified-plan/ • In a nutshell, a way to pair a sender and a receiver • e.g., a local track and a remote track • Each transceiver maps to a specific m-line • State can then be controlled/monitored there (e.g., media direction) • As such, first step was to check if the browser supports transceivers • Chrome >= 72 and Firefox >= 59 both do • More specific checks also available to help in other cases • Chrome may actually need an explicit way of enabling it • new RTCPeerConnection ({sdpSemantics: "unified-plan"});

98. Last step: updating the client side (janus.js) • Second step was updating how we notified remote streams • Before: onlocalstream / onremotestream • Now: onlocaltrack / onremotetrack • Before, we notified a MediaStream object • No issue since it could only be 1 audio + 1 video • Subsequent calls to same callback would update the stream state • Now that we can have multiple heterogeneous m-lines, we notify tracks instead • Notification when track is added/unmuted/muted/removed (with info on mid) • Each track is also played in a separate element, in the updated demos • Avoids the “no audio while still waiting for video” annoyance Still missing... An easy way to add/replace/remove local tracks (because, again, I’m lazy!)

103. Next steps? • The effort is basically done, and we’ll probably merge soon • Mostly reacting to feedback and bug reports right now • JavaScript code needs some love, though... • Of course, there’s always room for improvements! • VideoRoom will need a new scaling mechanism (material for another talk!) • Possibly extend multistream support to other plugins too? (e.g., Lua/Duktape) • ... or maybe even cross-plugins! (if that makes sense) Test test test! • If you’re using Janus already, start playing with this! • If you’ve never used Janus before, then it’s the perfect moment to start

106. See you soon in Napoli! September 23-25, 2019, Napoli — https://januscon.it

107. See you soon in Napoli! September 23-25, 2019, Napoli — https://januscon.it

108. Thanks! Questions? Comments? Contacts • https://twitter.com/elminiero • https://twitter.com/meetecho • https://www.meetecho.com

Multistream in Janus @ CommCon 2019

More Related Content

What's hot (20)

Similar to Multistream in Janus @ CommCon 2019 (20)

More from Lorenzo Miniero (20)

Recently uploaded (20)

Multistream in Janus @ CommCon 2019