Multistream in SIP and NoSIP @ OpenSIPS Summit 2025
- 1. Multistream support in the Janus SIP and NoSIP plugins Lorenzo Miniero @ Meetecho OpenSIPS Summit @ Amsterdam May 28, 2025
- 2. Janus and OpenSIPS: super cool!
- 3. Who am I? Lorenzo Miniero • Ph.D @ UniNA • Chairman @ Meetecho • Main author of Janus and imquic Contacts and info • lorenzo@meetecho.com • https://fosstodon.org/@lminiero • https://bsky.app/profile/lminiero.it • https://www.threads.net/@lminier0 • https://www.meetecho.com • https://lminiero.it
- 4. Just a few words on Meetecho • Co-founded in 2009 as an academic spin-off • University research efforts brought to the market • Completely independent from the University • Focus on real-time multimedia applications • Strong perspective on standardization and open source • Several activities • Consulting services • Commercial support and Janus licenses • Streaming of live events (IETF, ACM, etc.) • Proudly brewed in sunny Napoli, Italy
- 5. Speaking of Napoli... we did it #ag4in!
- 6. Speaking of Napoli... we did it #ag4in!
- 7. Speaking of Napoli... we did it #ag4in!
- 8. A bit of context: Janus, WebRTC and SIP General purpose, open source WebRTC server • https://github.com/meetecho/janus-gateway • Demos and documentation: https://janus.conf.meetecho.com • Community: https://janus.discourse.group/
- 9. An endpoint of behalf of WebRTC users • Janus SIP plugin acts as a collection of SIP endpoints • SIP stack implemented with Sofia-SIP • WebRTC users only see the Janus API (JSON), no SIP • No transcoding, media is only relayed • Built-in recording (separate media legs) • Experimental trunking support 1 1https://github.com/meetecho/janus-gateway/pull/3501
- 10. An endpoint of behalf of WebRTC users • Janus SIP plugin acts as a collection of SIP endpoints • SIP stack implemented with Sofia-SIP • WebRTC users only see the Janus API (JSON), no SIP • No transcoding, media is only relayed • Built-in recording (separate media legs) • Experimental trunking support 1 • Simplifies life for web developers • No need to worry about a SIP stack (only SIP URIs) • Basic methods/events to handle dialogs (call, answer, hangup, message, etc.) • Allows SIP headers injection/interception in many requests • Support for more advanced features too (e.g., hold, transfer, etc.) 1https://github.com/meetecho/janus-gateway/pull/3501
- 11. An endpoint of behalf of WebRTC users • Janus SIP plugin acts as a collection of SIP endpoints • SIP stack implemented with Sofia-SIP • WebRTC users only see the Janus API (JSON), no SIP • No transcoding, media is only relayed • Built-in recording (separate media legs) • Experimental trunking support 1 • Simplifies life for web developers • No need to worry about a SIP stack (only SIP URIs) • Basic methods/events to handle dialogs (call, answer, hangup, message, etc.) • Allows SIP headers injection/interception in many requests • Support for more advanced features too (e.g., hold, transfer, etc.) • NoSIP plugin is there when you only need a media gateway • You handle SIP, Janus deals with the media (e.g., in OpenSIPS) 1https://github.com/meetecho/janus-gateway/pull/3501
- 12. SIP plugin in Janus https://janus.conf.meetecho.com/docs/sip
- 13. NoSIP plugin in Janus (media gw only) https://janus.conf.meetecho.com/docs/nosip
- 14. A known limitation, though... • In the past, PeerConnections in Janus had a well-known “limitation” • Only one stream and m-line per media type allowed • PeerConnections limited to 1 audio + 1 video + 1 data channel
- 15. A known limitation, though... • In the past, PeerConnections in Janus had a well-known “limitation” • Only one stream and m-line per media type allowed • PeerConnections limited to 1 audio + 1 video + 1 data channel • Limitation removed in Janus 1.x (multistream refactoring)... • Support for multiple streams of the same type in the same PeerConnection • Functionality added to EchoTest, Streaming and VideoRoom plugins
- 16. A known limitation, though... • In the past, PeerConnections in Janus had a well-known “limitation” • Only one stream and m-line per media type allowed • PeerConnections limited to 1 audio + 1 video + 1 data channel • Limitation removed in Janus 1.x (multistream refactoring)... • Support for multiple streams of the same type in the same PeerConnection • Functionality added to EchoTest, Streaming and VideoRoom plugins • ... but SIP and NoSIP plugins still stuck with it! • Core supported multiple streams per call, those plugins didn’t • “Lazy” rationale: most calls don’t need multiple audio/video streams anyway • Made adding support for new media harder, though (e.g., data channels)
- 19. Bringing multistream to SIP/NoSIP • Media stack in SIP and NoSIP plugins mostly the same • Struct with hardcoded audio/video details in user object • Sockets, ports, payload types, etc. • Helper functions hardcoded to deal with audio/video too
- 20. Bringing multistream to SIP/NoSIP • Media stack in SIP and NoSIP plugins mostly the same • Struct with hardcoded audio/video details in user object • Sockets, ports, payload types, etc. • Helper functions hardcoded to deal with audio/video too • Changed struct to array of generic “media” lines • Property defines type of media (audio, video, data) • Everything else is generic and shared
- 21. Bringing multistream to SIP/NoSIP • Media stack in SIP and NoSIP plugins mostly the same • Struct with hardcoded audio/video details in user object • Sockets, ports, payload types, etc. • Helper functions hardcoded to deal with audio/video too • Changed struct to array of generic “media” lines • Property defines type of media (audio, video, data) • Everything else is generic and shared • Had to deal with a few challenges • File descriptors moved to hash tables (demultiplexing) • Optional SDES-SRTP setup had to be made more flexible too • Aligning media differences in SIP and NoSIP stacks wasn’t always easy
- 22. Argh, Sofia SIP, not you too! • Bumped into a bit of a head scratcher in the SIP plugin • SDP in the NoSIP plugin handled by Janus entirely • SDP in the SIP plugin goes through Sofia SIP (NUA)
- 23. Argh, Sofia SIP, not you too! • Bumped into a bit of a head scratcher in the SIP plugin • SDP in the NoSIP plugin handled by Janus entirely • SDP in the SIP plugin goes through Sofia SIP (NUA) • By default, though, Sofia uses SOA to deal with SDP • SOA = SDP Offer/Answer Engine • Implements filtering of SDP as part of offer/answer
- 24. Argh, Sofia SIP, not you too! • Bumped into a bit of a head scratcher in the SIP plugin • SDP in the NoSIP plugin handled by Janus entirely • SDP in the SIP plugin goes through Sofia SIP (NUA) • By default, though, Sofia uses SOA to deal with SDP • SOA = SDP Offer/Answer Engine • Implements filtering of SDP as part of offer/answer • Normally not an issue, but it breaks with multistream • Second m-line of the same type always rejected (port=0) • Possibly SOA was never coded to deal with multiple audio/video streams?
- 25. Argh, Sofia SIP, not you too! • Bumped into a bit of a head scratcher in the SIP plugin • SDP in the NoSIP plugin handled by Janus entirely • SDP in the SIP plugin goes through Sofia SIP (NUA) • By default, though, Sofia uses SOA to deal with SDP • SOA = SDP Offer/Answer Engine • Implements filtering of SDP as part of offer/answer • Normally not an issue, but it breaks with multistream • Second m-line of the same type always rejected (port=0) • Possibly SOA was never coded to deal with multiple audio/video streams? • Solution: simply disable SOA entirely, and deal with SDP in Janus • NUTAG_MEDIA_ENABLE(0) • SIPTAG_CONTENT_TYPE_STR(s) + SIPTAG_PAYLOAD_STR(s)
- 28. (about the finger, no, I don’t know why!)
- 29. What about generic data? • Topic of my presentation last year in Valencia • “WebRTC-to-SIP and back – It’s not all about audio and video!”
- 30. What about generic data? • Topic of my presentation last year in Valencia • “WebRTC-to-SIP and back – It’s not all about audio and video!” • Non audio/video media could be gateway-ed through data channels • e.g., Real-Time Text (T.140), MSRP, BFCP, ... • All could be exchanged via data channels on the WebRTC side • Before, we needed new hardcoded details for that (audio/video/XYZ)
- 31. What about generic data? • Topic of my presentation last year in Valencia • “WebRTC-to-SIP and back – It’s not all about audio and video!” • Non audio/video media could be gateway-ed through data channels • e.g., Real-Time Text (T.140), MSRP, BFCP, ... • All could be exchanged via data channels on the WebRTC side • Before, we needed new hardcoded details for that (audio/video/XYZ) • Multistream support helps, but there are challenges • Main one would be SDP asymmetry, if more protocols are used • N m-lines on the SIP side, only one for WebRTC (data channels)
- 32. What about generic data? • Topic of my presentation last year in Valencia • “WebRTC-to-SIP and back – It’s not all about audio and video!” • Non audio/video media could be gateway-ed through data channels • e.g., Real-Time Text (T.140), MSRP, BFCP, ... • All could be exchanged via data channels on the WebRTC side • Before, we needed new hardcoded details for that (audio/video/XYZ) • Multistream support helps, but there are challenges • Main one would be SDP asymmetry, if more protocols are used • N m-lines on the SIP side, only one for WebRTC (data channels) • Mapping of m-lines would need to be taken care of accordingly • Especially important for renegotiations and RTP bridging
- 33. Next steps • Merge this! • Currently marked as a draft PR, since it’s experimental • Waiting for testing/feedback from the community
- 34. Next steps • Merge this! • Currently marked as a draft PR, since it’s experimental • Waiting for testing/feedback from the community • Will this actually be needed? • How many use cases are there for multiple A/V streams in SIP? • Could this be useful in OpenSIPS’s integration of NoSIP?
- 35. Next steps • Merge this! • Currently marked as a draft PR, since it’s experimental • Waiting for testing/feedback from the community • Will this actually be needed? • How many use cases are there for multiple A/V streams in SIP? • Could this be useful in OpenSIPS’s integration of NoSIP? • We need a plan for data channels too • See my talk @ OpenSIPS Summit 2024 in Valencia • Porting the existing Real-Time Text (T.140) PR could be a good start
- 36. Next steps • Merge this! • Currently marked as a draft PR, since it’s experimental • Waiting for testing/feedback from the community • Will this actually be needed? • How many use cases are there for multiple A/V streams in SIP? • Could this be useful in OpenSIPS’s integration of NoSIP? • We need a plan for data channels too • See my talk @ OpenSIPS Summit 2024 in Valencia • Porting the existing Real-Time Text (T.140) PR could be a good start • What about RTP Over QUIC (RoQ)? • Will there be a future for SIP Over QUIC too?
- 37. Thanks! Questions? Comments? Contacts • https://fosstodon.org/@lminiero • https://bsky.app/profile/lminiero.it • https://www.threads.net/@lminier0 • https://www.meetecho.com/blog/ • https://lminiero.it