Insertable Streams and E2EE @ ClueCon2020

1. Insertable Streams and E2EE in WebRTC: a Janus story Lorenzo Miniero ClueCon – Chicago, IL, USA (kinda!) August 5th 2020

2. Who am I? Lorenzo Miniero • Ph.D @ UniNA • Chairman @ Meetecho • Main author of Janus® Contacts and info • lorenzo@meetecho.com • https://twitter.com/elminiero • https://www.slideshare.net/LorenzoMiniero • https://soundcloud.com/lminiero

3. Just a few words on Meetecho • Co-founded in 2009 as an academic spin-off • University research efforts brought to the market • Completely independent from the University • Focus on real-time multimedia applications • Strong perspective on standardization and open source • Several activities • Consulting services • Commercial support and Janus licenses • Streaming of live events (IETF, ACM, etc.) • Proudly brewed in sunny Napoli, Italy

4. Home Sweet Home!

5. WebRTC reference architecture: peer-to-peer

6. WebRTC reference architecture: peer-to-peer

7. Involving a server as a peer

8. Involving a server as a peer

9. Breaking end-to-end encryption

10. Many reasons why that may be needed • Server being a “MITM” is often a feature, actually • It may need to transcode media • An MCU even more so! (e.g., FreeSwitch) • Recordings typically need access to unencrypted media too • Even when just relaying, access to payload may be sometimes required • e.g., Simulcast or SVC • Codec-speciﬁc info (e.g., temporal layer) may be in the payload • RTP headers would still need to be editable Can you get true E2EE even through a server? • Short answer: YES! • Slightly longer answer: it’s not that easy, though...

14. An excellent talk on the challenges of E2EE! https://www.youtube.com/watch?v=a0vhHmONWlw

15. Privacy Enhanced RTP Conferencing (PERC) https://datatracker.ietf.org/wg/perc/about/

16. Not very popular among WebRTC developers, though... https://mailarchive.ietf.org/arch/msg/perc/x1HjohPL6ISuoNj7eE6HEre7O1E/

17. A simpler approach: PERC “Lite”

18. PERC Lite and Janus https://www.meetecho.com/blog/meetecho-and-cosmo-strike-again-perc-lite-integration-in-janus/

19. PERC Lite and Janus https://www.meetecho.com/blog/meetecho-and-cosmo-strike-again-perc-lite-integration-in-janus/

20. Replaying encrypted recordings (“DRM”)

21. Replaying encrypted recordings (“DRM”)

22. Enter Insertable Streams! • New browser API that allows custom processing on WebRTC frames • Key use case −→ E2EE! • Works with encoded frames, not individual packets • [OUT] audio/video frames after they’ve been encoded, but before RTP packetization • [IN] audio/video frames before they’ve been decoded, but after RTP de-packetization • Working with frames makes it protocol agnostic • Today it’s RTP, tomorrow it might be QUIC Some useful reads • https://www.meetecho.com/blog/janus-e2ee-sframe/ • https://webrtcbydralex.com/index.php/2020/03/30/ • https://webrtchacks.com/true-end-to-end-encryption-with-webrtc-insertable-streams/

26. Insertable Streams workﬂow

27. Insertable Streams workﬂow

28. A few challenges with this approach • Encrypted payload may make life harder for SFUs • Useful info usually available in the payload • e.g., how to detect if we got a keyframe? • A few different solutions to this • Keep the first bytes unencrypted? (where that info is) • Ad-hoc RTP extensions for metadata (spec on its way!) • Encrypting the payload may confuse the RTP (de)packetizer • Each video codec has different rules • e.g., H.264 uses NALs, which wouldn’t be visible anymore • Solution more tricky to find, here • Keep info used for splitting unencrypted? (codec specific) • Even better, use a global RTP packetization for all codecs (spec coming!)

32. Enabling Insertable Streams pc = new RTCPeerConnection({ encodedInsertableStreams: true, forceEncodedAudioInsertableStreams: true, forceEncodedVideoInsertableStreams: true });

33. Adding a Sender Trasform senderTransform = new TransformStream({ start() { // Called on startup. }, transform(chunk, controller) { // Frame encoded, edit payload and return it [..] controller.enqueue(chunk); }, flush() { // Called when the stream is about to be closed } }); sender = pc.addTrack(track, stream) senderStreams = sender.createEncodedVideoStreams(); senderStreams.readableStream .pipeThrough(senderTransform) .pipeTo(senderStreams.writableStream);

34. Adding a Receiver Trasform as well receiverTransform = new TransformStream({ start() { // Called on startup. }, transform(chunk, controller) { // Received frame, edit payload and return it [..] controller.enqueue(chunk); }, flush() { // Called when the stream is about to be closed } }); pc.ontrack = function(event) { [..] receiverStreams = event.receiver.createEncodedVideoStreams(); receiverStreams.readableStream .pipeThrough(receiverTransform) .pipeTo(receiverStreams.writableStream); }

35. Integrated in janus.js a few weeks ago echotest.createOffer({ media: { audio: true, video: true }, senderTransforms: { audio: new TransformStream({ .. }), video: new TransformStream({ .. }) }, receiverTransforms: { audio: new TransformStream({ .. }), video: new TransformStream({ .. }) }, [..] success: function(jsep) { // Send offer to Janus } }); https://github.com/meetecho/janus-gateway/pull/2074

36. Testing basic E2EE in the Janus demos https://janus.conf.meetecho.com/e2etest

37. Testing basic E2EE in the Janus demos https://janus.conf.meetecho.com/e2etest

38. A step forward: Secure Frames (SFrame) • Co-developed by CoSMo and Google • Used in Google Duo for E2EE for a long time already • https://cosmosoftware.io/secure-frames-sframes/ • https://www.gstatic.com/duo/papers/duo_e2ee.pdf/ • Recently brought to the IETF as well (presented in DISPATCH @ IETF108) • https://tools.ietf.org/html/draft-omara-sframe-00 • https://mailarchive.ietf.org/arch/browse/sframe/ • Learned lessons from previous E2EE attempts • Minimizes overhead and impact on servers and clients • Works of frames, not packets, so transport agnostic • Interaction with KMS done out of band, and KMS agnostic (e.g., Signal or MLS)

41. A step forward: Secure Frames (SFrame)

42. A step forward: Secure Frames (SFrame)

43. SFrame.js: an open source SFrame library • Pure javascript library implementing SFrame using webcrypto • Developed by Sergio Garcia Murillo • https://github.com/medooze/sframe • Described in detail in a blog post and in the CommCon presentation • https://medium.com/@medooze/sframe-js-end-to-end-encryption-for-webrtc- f9a83a997d6d • https://www.youtube.com/watch?v=a0vhHmONWlw • Basically provides everything except the Key Management System (KMS) • Which makes sense, that should be up to you anyway! Helped Sergio testing it while integrating it in janus.js https://www.meetecho.com/blog/janus-e2ee-sframe/

47. Integrating SFrame.js in janus.js import {Janus} from ’./janus.js’; import {SFrame} from ’./sframe/Client.js’; ... echotest.createOffer({ media: { audio: true, video: true }, sframe: { outgoingId: 0, incomingId: 0, sharedKey: cryptoKey, // Generated previously keyPair: keyPair // Generated previously }, [..] success: function(jsep) { // Send offer to Janus } });

48. Integrating SFrame.js in janus.js if(callbacks.sframe) { Janus.log("Using SFrame to encrypt media end-to-end:", callbacks.sframe); config.sframe = callbacks.sframe; config.sframeClient = await SFrame.createClient(config.sframe.outgoingId, {}); // Sender part await config.sframeClient.setSenderEncryptionKey(config.sframe.shared); if(config.sframe.keyPair && config.sframe.keyPair.privateKey) { await config.sframeClient.setSenderSigningKey(config.sframe.keyPair.privateKey); } // Receiver part if(config.sframe.incomingId !== undefined && config.sframe.incomingId !== null) { await config.sframeClient.addReceiver(config.sframe.incomingId); await config.sframeClient.setReceiverEncryptionKey(config.sframe.incomingId, callbacks.sframe.shared); if(config.sframe.keyPair && config.sframe.keyPair.publicKey) { await config.sframeClient.setReceiverVerifyKey(config.sframe.incomingId, config.sframe.keyPair.publicKey); } } }

49. Integrating SFrame.js in janus.js // Have SFrame.js configure the Sender Transform var sender = config.pc.addTrack(track, stream); config.sframeClient.encrypt(sender.track.id, sender); ... // Have SFrame.js configure the Receiver Transform config.pc.ontrack = function(event) { [..] config.sframeClient.decrypt(event.track.id, event.receiver); [..] }

50. Testing SFrame with Janus https://www.meetecho.com/blog/janus-e2ee-sframe/

51. Testing SFrame with Janus https://www.meetecho.com/blog/janus-e2ee-sframe/

52. Thanks! Questions? Comments? Get in touch! • https://twitter.com/elminiero • https://twitter.com/meetecho • https://www.meetecho.com

Insertable Streams and E2EE @ ClueCon2020

More Related Content

What's hot (20)

Similar to Insertable Streams and E2EE @ ClueCon2020 (20)

More from Lorenzo Miniero (19)

Recently uploaded (20)

Insertable Streams and E2EE @ ClueCon2020