WebRTC Standards & Implementation Q&A - getDisplayMedia 1.0
1 like1,004 views
This document summarizes a webinar about WebRTC standards hosted by Jan-Ivar Bruaroey of Mozilla. The webinar discussed the getDisplayMedia specification for screen sharing, constraints for downscaling shared screens, and security considerations for sharing full screens and browser tabs. It also covered the history of screen sharing in browsers, implementation status across browsers, and questions about supporting audio with screen sharing and handling requests from iframes.
![Downscaling with track.applyConstraints()
Early 2018: constraints on gDM forbidden. Mustn’t influence user-selection. Instead:
const stream = video.srcObject = await navigator.getDisplayMedia({video: true});
// Got it! Now downscale to not be huge
await stream.getVideoTracks()[0].applyConstraints({width: 320, height: 200});](https://image.slidesharecdn.com/webrtcstandardslivesession28-180924174819/85/WebRTC-Standards-Implementation-Q-A-getDisplayMedia-1-0-11-320.jpg)
![Downscaling with getDisplayMedia()
Of course you can still use applyConstraints and the rest of the API:
stream = await navigator.getDisplayMedia({video: {width: 320, height: 200}});
const [track] = stream.getVideoTracks();
const halve = async () => {
const {width, height} = track.getSettings();
await track.applyConstraints({width: width / 2, height: height / 2});
}
halve();](https://image.slidesharecdn.com/webrtcstandardslivesession28-180924174819/85/WebRTC-Standards-Implementation-Q-A-getDisplayMedia-1-0-13-320.jpg)
![Constraints in getDisplayMedia() 1.0
Some differences from camera. Constraints are for downscaling, not discovery.
Therefore, the following constraint keywords are disallowed:
await navigator.getDisplayMedia({video: {width: {min: 320}}); // TypeError
await navigator.getDisplayMedia({video: {width: {exact: 320}}); // TypeError
await navigator.getDisplayMedia({video: {advanced: [{width: 320}]}); // TypeError
But the ideal and max constraints are allowed (more on this later).
This eliminates the risk of post-prompt OverconstrainedError.
Again, getDisplayMedia() is not for discovery.](https://image.slidesharecdn.com/webrtcstandardslivesession28-180924174819/85/WebRTC-Standards-Implementation-Q-A-getDisplayMedia-1-0-14-320.jpg)
![How to detect Retina displays
These render at 2x size. Unexpected? Detectable? The plan is to downscale these to ½ by default to
render at the expected same scale. This also makes it detectable:
stream = await navigator.getDisplayMedia({video: true}); // unconstrained
const [track] = stream.getVideoTracks();
const isRetina = track.getSettings().resizeMode != ‘none’; // already downscaled?
// I have 1 gigabit upload, no defaults for me, I want the whole enchilada
await track.applyConstraints({width: 9999, height: 9999}); // no worry no upscale](https://image.slidesharecdn.com/webrtcstandardslivesession28-180924174819/85/WebRTC-Standards-Implementation-Q-A-getDisplayMedia-1-0-17-320.jpg)
![How to detect Retina displays (alternative)
Another approach that could someday work with other constraints applied:
stream = await navigator.getDisplayMedia({video: myConstraints}); // constrained
const [track] = stream.getVideoTracks();
const {width, height} = track.getSettings();
const cap = track.getCapabilities();
const isRetina = width < cap.width.max || height < cap.height.max;
await track.applyConstraints({width: cap.width.max, height: cap.height.max});
But this depends on expectations of getCapabilities() for windows that change size.](https://image.slidesharecdn.com/webrtcstandardslivesession28-180924174819/85/WebRTC-Standards-Implementation-Q-A-getDisplayMedia-1-0-18-320.jpg)
WebRTC Standards & Implementation Q&A - getDisplayMedia 1.0
- 1. WebRTC Standards Webinar & Q&A Amir Zmora TheNewDialTone Jan-Ivar Bruaroey Mozilla
- 2. Watch video recording of this session http://ccst.io/e/webrtcstandards28
- 3. Jan-Ivar Bruaroey jib on irc.mozilla.org #media @jibrewery on Twitter On the Firefox Media team, working to implement WebRTC: getUserMedia(), getDisplayMedia(), RTCPeerConnection(). W3C editor of mediacapture-main and webrtc-pc. Contributor to mediacapture-screen-share and other specs. I’m currently implementing getDisplayMedia() in Firefox, and contributing to finish off its specification.
- 4. Session sponsored by WebRTC.ventures is a custom design and development shop dedicated to building WebRTC based applications for web and mobile. We have built end-to-end broadcast solutions for events and entertainment clients, telehealth solutions for multiple clients, live support tools, as well as communication tools for a variety of other applications. WebRTC.ventures is a recognized development partner of TokBox and has also built native WebRTC solutions
- 5. We use CrowdCast….It’s WebRTC
- 7. Outline 1. Introduction to getDisplayMedia() 2. Constraints for downscaling 3. Finalizing the spec. 4. Security concerns. Full-screen/browser tab-sharing is “scary”. 5. Permissions 6. Iframe permissions (Feature Policy)
- 8. History Since 2013-14,Chrome and Firefox have supported screen capture through incompatible proprietary and non-standard extensions of getUserMedia(). Firefox with the mediaSource constraint (dropped white-list in 52). Chrome requires an extension be installed; with chromeMediaSource constraint. Progress lingered on standard specification, until recently. Chrome to drop inline installation of extensions soon; new friction injects urgency.
- 9. W3C Screen Capture specification Spec doc: mediacapture-screen-share. Standard way to let websites request the end-user to share their display of choice. Nearing completion. Implementations available: • Edge: Yes (early 2018 draft) • Chrome: In development (yes behind pref in Canary, early 2018 draft) • Safari: In development • Firefox: In development. Targeting 64 or 65. • Adapter.js: Yes (early 2018 draft), w/setup in Firefox/Chrome w/Chrome plugin.
- 10. Screen-sharing with getDisplayMedia() Early 2018: video.srcObject = await navigator.getDisplayMedia({video: true});
- 11. Downscaling with track.applyConstraints() Early 2018: constraints on gDM forbidden. Mustn’t influence user-selection. Instead: const stream = video.srcObject = await navigator.getDisplayMedia({video: true}); // Got it! Now downscale to not be huge await stream.getVideoTracks()[0].applyConstraints({width: 320, height: 200});
- 12. Downscaling with getDisplayMedia() Late 2018: constraints allowed, but browsers MUST apply only after user selection. stream = await navigator.getDisplayMedia({video: {width: 320, height: 200}}); // Got it! And it’s downscaled already. Simpler. But what about OverconstrainedError? Would stink post-prompt (more later).
- 13. Downscaling with getDisplayMedia() Of course you can still use applyConstraints and the rest of the API: stream = await navigator.getDisplayMedia({video: {width: 320, height: 200}}); const [track] = stream.getVideoTracks(); const halve = async () => { const {width, height} = track.getSettings(); await track.applyConstraints({width: width / 2, height: height / 2}); } halve();
- 14. Constraints in getDisplayMedia() 1.0 Some differences from camera. Constraints are for downscaling, not discovery. Therefore, the following constraint keywords are disallowed: await navigator.getDisplayMedia({video: {width: {min: 320}}); // TypeError await navigator.getDisplayMedia({video: {width: {exact: 320}}); // TypeError await navigator.getDisplayMedia({video: {advanced: [{width: 320}]}); // TypeError But the ideal and max constraints are allowed (more on this later). This eliminates the risk of post-prompt OverconstrainedError. Again, getDisplayMedia() is not for discovery.
- 15. Captured windows may have surprising aspect. await gDM({video: {width: 320, height: 200}}); Downscaling always preserves aspect, never crops Best effort based on fitness distance. There’s some question whether fitness distance alone gives good results. Firefox uses-area-based approximation when aspects differ. Allowed to go out of bounds. ideal
- 16. Why? End-user might resize window extremely tall or wide during live capture. await gDM({video: {width: {ideal:320, max:320}, height: {ideal:200, max:340}}}); Use max constraint to define outer bounds Having two forms of constraints allows room for aspect changes within limits. If end-user makes window any taller, it’s always downscaled to fit within outer bounds. ideal max User resizes live. Video stays inside bounds
- 17. How to detect Retina displays These render at 2x size. Unexpected? Detectable? The plan is to downscale these to ½ by default to render at the expected same scale. This also makes it detectable: stream = await navigator.getDisplayMedia({video: true}); // unconstrained const [track] = stream.getVideoTracks(); const isRetina = track.getSettings().resizeMode != ‘none’; // already downscaled? // I have 1 gigabit upload, no defaults for me, I want the whole enchilada await track.applyConstraints({width: 9999, height: 9999}); // no worry no upscale
- 18. How to detect Retina displays (alternative) Another approach that could someday work with other constraints applied: stream = await navigator.getDisplayMedia({video: myConstraints}); // constrained const [track] = stream.getVideoTracks(); const {width, height} = track.getSettings(); const cap = track.getCapabilities(); const isRetina = width < cap.width.max || height < cap.height.max; await track.applyConstraints({width: cap.width.max, height: cap.height.max}); But this depends on expectations of getCapabilities() for windows that change size.
- 19. Adapter.js polyfill Works in Firefox (and Edge of course): https://jsfiddle.net/jib1/q75yb8pf/ adapter.browserShim.shimGetDisplayMedia(window, ‘window’); // or ‘screen’ const stream = video.srcObject = await navigator.getDisplayMedia({video: true}); We will combine “screen” and “window” choices in Firefox Nightly soon. Constraint support demo: https://jsfiddle.net/jib1/dkt5wexz/ Works with Firefox and Canary (Edge does not allow constraints on getDisplayMedia) Shim for Chrome plugin is more involved. Google “adapter.js getDisplayMedia”.
- 20. Finalizing the spec: How to ask for audio? This isn’t in the spec yet. But Chrome apparently has this ability, but only for browser tabs. Maybe: await navigator.getDisplayMedia({video: true, audio: true}); Should this fail immediately with OverconstrainedError if audio is unsupported, or just return a video track with no audio? Some sources may have audio, others may not. How to handle that? Silence? What about: await navigator.getDisplayMedia({audio: true}); ?
- 21. Security concerns Full-screen/browser sharing is scary! Not just passive threats. If a web surface under site control is captured, that website has keys to the car, and can iframe-navigate as the logged-in user effectively. Sidesteps cross-origin protections. Firefox warns, but hard to explain ! Google “share screen trust” for more
- 22. SecureContext getDisplayMedia() will be https only, pending w3c/mediacapture-screen-share#77 if (!(‘getDisplayMedia’ in navigator)) { // http or unsupported in browser return; }
- 23. Persistent “granted” permissions are forbidden. Persistent “denied” permissions are allowed. Prompts user before every access. Website cannot narrow user choice, (to avoid steering them toward scary sources). Spec recommends elevated permission for scary sources. Question: How to prompt for video + audio? Permissions
- 24. How to tackle requests from iframes? Users don’t read URLs in prompts! Too hard. wicg.github.io/feature-policy to the rescue. Says: All grants to top-level domain. It’s up to top-level domain to delegate to iframes. Similar to camera and microphone: <iframe allow="camera; microphone; display"> Disallow by default. navigator.getDisplayMedia remains but throws SecurityError Should “display” cover both video and related audio? Iframe permissions (Feature Policy) Cross-origin domain! “screen” or “display”?
- 25. Questions? Use-cases? We’re interested in hearing of screen-sharing use-cases we haven’t thought of. For instance, appear.in uses screen-sharing to record a WebRTC meeting. Audio use-cases? Any not covered by web-audio? Audio-only? Do security measures seem sufficient? Expect any different in Private Browsing?
- 26. ?
- 28. WebRTC Standards Webinar & Q&A Amir Zmora TheNewDialTone Jan-Ivar Bruaroey Mozilla