FIWARE Tech Summit - Stream Processing with Kurento Media Server
1 like857 views
Kurento Media Server is an open-source platform designed to simplify the creation of streaming applications by processing audio/video streams and supporting WebRTC communications. It utilizes a modular architecture with endpoints for data flow and filters for processing, allowing for complex media handling and integration with various external data sources. The server can be controlled via an RPC API, and it supports various protocols and endpoints tailored for specific needs such as video playback and recording.
FIWARE Tech Summit - Stream Processing with Kurento Media Server
- 1. Stream processing with Kurento Media Server Juan Navarro Moreno Kurento Software Developer jnavarro@naevatec.com github.com/j1elo www.kurento.org
- 2. Overview
- 3. 3 Overview – What is Kurento? ● A server that simplifies creation of streaming applications. ● Focused on processing of Audio/Video media streams. ● Supporting WebRTC communications. ● Based on modules that perform input → processing → output. Kurento is an Open Source project. Hosted on GitHub: https://github.com/Kurento
- 4. 4 Overview – What can be done with Kurento? ● Abstracts compatibility issues between sender and receiver. ● Manipulates or redistributes the streams. ● Extracts information from the streams. Input stream Kurento Media Server Output stream Product-specific data Eg.: FIWARE data-store
- 5. 5 Overview – What can be done with Kurento? Examples: ● Abstraction of incompatible video codecs ● Multi-point distribution of media ● Processing / Storing media
- 6. 6 Overview – What can be done with Kurento?
- 7. 7 Overview – How does Kurento work? A server with two main components: ● Endpoints: where data flows In/Out. ● Filters: composable modules where data is processed or transformed. Media stream Src Sink Media stream Input Endpoint Output Endpoint
- 8. 8 Overview – How does Kurento work? Endpoints and Filters are linked together, mix-and-matched to form a Pipeline. Input Endpoint Src Sink Output Endpoint Src Sink Src Sink Src Sink
- 9. 9 Overview – How does Kurento work? The server is controlled with an RPC API. ● Client applications manipulate Endpoints and Filters through this API. ● Ready-made client SDKs for Java, Node.js and in-browser JavaScript. Components of a Real-World application: ● Kurento Media Server ● Client Application – usually a server too. ● User Interface – common case is a webpage.
- 10. 10 Overview – How does Kurento work?
- 12. 12 Getting technical – Component overview Kurento Media Server (aka. “KMS”) ● Core: Basic functionality and RPC API. ● Elements: Endpoints used for input and output of streams. ● Filters: Implementation of all stream- processing modules.
- 13. 13 Getting technical – KMS Elements Protocols and Codecs Media Repository WebRtcEndpoint RtpEndpoint HttpEndpoint PlayerEndpoint RecorderEndpoint
- 14. 14 Getting technical – KMS Elements Different Endpoints for different needs: ● WebRtcEndpoint – Full support of WebRTC standard. • Currently compatible with Chrome and Firefox (Safari and Edge are work-in-progress). • All WebRTC lingo: SDP, (Trickle-)ICE, STUN, TURN, Google REMB. ● RtpEndpoint – For RTP and SRTP streams. • Supports port auto-discovery as an alternative to ICE. ● HttpEndpoint – Accepts GET/POST requests (eg. file uploading to KMS).
- 15. 15 Getting technical – KMS Elements Special Endpoints: ● PlayerEndpoint – Retrieves content from either of the file system, HTTP servers, or RTSP sources. Input-only. ● RecorderEndpoint – Redirects streams to storage. Output-only.
- 16. 16 Getting technical – KMS Filters Process or transform data as it flows through filters. ● Could be a simple transformation. Eg: Convert video to black and white. ● Could be a complex task, involving external libraries. Eg: Apply Computer Vision algorithms and extract features from the video. ● Imagination is the limit.
- 17. 17 Getting technical – KMS Filters Computer Vision Generic filter Zbar FaceDetector PlateDetector GStreamerFilter
- 18. 18 Getting technical – Pipelines ● All elements get created and linked through specific RPC commands from the Client Application. ● As seen before, complex topologies are possible.
- 19. 19 Getting technical – Pipelines SinkSink SRCSRC SinkSink SRCSRCSinkSink SinkSinkSRCSRC WebRtcEndpoint AR Filter RecorderEndpoint HttpGetEndpoint RtpEndpoint MP4 fle stored in media repository Web application using HTML5 <video> tag RTP full duplex client video phone WebRTC full duplex client video application SinkSink
- 20. 20 Getting technical – Client Application ● Implements or includes an RPC client. ● Orchestrates the creation of the Pipeline, with Elements and Filters. ● WebRtcEndpoint follows the standard way of configuration via SDP Negotiation (aka. SDP Offer/Answer Model). • The Client Application is in charge of passing around ICE Candidates. ● RtpEndpoint also uses SDP for configuration, but no ICE. ● Other Endpoints have custom RPC methods for configuration.
- 21. 21 Getting technical – Client Application ● Provides access to external sources such as FIWARE data-stores. ● Proceeds with the rest of the business logic. ● Kurento provides a FIWARE integration package for Java applications. ● Also there are multiple example applications made available.
- 22. 22 Getting technical – Client Application Application Server Application Server Media Server Media ServerClientClient Media Negotiation phase 11 Media channel preparation phase (optional) 22 Media exchange phase 33 Specifc app logic Specifc pipeline building Pipeline managed media
- 23. A closer look
- 24. 24 A closer look – GStreamer media library Kurento is powered under the hood by the GStreamer project: https://gstreamer.freedesktop.org GStreamer is in charge of all media handling: ● Decoding / encoding of input / output video & audio. ● Establishment of communication streams (eg. RTP Sessions). ● Special interest: Agnostic bin.
- 25. 25 A closer look – Agnostic bin Kurento abstracts codec compatibility issues between Endpoints. ● Achieved through an “invisible” intermediate filter – ‘agnosticbin’. ● Applies on-the-fly transcoding iff the codecs are not compatible. Media Element Media Element Agnostic media adapter, “hidden” behind every connection Agnostic media adapter, “hidden” behind every connection H.264VP8 Src Sink Src Sink
- 26. 26 A closer look – SDP Offer/Answer ● The standard signaling format for WebRTC. ● Used to configure codecs & advanced settings in WebRtcEndpoint and RtpEndpoint. ● SDP is a mess and we all hate it (if you don’t yet, you should). (The problem is, the IETF RTCWEB working group was/is dominated by players from the SIP world who wanted to re-use their existing code.) See: https://webrtchacks.com/sdp-anatomy/
- 27. 27 A closer look – SDP Offer/Answer v=0 o=- 0 0 IN IP4 127.0.0.1 s=- c=IN IP4 127.0.0.1 t=0 0 m=audio 9 RTP/AVPF 96 a=rtpmap:96 opus/48000/2 a=sendonly a=direction:active a=ssrc:445566 cname:user@example.com m=video 9 RTP/AVPF 103 a=rtpmap:103 H264/90000 a=sendonly a=direction:active a=ssrc:112233 cname:user@example.com v=0 o=- 372 372 IN IP4 192.168.1.15 s=Kurento Media Server c=IN IP4 192.168.1.15 t=0 0 m=audio 41654 RTP/AVPF 96 a=rtpmap:96 opus/48000/2 a=recvonly a=direction:passive a=ssrc:3224 cname:user225@host-9b2 m=video 61134 RTP/AVPF 103 a=rtpmap:103 H264/90000 a=recvonly a=direction:passive a=ssrc:2717 cname:user225@host-9b2 SDP Offer to RtpEndpoint: SDP Answer from RtpEndpoint:
- 29. 29 A closer look – Application example: RTP Player Kurento RTP Player is a new application example, not yet published. ● Receives an audio/video stream. ● Redirects it and sends to a browser through WebRTC. ● The RTP can originate from any source. A sample tool is provided. External RTP producer
- 30. 30 A closer look – Application example: RTP Player void start(String browserSdpOffer) { // Create and link Endpoints MediaPipeline pipeline = kurento.createMediaPipeline(); RtpEndpoint rtpEndpoint = new RtpEndpoint.Builder(pipeline).build(); WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build(); rtpEndpoint.connect(webRtcEndpoint); // Configure the RtpEndpoint String fakeSdpOffer = "..."; String kmsSdpAnswer = rtpEndpoint.processOffer(fakeSdpOffer); String browserSdpAnswer = webRtcEndpoint.processOffer(browserSdpOffer); webRtcEndpoint.gatherCandidates(); }
- 31. 31 A closer look – Application example: Magic Mirror ● Receives a video stream from a browser. ● Applies Computer Vision to detect a face and overlays a picture. ● Sends back the modified video to the same Endpoint it came from.
- 32. 32 A closer look – Application example: Magic Mirror void start() { // Create and link Endpoints, Filters MediaPipeline pipeline = kurento.createMediaPipeline(); WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build(); FaceOverlayFilter faceOverlayFilter = new FaceOverlayFilter.Builder(pipeline).build(); faceOverlayFilter.setOverlayedImage("mario-hat.png"); webRtcEndpoint.connect(faceOverlayFilter); faceOverlayFilter.connect(webRtcEndpoint); webRtcEndpoint.gatherCandidates(); }
- 33. Closing
- 34. Thank you! http://fiware.org Follow @FIWARE on Twitter Juan Navarro Moreno Kurento Software Developer jnavarro@naevatec.com github.com/j1elo www.kurento.org