MongoDB .local London 2019: Streaming Data on the Shoulders of Giants
0 likes285 views
The document discusses the importance of speed and agility in data processing for businesses to remain relevant, emphasizing the shift from traditional batch processing to modern streaming data platforms. It highlights the use of technologies like Apache Kafka and MongoDB for real-time data handling, integration, and analysis. The document also outlines practical use cases for these technologies, such as data synchronization and creating recommendation engines.
![#MDBlocal
Streaming Data on the
Shoulders of Giants
[NETCONOMY]
Hans-Peter Grahsl
https://www.surveymonkey.com/r/KXC6DB6](https://image.slidesharecdn.com/streamingdatamdblondon20190917-190927054858-190928123156/85/MongoDB-local-London-2019-Streaming-Data-on-the-Shoulders-of-Giants-36-320.jpg)
MongoDB .local London 2019: Streaming Data on the Shoulders of Giants
- 1. @ #MDBlocal Hans-Peter Grahsl, NETCONOMY Streaming Data on the Shoulders of Giants hpgrahsl LONDON
- 2. Speed & Agility Among Top Tech Risks For businesses to stay relevant they must deliver value at a breakneck pace and be constantly seeking new sources of value…
- 3. #MDBLocal Managing, Processing & Analyzing Data We all use Data to unlock insights and drive value mongodb.com/local/london
- 4. #MDBLocal Diminishing Value of Data mongodb.com/local/london near real time seconds minutes hours days months VALUEofdatatodecision-making Traditional “batch” business intelligenceTime critical decisions actionable reactive historical preventive/predictive Source: Perishable Insights, Mike Gualtieri, Forrester
- 5. #MDBLocal Historic ETL can be very painful • batch-driven • brittle & error prone • (too) late answers Speed & Agility Antipattern mongodb.com/local/london
- 6. #MDBLocal Alleviate ETL Pain with Streaming Speed & Agility Enabler mongodb.com/local/london • event-centric & stream-oriented • loosely coupled fabric • quick insights & fast answers
- 7. #MDBLocal Streams Processors Connected Apps Architecture of a Modern Data Platform? mongodb.com/local/london Streaming Data Fabric Data Sources Connected Apps Data Sources
- 8. #MDBLocal On the Shoulders of Giants mongodb.com/local/london Kafka®MongoDB®
- 10. #MDBLocal Modern Database Document Model Run Anywhere Distributed & Scalable Resilient & Performant mongodb.com/local/london
- 11. Apache Kafka® Minimum Viable Introduction
- 13. #MDBLocal Streaming Platform ü distributed ü horizontally scalable ü highly fault-tolerant mongodb.com/local/london
- 14. #MDBLocal What is Streaming? “… a type of data processing that is designed with infinite data sets in mind …” – Tyler Akidau mongodb.com/local/london
- 15. "…everything that happens in a company – every customer interaction, every API request, every database change – can be represented as real-time stream that anything else can tap into, process or react to."
- 16. "…Kafka and the whole category of stream processing represents a fundamental paradigm shift in how the digital part of a company is built, how data is used, and how applications are built. This is actually a pretty rare thing…" – Jay Kreps
- 17. #MDBLocal KStreams App Data Sources Data Sinks KSQL App Streams API KSQL Consumer API Connect API App Apps App Apps Connect API Producer API mongodb.com/local/london
- 18. #MDBLocal Kafka APIs in a Nutshell… • Producer & Consumer API à publish-subscribe scenarios • Connect API à streaming data integration scenarios • Streams API & KSQL à code or SQL-based streaming scenarios mongodb.com/local/london
- 19. Kafka® Connect What’s it about?
- 20. #MDBLocal Kafka Connect Basics ANY sink Connect Connect ANY sink ANY à e.g. file systems, data stores, REST endpoints, … mongodb.com/local/london
- 21. #MDBLocal Kafka Connect Basics often about data stores Connect ConnectSOURCE SINK mongodb.com/local/london
- 22. #MDBLocal Kafka Connect Basics or more concretely Connect Connect https://hub.confluent.io à many many more mongodb.com/local/london
- 23. #MDBLocal Kafka Connect Basics or more concretely Connect Connect mongodb.com/local/london https://hub.confluent.io à many many more
- 24. Kafka® Connect How do connectors operate?
- 25. #MDBLocal Kafka Source Connectors Source Connector Converter Serialize S M T 1 … N Single Message Transforms for basic in-flight manipulations … S M T mongodb.com/local/london
- 26. #MDBLocal Kafka Sink Connectors Converter Deserialize Sink Connector S M T 1 … N Single Message Transforms for basic in-flight manipulations … S M T mongodb.com/local/london
- 27. Official Connector Announced at #MDBW19
- 28. #MDBLocal MongoDB Connector for Apache Kafka mongodb.com/local/london Map and persist events from Kafka topics directly to MongoDB Publish data changes from MongoDB into Kafka topics
- 29. #MDBLocal MongoDB Connector for Apache Kafka ü officially supported by MongoDB ü Verified Gold certified by Confluent mongodb.com/local/london Available for Download on the Confluent Hub https://www.conflent.io/hub/mongodb/kafka-connect-mongodb
- 30. Use Cases MongoDB Connector for Apache Kafka
- 31. #MDBLocal Single Customer View for eCommerce MongoDB Sinks Single Source of Truth Source Connectors
- 32. #MDBLocal Data Synchronization between Microservices Service 1 Service N MongoDB Sinks . . .
- 33. #MDBLocal Recommendation Engine for Opinion Mining Surveys & Polls Data MongoDB Source Change Streams Change Streams User Recommendation Engine
- 34. Demo Scenario Let’s see it in action!
- 35. #MDBLocal Producer APIdata generation Stream Processor MongoDB Sink Connector MongoDB Source Connector data serving REST Change Streams device management SSE Demo Scenario ! mongodb.com/local/london
- 36. #MDBlocal Streaming Data on the Shoulders of Giants [NETCONOMY] Hans-Peter Grahsl https://www.surveymonkey.com/r/KXC6DB6
- 37. THANK YOU!