DataOps on Streaming Data: From Kafka to InfluxDB via Kubernetes Native Flows by Christos Kotsis, Reliability Engineer | Lenses.io
0 likes576 views
The document discusses DataOps in the context of streaming data using Kafka and InfluxDB within Kubernetes, highlighting the challenges of managing large volumes of IoT data by 2025. It outlines how to implement a streaming architecture for real-time insights and data flows, emphasizing the integration of multiple microservices and the use of Lenses for data manipulation and monitoring. A demo featuring Lenses Box is also mentioned, showcasing its capabilities for developers.
DataOps on Streaming Data: From Kafka to InfluxDB via Kubernetes Native Flows by Christos Kotsis, Reliability Engineer | Lenses.io
- 1. DataOps on streaming data Kafka to InfluxDb via Kubernetes Native Flows - IoT demo Chris Kotsis lenses.io
- 2. #About me Reliability Engineer at Lenses.io Data Flows on Kubernetes Kafka Cloud Native techs DataOps Expert
- 3. #About lenses.io Enterprise DataOps over Streaming Data
- 4. #About lenses.io Enterprise DataOps over Streaming Data
- 5. #About lenses.io Enterprise DataOps over Streaming Data all Distributions - on Cloud - on Premises
- 6. #DataOps Embrace Collaboration Eliminate Friction, turn data to Value Faster
- 7. #DataOps Personas people who ● Collect and Prepare the data ● Analyse the data ● Use the findings for business value
- 8. #DataOps IoT Challenges Enormous Volumes of Data “500 billion connected devices by 2025”
- 9. #DataOps IoT Challenges Enormous Volumes of Data “500 billion connected devices by 2025” Its Management and Analysis will become harder and continue to break traditional tools
- 10. The IoT Data Flow
- 11. How to implement a Streaming Architecture for IoT Time-Series data and Real-Time insights?
- 12. Infrastructure Layer Where the Data lives Self-service Data Access, Multi-tenancy, Security, Governance to Accessibility & Visibility for ALL
- 14. IoT & kafka High Volumes, N devices & irregular intervals Real Time Analytics & Microservices Multiple sources of data & long term storage An open source streaming framework with messaging semantics where records are key-value pairs ➔ Unlimited streams of data, ➔ Producers & Consumers (pub/sub) ➔ Processing and analysing data in motion ➔ Connect API to move data with pluggable reusable components
- 15. The flow implementation ➔ Sensor produces data to Kafka ➔ Create my Data Flows: ◆ Kafka - to - Kafka processing ◆ Route to InfluxDB
- 16. Application Layer Multiple Distributed Microservices for complex Data Flows
- 18. How Lenses integrates with the Application Layer?
- 19. Lenses SQL Processors ● Simply Filter, Enrich, Split & Bind your data ● Manipulate Live Streams of Data ● Scalable: Cloud / Kubernetes Native PROCESS
- 20. Connect Kafka to InfluxDB Real Time Ingestion Distributed Fault tolerance Scalability Error Handling Monitoring & Alerting Governance & Security Easy Data Manipulation ... INGEST
- 21. Lenses Connectors: Binding external sources and sinks is a few clicks away
- 22. InfluxDB sink connector by Lenses.io Kafka to InfluxDB No code required CLI/UI/API/Monitor SQL Support Multiple inserts supported AVRO & JSON Support Supported Features: MEASUREMENTS (SQL: INSERT TO) TIMESTAMPS (SQL: WITHTIMESTAMP) TAGS (SQL: WITHTAG) DURATION
- 24. The uncontrolled “Chaos” of modern data streams
- 25. Lenses Topology, one view to bind them all
- 28. Demo
- 29. What it takes to demo this architecture
- 30. Lenses Box ● FREE for Developers ● Single Broker setup ● All ecosystem services ● 25+ Connectors ● Synthetic Data generators ● Live examples ● Lenses intuitive UI ● The powerful Lenses SQL ● Works on your Laptop ● Works on Cloudhttps://lenses.io/lenses-box
- 31. Thank you and <3