Moving Data Efficiently with Real-Time Streaming
0 likes228 views
The document discusses the WSO2 Streaming Integrator, focusing on its ability to move data efficiently through real-time streaming rather than traditional batch processing methods. It highlights advantages such as lower resource usage, real-time alerts, and a web-based IDE for developing streaming data flows. The presentation includes demonstrations and an overview of future capabilities and features like real-time statistics and enhanced error management.
Moving Data Efficiently with Real-Time Streaming
- 1. WSO2 Streaming Integrator Moving Data Efficiently with Real-Time Streaming Ramindu De Silva, Associate Technical Lead Lasantha Samarakoon, Senior Software Engineer
- 2. Agenda • WSO2 Streaming Integrator • Moving Data Efficiently with Real-Time Streaming • Demo on file processing • Demo on CDC • Q&A 2
- 3. Streaming and WSO2 Streaming Integrator
- 4. WSO2 Streaming Integrator to Connect Anything to Anything.. Streaming Integrator Transform Enrich Cleanse Correlate Aggregate Insights Streaming Messaging Systems Software and Sensors Cloud Databases Files Software Cloud Databases Files Input Stream Event Tables Aggregation Input Stream Result Stream Fetch Data On Demand Via REST API Micro IntegratorTrigger Integrations
- 5. Integrating Streaming Data • Integrating data streams with various destinations • Integrating static sources in real time (as streams)
- 6. Moving Data Efficiently with Real-Time Streaming
- 7. Why not Traditional methods for Moving Data ● Data was processed by batches ○ Large number of resource allocation needed ● Alerts, notifications won’t be in real-time ● Decisions cannot be taken on the process of moving data ○ The moved data will have to be processed again ● Modern systems often include real time streaming data
- 8. Streaming ETL with WSO2 Streaming Integrator
- 9. Advantages on using Real-Time Streaming (a.k.a Streaming ETL) ● Resource usage only depends on data rate. ○ Hence, less resource usage ● Alerts, notifications will happen in near real-time ● Data will be available in realtime for further processing, aggregating, etc.
- 10. Supported Features for Real Time ETL ● Web-based IDE for developing streaming data flows ○ Auto generating stream definitions ○ Drag and drop functionality for ease of developing siddhi apps ● Powered by Siddhi.io (recognized by CNCF as a streaming and messaging system) ● 60+ connectors ○ Readily available connectors to connect with major systems ● High resilience ● Easily deployable 10
- 11. What’s coming next on SI? • Real-time statistics using Grafana • Streaming ETL Task Wizard • Error management, event re-play 11
- 12. Demo
- 13. Scenario 1
- 14. Scenario 2 14
- 15. Questions?