Building a Self-Service Stream Processing Portal: How And Why
0 likes280 views
The document discusses building a self-service stream processing portal, highlighting the architecture and features such as real-time data manipulation and integration. It outlines user needs for checking and managing real-time data while emphasizing automation, data governance, and storage. The document also mentions tools like h-stream, ksqldb, and the importance of user self-service in stream processing.
Building a Self-Service Stream Processing Portal: How And Why
- 1. Building a Self-Service Stream Processing Portal : How And Why Jaeseung Park
- 2. 270,000 EPS 35 GB / sec 0 500 1000 1500 2000 2500 3000 0 50000 100000 150000 200000 250000 300000 2019 2020 2021 2022 2023 Event Equipment
- 5. Data Hub Operation Automation Data Governance Storage Ingestion Delivery Data Pipeline Hub DB Data Mart Processing Streaming Batch • CDC Pipeline • Event Pipeline Hub Queue h-STREAM Pub/Sub h-CONNECT ETL Single Access Point Target … Fab Fab Source Data Lakehouse Analysis Biz. App … Fab Hub Fab Hub Legacy Real-time Real-time support Real-time Real-time Real-time Real-time Real-time Real-time Data Platform
- 6. Data Hub Operation Automation Data Governance Storage Ingestion Delivery Data Pipeline Hub DB Data Mart Processing Streaming Batch • CDC Pipeline • Event Pipeline Hub Queue h-STREAM Pub/Sub h-CONNECT ETL Single Access Point Target … Fab Fab Source Data Lakehouse Analysis Biz. App … Fab Hub Fab Hub Legacy Real-time Real-time support Real-time Real-time Real-time Real-time Real-time Real-time Data Platform
- 7. Data Hub Operation Automation Data Governance Storage Ingestion Delivery Data Pipeline Hub DB Data Mart Processing Streaming Batch • CDC Pipeline • Event Pipeline Hub Queue h-STREAM Pub/Sub h-CONNECT ETL Single Access Point Target … Fab Fab Source Data Lakehouse Analysis Biz. App … Fab Hub Fab Hub Legacy Real-time Real-time support Real-time Real-time Real-time Real-time Real-time Real-time Data Platform
- 8. Data Hub Operation Automation Data Governance Storage Ingestion Delivery Data Pipeline Hub DB Data Mart Processing Streaming Batch • CDC Pipeline • Event Pipeline Hub Queue h-STREAM Pub/Sub h-CONNECT ETL Single Access Point Target … Fab Fab Source Data Lakehouse Analysis Biz. App … Fab Hub Fab Hub Legacy Real-time Real-time support Real-time Real-time Real-time Real-time Real-time Real-time Data Platform
- 9. Data Hub Operation Automation Data Governance Storage Ingestion Delivery Data Pipeline Hub DB Data Mart Processing Streaming Batch • CDC Pipeline • Event Pipeline Hub Queue h-STREAM Pub/Sub h-CONNECT ETL Single Access Point Target … Fab Fab Source Data Lakehouse Analysis Biz. App … Fab Hub Fab Hub Legacy Real-time Real-time support Real-time Real-time Real-time Real-time Real-time Real-time Data Platform
- 10. Why User "I want to easily check real-time data." "I want to manipulatereal-time data to create my own." "I lackseparate computing resources, so I can't do Stream Processing." "I want to integrate stable Stream Processing logic into the operating system." "I would like to receiveadvice on Stream Processing know-how."
- 11. h-STREAM Architecture ksqlDB Deploy Replicate API Pub Dev (Sand-box) Ops Sub Brower Pub/Sub Pub/Sub API Master DB Kafka Connector Event h-STREAM Portal ksqlDB ksqlDB User 1 2 3 4 5
- 12. h-STREAM Architecture ksqlDB Deploy Replicate API Pub Dev (Sand-box) Ops Sub Brower Pub/Sub Pub/Sub API Master DB Kafka Connector Event h-STREAM Portal ksqlDB ksqlDB User 1 2 3 4 5
- 13. h-STREAM Architecture ksqlDB Deploy Replicate API Pub Dev (Sand-box) Ops Sub Brower Pub/Sub Pub/Sub API Master DB Kafka Connector Event h-STREAM Portal ksqlDB ksqlDB User 1 2 3 4 5
- 14. h-STREAM Features CDC/Event Data Pipeline Data Hub AI Ops Process h-STREAM Data Source BI AI Product System Log Event Output Key Features Explore Topic ·Stream User Self Service Deploy to Ops Process Monitoring Log Management Server Select Topic&Stream Work on Sandbox Deploy to Ops Target A B C Topic 1 Topic 2 Topic … Mirroring to Sandbox Ops Dev Topic Stream User Choice Stream Data Topic Stream User Make Stream Data Stream Designer Query User Stream Ops Stream Data Ops Environment User Stream Log Monitoring
- 15. h-STREAM Demo