Realtime Reporting using Spark Streaming
2 likes2,867 views
Concur, a part of SAP, focuses on travel and expense management, leveraging real-time reporting with Kafka and Spark Streaming to overcome traditional ETL challenges such as high latency and scalability issues. The architecture employs a robust streaming data pipeline, utilizing various data sources and systems, enabling scalable and fault-tolerant streaming applications. Additionally, it emphasizes the benefits of the Lambda architecture to efficiently process large data volumes through both batch and stream methods.
](https://image.slidesharecdn.com/sparkmeetup-realtimereporting-santoshsahoo-150713212804-lva1-app6892/85/Realtime-Reporting-using-Spark-Streaming-17-320.jpg)
Realtime Reporting using Spark Streaming
- 1. Breaking ETL barrier with Real-time reporting using Kafka, Spark Streaming
- 2. About us Concur (now part of SAP) provides travel and expense management services to businesses.
- 3. Data Insights A team that is building solutions to provide customer access to data, visualization and reporting. Expense Travel Invoice
- 4. About me Santosh Sahoo Principal Architect III, Data Insights
- 5. Stack so far.. OLAP ReportETL OLTP App
- 6. Numbers 7K OLTP database sources 14K OLAP Reporting dbs 28K ETL Jobs 2B row changes 300M rows (Compacted) Only ~20 failure a night
- 7. Traditional ETL challenges Scheduled (High latency) Hard to scale. Failover and recovery. Monolithic-ness Spaghetti (Logic +SQL)
- 8. Moving forward Streaming, real time Scalable Highly available Reduce maintenance overhead Eventual Consistency
- 9. Streaming Data Pipeline Source Flow Management Processor Storage Querying
- 10. Data Source Event bus for business events Log Scrapping Transaction log scraping (Oracle GoldenGate, MySQL binlog, MongoDB oplog, Postgres BottledWater, SQL Server fn_dblog) Change Data Capture Application messaging/JMS Micro batching (High watermarked, change tracking)
- 11. Kafka - Flow Management No nonsense logging 100K/s throughput vs 20k of RabbitMQ Log compaction Durable persistence Partition tolerance Replication Best in class integration with Spark
- 12. Columnar Storage Optimized for analytic query performance. Vertical partitioning Column Projection Compression Loosely coupled schema. HBase AWS Redshift Parquet ORC Postgres (Citrus) SAP HANA
- 13. Hadoop/HDFS Pro - Scale Con- Latency
- 14. Spark Streaming What? A data processing framework to build scalable fault-tolerant streaming applications. Why? It lets you reuse the same code for batch processing, join streams against historical data, or run ad-hoc queries on stream state.
- 15. Spark Streaming Architecture Worker Worker Worker Receiver Driver Master Executor Executor Executor Source D1 D2 D3 D4 WAL D1 D2 Replication Data Store TASK DStream- Discretized Stream of RDD RDD - Resilient Distributed Datasets
- 16. Optimized Direct Kafka API https://databricks.com/blog/2015/03/30/improvements-to-kafka-integration-of-spark-streaming.html
- 17. How val kafkaParams = Map("metadata.broker.list" -> "localhost:9092, anotherhost:9092") val topics = Set("sometopic", "anothertopic") val kafkaStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](streamingContext, kafkaParams, topics)
- 18. Architecture
- 20. OLTP Reporting Cognos Tableau ? Archive Flume Camus Stream Processor Spark Samza, Storm, Flink HDFS Import FTP HTTP SMTP C Tachyon P Standby Protobuf Json Broker Kafka Hive/ Spark SQL HANA Load balance Failover HANA HANA HANA Replication Service bus SqoopSnapshot Pig/Hive/MR - Normalization Extract Compensate Data {Quality, Correction, Analytics} Migrate method API/SQL Expense Travel TTX API Complete Architecture
- 21. Can Spark Streaming survive Chaos Monkey? http://techblog.netflix.com/2015/03/can-spark-streaming-survive-chaos-monkey.html
- 22. Lambda Architecture Lambda architecture is a data-processing pattern designed to handle massive quantities of data by taking advantage of both batch- and stream-processing methods.
- 23. Demo ….
- 24. QnA
- 26. Thank you!