Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink Forward
3 likes478 views
The document discusses Zalando's microservices architecture and the integration of Apache Flink for real-time data processing and business intelligence. It highlights various use cases including business process monitoring and continuous ETL, emphasizing Flink's advantages over other frameworks. The document concludes that Flink is well-suited for Zalando's next-generation BI platform due to its low latency and scalable stream processing capabilities.
Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink Forward
- 1. FLINK IN ZALANDO’S WORLD OF MICROSERVICES JAVIER LOPEZ MIHAIL VIERU 12-09-2016
- 2. 2 AGENDA ● Zalando’s Microservices Architecture ● Saiki - Data Integration and Distribution at Scale ● Flink in a Microservices World ● Stream Processing Use Cases: o Business Process Monitoring o Continuous ETL ● Future Work
- 3. 3 ABOUT US Mihail Vieru Big Data Engineer, Business Intelligence Javier López Big Data Engineer, Business Intelligence
- 4. 4
- 5. 5 One of Europe's largest online fashion retailers 15 countries ~19 million active customers ~3 billion € revenue 2015 1,500 brands 150,000+ products 11,000+ employees in Europe
- 6. 6 ZALANDO TECHNOLOGY 1300+ TECHNOLOGISTS Rapidly growing international team http://tech.zalando.com
- 8. 8 VINTAGE BUSINESS INTELLIGENCE Classical ETL process Business Logic Data Warehouse (DWH) DatabaseDBA BI Business Logic Database Business Logic Database Business Logic Database Dev
- 9. 9 VINTAGE BUSINESS INTELLIGENCE DWH Oracle Exasol
- 10. RADICAL AGILITY
- 12. 12 RADICAL AGILITY - AUTONOMY Technologies Operations Teams
- 14. 14 SUPPORTING AUTONOMY: MICROSERVICES Business Logic Database Team A Business Logic Database Team B RESTAPI RESTAPI public Internet Applications communicate using REST APIs Databases hidden behind the walls of AWS VPC
- 15. 15 SUPPORTING AUTONOMY: MICROSERVICES Business Logic Database Team A Business Logic Database Team B RESTAPI RESTAPI public Internet Classical ETL process is impossible!
- 17. SAIKI
- 18. 18 SAIKI DATA PLATFORM SAIKI App A App B App DApp C BI Data Warehouse
- 19. 19 SAIKI — DATA INTEGRATION & DISTRIBUTION BI Data Warehouse E.g. Forecast DB SAIKI App A App B App DApp C Exporter REST API Stream Processing via Apache Flink Data Lake . AWS S3
- 20. 20 SAIKI — SUMMARY Δ J D B C REST B E F O R E A F T E R Data sources Technologies Data sources Connections Data sources Extraction Data Delivery
- 21. FLINK IN A MICROSERVICES WORLD
- 22. 22 OPPORTUNITIES FOR NEXT GEN BI Cloud Computing - Distributed ETL - Scale Access to Real Time Data - All teams publish data to central event bus Hub for Data Teams - Data Lake provides distributed access and fine grained security - Data can be transformed (aggregated, joined, etc.) before delivering it to data teams Semi-Structured Data “General-purpose data processing engines like Flink or Spark let you define own data types and functions.” - Fabian Hueske, dataArtisans
- 23. 23 THE RIGHT FIT STREAM PROCESSING
- 24. 24 THE RIGHT FIT — STREAM PROCESSING ENGINE Candidates: Storm & Samza ruled out because of batch processing requirement
- 25. 25
- 26. 26 SPARK VS. FLINK DIFFERENCES Feature Apache Spark 1.5.2 Apache Flink 0.10.1 Processing mode micro-batching tuple at a time Temporal processing support processing time event time, ingestion time, processing time Latency seconds sub-second Back pressure handling manual configuration implicit, through system architecture State access full state scan for each microbatch value lookup by key Operator library neutral ++ (split, windowByCount..) Support neutral ++ (mailing list, direct contact & support from data Artisans)
- 27. 27 APACHE FLINK • true stream processing framework • process events at a consistently high rate with low latency • scalable • great community and on-site support from Berlin/ Europe • university graduates with Flink skills https://tech.zalando.com/blog/apache-showdown-flink-vs.-spark/
- 28. 28 FLINK ON AWS - OUR APPLIANCE MASTER ELB EC2 Docker Flink Master EC2 Docker Flink Shadow Master WORKERS ELB EC2 Docker Flink Worker EC2 Docker Flink Worker EC2 Docker Flink Worker
- 29. USE CASES
- 31. 31 BUSINESS PROCESS A business process is in its simplest form a chain of correlated events: start event completion event ORDER_CREATE D ALL_PARCELS_SHIPPED Business Events from the whole Zalando platform flow through Saiki => opportunity to process those streams in near real time
- 32. 32 REAL-TIME BUSINESS PROCESS MONITORING • Check if business processes in the Zalando platform work • Analyze data on the fly: o Order velocities o Delivery velocities o Control SLAs of correlated events, e.g. parcel sent out after order
- 33. 33 Saiki BPM ARCHITECTURE BPM Cfg Service App A App B Nakadi Event Bus App C Operational Systems Kafka2Kafka Unified Log PUBLICINTERNET OAUTH Alert Svc UI Elasticsearch Stream Processing
- 34. 34 HOW WE USE FLINK IN BPM • 1000+ Event Types; 1 Event Type -> 1 Kafka topic • Analyze processes with correlated event types (Join & Union) • Enrich data based on business rules • Sliding Windows (1min to 48hrs) for Platform Snapshots • State for alert metadata • Generation and processing of Complex Events (CEP lib)
- 35. STREAMING ETL
- 36. 36 Extract Transform Load (ETL) Traditional ETL process: • Batch processing • No real time • ETL tools • Heavy processing on the storage side
- 37. 37 WHAT CHANGED WITH RADICAL AGILITY? • Data comes in a semi-structured format (JSON payload) • Data is distributed in separate Kafka topics • There would be peak times, meaning that the data flow will increase by several factors • Data sources number increased by several factors
- 38. 38 ` Saiki Streaming ETL ARCHITECTURE STREAMING ETL Stream Processing App A App B Nakadi Event Bus App C Operational Systems Kafka2Kafka Unified Log Exporter Oracle DWH Importer
- 39. 39 HOW WE (WOULD) USE FLINK IN STREAMING ETL • Transformation of complex payloads into simple ones for easier consumption in Oracle DWH • Combine several topics based on Business Rules (Union, Join) • Pre-Aggregate data to improve performance in the generation of reports (Windows, State) • Data cleansing • Data validation
- 40. FUTURE USE CASES
- 41. 41 COMPLEX EVENT PROCESSING FOR BPM Cont. example business process: • Multiple PARCEL_SHIPPED events per order • Generate complex event ALL_PARCELS_SHIPPED, when all PARCEL_SHIPPED events received (CEP lib, State)
- 42. 42 DEPLOYMENTS FROM OTHER BI TEAMS Flink Jobs from other BI Teams Requirements: • manage and control deployments • isolation of data flows o prevent different jobs from writing to the same sink • resource management in Flink o share cluster resources among concurrently running jobs StreamSQL would significantly lower the entry barrier
- 43. 43 REPLACE KAFKA2KAFKA COMPONENT • Python app • extracts events from REST API Nakadi Event Bus • writes them to our Kafka cluster Idea: Create Nakadi consumer/producer to enable stream processing with Flink to other internal users (first POC done)
- 44. 44 OTHER FUTURE TOPICS • New use cases for Real Time Analytics/ BI o Sales monitoring o Price monitoring • Fraud detection for payments (evaluation) • Contact customer according to variable event pattern (evaluation)
- 45. 45 CONCLUSION Flink proved to be the right fit for our current stream processing use cases. It enables us to build Zalando’s Next Gen BI platform. https://tech.zalando.de/blog/?tags=Saiki
- 46. THANK YOU