Big Data Logging Pipeline with Apache Spark and Kafka
1 like840 views
The document discusses a presentation on shipping YaaS logs using Apache Spark and Kafka, highlighting its architecture, technology, and challenges. It covers the logging pipeline's components, including high throughput messaging, fault tolerance, and the role of elasticsearch indexing. Key challenges such as reliability and checkpointing are addressed, along with proposed configurations for improvement.
Big Data Logging Pipeline with Apache Spark and Kafka
- 2. Shipping YaaS logs with Apache Spark and Kafka Dogukan Sonmez Senior Software Engineer @hybris Software @dogukansonmez
- 3. Agenda ² Introduction to Yaas ² Architecture of Logging pipeline ² Technology behind logging pipeline ² Challenges ² Recap ² Q&A
- 4. What is YaaS
- 5. SAP hybris as a Service (YaaS) A micro-service based Business PaaS Integrated with hybris and SAP Solutions Build Publish Fast
- 6. yaas.io
- 7. Architecture of Logging pipeline
- 8. Architecture of Logging pipeline
- 9. Technology behind logging pipeline High Throughput messaging Broker Distributed Scalable Fault Tolerant Topic Partition Replicated Offset
- 10. Technology behind logging pipeline Micro Batching RDD Streaming DAG Reliable ML Scalable Graph Fast
- 11. Big Data pipeline challenges Reliability of Kafka v 3 Brokers v 3 Zookeeper instances v default.replication.factor=2 v Mainly with Default Configurations v 5 Brokers v 5 Zookeeper instances v unclean.leader.election.enable=false v min.insync.replicas=2 v default.replication.factor=3 BEFORE AFTER
- 12. Big Data pipeline challenges Spark Streaming Checkpointing v Spark checkpointing v All RDD serialized and stored at HDFS v Custom kafka checkpointing (Only latest offset stored at kafka) BEFORE AFTER
- 13. Big Data pipeline challenges Elasticsearch indexing big data v Default mapping v index.refresh_interval = 1s v Indices.memory_index_buffer_size= 10% v Custom mapping with disabled norms v Mapping using simple analyzer v index.refresh_interval = 30s v Indices.memory_index_buffer_size= 30% v spark.streaming.kafka.maxRatePerPartition=10000 BEFORE AFTER
- 14. Recap
- 15. Recap
- 16. Q&A