a Real-time Processing System based on Spark streaming int he field of Telecommunications
Download as PPTX, PDF2 likes677 views
The document discusses a real-time processing system using Spark Streaming for telecommunications, highlighting the increasing data demands and real-time requirements in the field. It outlines a framework that incorporates various components like Hadoop, Flume, and Kafka for data input, processing, and output, while addressing performance metrics for scalability. The future work focuses on supporting more scenarios, integrating additional streaming frameworks, and improving system reliability and processing speed.
a Real-time Processing System based on Spark streaming int he field of Telecommunications
- 1. A Real-time Processing System based on Spark streaming in the filed of Telecommunications FOR Hadoop SUMMIT，2017 Geng Wang Dong Wang
- 2. CONTENT
- 3. CONTENT 01
- 4. What we faced in Telecommunications 1 3 2 4 0.166 million new users / day 33G data / sec by mobile 10T Voice data / day 100T Signal data / day More data are produced
- 5. What we faced in Telecommunications More Real time Requirements • Smart Tourism • Intelligent marketing • Tourist count and analysis • Best choice of tourist resort • Recommendation of route for travelling • … • Recommendation of product for specific customer • Based on multiple dimensions (location, age, salary …)
- 6. Evolution of Requirement 2014 Real time Marketing 2015 • Operation based on Location 2017 Hard Real time … CEP, Esper 2016 • More data input 2/3/4 G Signal of Location Content of Business
- 7. CONTENT 02
- 8. Framework – High level Data output Tagging Data Input
- 9. Detailed Framework • Hadoop Layer • Basic components • OCSP core • Data pre-processing • Tagging • Event output (select and filter) • Multiple engine ( Spark Streaming and Storm) • Muti-tenant • Check point • Data transformation • Parse data to Kafka • Nifi and Flume • Customized processor & sink • Data source • Socket • Local files • HDFS
- 10. Framework - Data Input flume agent 1 • 2，3G Signal of location flume agent 2，3，4 • 4G Signal of location Nifi • Content data of Business Kafka Partition
- 11. Data Preprocessing Kafka transform transform transform Schema 2 Schema 3 Select Filter Select Expr 1 imsi Filter Expr 1 imsi！=0 Select Expr 2 Filter Expr 2 Select Expr 3 Filter Expr 3 Select Filter Select Filter Uniform Schema 1
- 13. Tagging process Customized operation Get by Key Codis User info Stay duration Cycle of Marketing User name imsi Phone number Base station
- 14. Select, filter & Output Codis Kafka Data with labels others Current location update for each user Output 3: User with specific location & specific business Output 2: New user marketing Output 1: User Path in a duration
- 16. Framework - Deployment & Configuration External system SDTP Socket Source HDFS I/O Codis I/O Web Deployment in single host Deployment/ Configuration OCSP
- 17. CONTENT 03
- 18. Performance - scale out flume flume Nifi Nifi flume Codis Codis Codis Tagging SparkData Input kafKa Spark Data Output Kafka Spark
- 19. How OCSP works in Smart Tourism? Tagging Filter Select Output Codis • Data Source: • 4 G signal data • imsi + location + timestamp • Data transformation: • Flume source: socket • Sink: Kafka (keyed message) • Streaming processing • Filter invalid data • Tagging, get user’s information from codis by imsi • Tagging, compute the user path in a duration • Output • Write the latest location to Kafka • Use flume to update latest location in Hbase 4G data socket flume Imsi | location | timestamp Imsi | location | timestamp Imsi | location | timestamp … Kafka Imsi|location|timestamp|name|age|longitude|latitude … Imsi|location|timestamp|name|age|longitude|latitude Imsi|location|timestamp|name|age|longitude|latitude HBase Kafka flume
- 20. Performance - time cost Scenario Data per 30 s Spark Codis Kafka Partition Output number Case 1 0.6 million 20/128G/32 core 10/128G 200 3 Case 2 10 million 28/512G/64 core 10/512G 1200 11 Tagging（Get cache） Tagging （Operation） Output Case1 5 seconds 0.5 s 3 s 1s 1.5 s Data Transformation 1s 11 s 3 s 2 s Case2 17 seconds
- 21. CONTENT 04
- 22. Next Work Support more Scenarios Faster HA • Join of multiple streams in a time window • More streaming framework, flink, beam etc. • Spark upgrade, structured streaming • Faster cache • No single point of failure
- 24. Thanks