umeng analytical arch
0 likes211 views
Umeng is a mobile analytics company that collects 1PB of data and 1TB of new data daily from 100,000 apps and 600 million devices. They use a real-time system based on technologies like Storm and Kafka to handle 15 nodes and 60k queries per second, and a batch system based on Hadoop, HDFS, HBase and MapReduce across 100 nodes to process data. They face challenges with data skew in real-time and optimizing HBase usage in batch. Going forward they aim to unify real-time and batch processing and improve data availability.
umeng analytical arch
- 1. Zhang Yan 2014.02 Umeng Analytical Arch
- 2. Outlines ● what are we doing ● how are we doing ● real-time system ● batch system ● lambda arch ● future & challenge
- 3. What are we doing ● founded on 2010.4 ● focus on mobile app analytics (and share) ● 100k apps, 600 million devices ● 1PB data, 1TB increment per day ● 1 billion messages per day ● real-time: 15 nodes, 60k QPS (data skew) ● batch: 100 nodes, 500+ jobs
- 4. Real-time Processing System(Thunder) How are we doing Batch Processing System(Iceberg) SDK SDK SDK torrent HDFS HBase MapReduce Jobs kvproxy Front End & Web nginx nginx nginx finagle log finagle log Kafka MongoDB Storm
- 5. Real-time system Real-time Processing System(Thunder) nginx nginx nginx finagle server finagle server Kafka MongoDB Storm Real-time Processing System(Thunder) nginx nginx nginx ruby server ruby server Resque MongoDB ruby worker Evolve ... To ….
- 6. Real-time system(cont.) ● technology we use ○ java, scala, ruby ○ nginx, resque, mongodb ○ finagle, kafka, storm ● challenge we face ○ data skew ○ mongodb ■ global lock is bad. ■ table lock, but still good not enough.
- 7. Batch system Batch Processing System(Iceberg) torrent HDFS HBase MapReduce Jobs kvproxy finagle finagle log log But we wish it could be ... Batch Processing System(Iceberg) Kafka HDFS HBase MapReduce Jobs kvproxy Kafka Mirror
- 8. Batch system(cont.) ● technology we use ○ java, scala, python, c/c++ ○ hadoop, hdfs, hbase, pig, hive, opentsdb ● challenge we face ○ data processing in a uniform way ○ optimize hbase usage(hash prefixed key, bulk load) ○ big amount of precomputation(HyperLogLog) ○ debugging, deployment, measurement etc.
- 9. Lambda arch ● merge real-time and batch ● complete each other ● real-time ○ fast, imprecise, narrow scope ● batch ○ slow, accurate, wide scope
- 10. Lambda Arch(cont.) Real-time(Thunder) Data Batch(Iceberg) Data MongoDB HBase kvproxy Front End & Web Index Selector (not implemented yet) eg. 1. recently 5min DAU and Installations 2. recently 1 months DAU and Installations
- 11. Future & Challenge ● just-in-time computation ○ druid, impala, spark etc. ● unify real-time & batch processing in one piece of code. ● data availability(internal & external)