How We Migrate PBs Data from Beijing to Shanghai
2 likes4,606 views
Wang Yuxi presented how Umeng migrated their PBs data from Beijing to Shanghai over the course of 6 months. They setup a dedicated 1Gbps fiber connection between the cities and built tools to transfer data from HBase, MongoDB, MySQL, and Redis. They also provisioned applications between the environments and performed benchmarking and stress tests. The migration was completed successfully over a Friday night with no outages.
How We Migrate PBs Data from Beijing to Shanghai
- 1. How We Migrate PBs Data from Beijing to Shanghai Wang Yuxi, Umeng w@umeng.com
- 2. Agenda ● Why migrating ● Current Infrastructure ● Environment Setup ● Data Transfer(HBase) ● Data Transfer (MongoDB) ● Data Transfer (Mysql/Redis) ● Application Provision ● Monitoring ● Benchmark and Stress Testing ● Go! ● Results ● Recap
- 3. About Me ● Before 2014, the only ops at Umeng ● Now, core member of ops team ● Technical generalist, responsible for the overall reliability and performance of Umeng ● ArchLinux user @Jasey_Wang | http://JaseyWang.Me
- 4. About Umeng ● Founded on April 2010 ● Incubated by Innovation Works ● $10 Million raised from Matrix China ● Acquired by Alibaba ● Largest Mobile app analytical platform in China ● 400K+ Apps ● ~1B mobile device
- 5. Why Migrating ● Capex/Opex ● Unlimited resources, no worries for IDC, Racks, Bandwidth, etc. ● Integration with group's internal systems, massive advanced tools ● Make our PBs data safer
- 6. Current Infrastructure ● Data center: 4 ● Server: 1000 ● Networking device: 100 ● Bandwidth: 4Gbps+ ● Realtime analytics: 150K qps ● Batch processing: 4P/5P storage usage ● Know more? see Umeng Operations Infrastructure & Practice
- 8. Environment Setup ● 1G dedicated fiber between Beijing and Shanghai ready ● Due to security reason, can only send SYN from Shanghai to Beijing ● Setup DNAT for Beijing cluster ● Raw data transfer test, saturate the bandwidth(iperf -P/netperf)
- 9. Data transfer(HBase) ● HBase, 0.94 @ Beijing, 0.98 @ Shanghai ● Build-in import/export tools don’t work ● Write own import/export tool, integrity check ● Task scheduler, historical & daily incremental data ● 2 months data transfer
- 11. Data transfer(MongoDB) ● Master/Slave ○ 2.4.11, obsolete ○ build-in replica mechanism ● Primary/Secondary/Arbiter ○ replica arch ○ since Beijing can’t connect to Shanghai, write tools to read Oplog and replay into Shanghai DB cluster ● Oplog, lag, TCP keepalive, slow query, dead lock
- 12. Data Transfer(MySQL/Redis) ● MySQL ○ percona-xtrabackup, quite handy ● Redis ○ single instance, slaveof command ○ twemproxy, export and import later by tools
- 13. Application Provision ● Stateless or stateful? ● Kafka & Mirror ○ consumer/producer queue ○ qps, topics, io ● Storm ○ throughput, lag ● Zookeeper ○ 5 nodes, 4 letter words, log cleaning
- 14. Monitoring ● Internal monitoring system backed by HBase ● Zabbix, Ganglia ● Graphite for metrics ● Monit for process monitoring
- 15. Benchmark and Stress Testing ● Single component ○ system level metrics ○ application level metrics ● Multiple components ● Part of online traffic
- 16. Go! ● What about PLAN B or C? ○ plan B usually does not work ● Friday night from 00:00 to 08:00 ● Route tens of products traffic to Shanghai smoothly ● The site is fully available without outage
- 17. Results ● Sophisticated projects, all members in ● 6 months work pays off ● Hugely successful, no roll-back ● Part of products now running on private cloud
- 18. Recap ● Tools are #1 productive forces ● Test, test and test ● Monitoring and metrics
- 19. End Q & A