E commerce data migration in moving systems across data centres
8 likes3,268 views
This document summarizes Flipkart's data migration process from one data center to another. Key points include: - Flipkart processes large volumes of data daily from users and orders and needed to migrate systems across data centers with minimal downtime. - The migration involved over 1000 systems and terabytes of data across various data stores like HBase, Elasticsearch, MySQL, Kafka and data platforms. - Custom tools were developed to efficiently migrate and replicate data across data centers to minimize bandwidth usage and ensure consistency, including a tool to migrate HBase data using hard disks and utilities for migrating MySQL, product catalogs and user sessions. - The migration was carefully planned and executed over several weeks to move systems
E commerce data migration in moving systems across data centres
- 1. eCommerce data migration in moving systems across Data Centres Regu, Sharad
- 2. Flipkart in recent times ● Leading eCommerce player in India ○ 10M page visits, 2M shipments a day ○ 30 million products across more than 70 categories ○ Big Billion Days ($300M sales, top ranked app on Google Play Store) ○ Ping - social collab in eCommerce ○ Progressive web app - native like experience on browser (debuted at #chromedevsummit 2015)
- 3. Data at Flipkart - User & Order path
- 4. Data at Flipkart - Data Platform ● 6 TB new data ingested daily ○ 30 TB on sale days ● 1100 Raw Streams ● 3 Billion Raw events in a day ● 0.6 PB data processed daily ● 10K Hadoop jobs daily ● 3000 Report views daily
- 5. DC Landscape DC2 (Chennai) DC1 (Chennai) DC3 (Chennai) 1 Gbps 2 x 10 Gbps Primary : All User, Order & FDP systems Secondary : Few batch processing systems (UIE, Reco, Ads) New : All User & FDP systems
- 6. Data migration needs, challenges ● User path systems ○ Minimize downtime. Site & App downtime is visible ■ Data - mostly eventually consistent ○ Session Data - Avoid User logout, Service scale : 250K RPS ○ Promise Data (Stock, Serviceability) - Avoid OOS, Over-booking (consistency matters) ○ Live Orders - Accept orders, Let customers checkout & pay (consistency matters) ○ User Accounts - No data loss. Change velocity is not much ● Order path systems ○ Availability not a constraint, Throughput and Data durability is ■ Data - strong durability, consistent ○ Current orders being fulfilled ○ Warehouse stock inwarding, movement
- 7. Data migration needs, challenges ● User Insights ○ Inter DC data bandwidth limitations (1Gbps shared link) ○ 130 TB (snappy compressed) data in HBase, Derived data(Insights) much smaller though ● MySQL instance footprint - 600+ ● Flipkart Data Platform ○ Data publishers/consumers not moving together ■ Data consumers could move earlier than the publishers, vice-versa ○ Migrating couple of PB data not feasible over network ○ Consistency for raw, prepared and reporting data
- 8. Migration planning and execution ● Most useful tool - Google spreadsheets & docs! ○ Inventory of systems in each business cluster - split by service, backing data store ○ Defined data migration recipes and SME group for each data store type ■ Advise on IaaS constructs - instance types, PaaS integration - service discovery, Data migration strategy (export vs live replication), built tooling ○ Create cutover sequence and interdependencies ■ for e.g. Catalog → Search → Cart/CO → Mobile apps ○ Wrote playbook for each cutover activity - including checklists, verification of data export/restore ● Program managed a plan that touched 1000+ systems and many of the 1000 member engineering org
- 9. Knowledge base on 3rd party data stores, packages
- 10. Hacks, Tools and Utilities ● “Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway” -- Andrew S. Tanenbaum (Computer Networks, 4th ed., p. 91) ○ We used disks instead to move User Insights data (stored in HBase) ■ Moves snapshots of derived/computed data over wire (relatively small) ■ Avoided HBase export. Instead transferred HFiles into disks using custom ‘distcp’ like tool which knapsack'ed ~40K files into 6 disks. Open sourced as : https://github. com/flipkart-incubator/blueshift ■ Disked shipped to new DC ■ Transferred HFiles into HDFS using Blueshift ■ Imported HFiles into HBase using HBase Bulk Load
- 11. Migrating live User sessions - dual write ● Cold data in HBase (9TB - compressed), hot in Memcached(1TB) ● Live read-writes on Memcached, async batched writes to HBase ● Migration via Dual writes ○ Fresh Memcached cluster in new DC ○ Added this cluster as another batched write destination in old DC ○ Data move initiated 21 days before actual-cutover to allow for catchup ○ Hbase data was exported using standard snapshotting and incremental copy table periodically ○ Batch interval reduced from 10 minutes to 1 minutes during cutover for aggressive copy ● No user logout, session loss after cutover
- 12. Migrating Product catalog data ● Data modelled as Entities & Relationships : clients have “Views” of this data ● Views expressed as JSON DSL ● Raw data exported from HBase, Elastic Search and copied to new DC ● Required a solution that could migrate updates after initial move ○ Developed JSON diff library that could work over 100 million views. Open sourced : https: //github.com/flipkart-incubator/zjsonpatch ■ Diffs are applied in order - important for DC move ○ Bandwidth consumption for applying updates dropped from 800 Mbps to 13-14 Mbps
- 14. Application Relay bridge over Kafka queues ● RESTBus : orchestrates all Order fulfillment systems ● Pattern : Locally committed messages in MySQL, relayed over Kafka to Http endpoints ● Bridge over 2 DCs with destinations resolved from ELB endpoints
- 15. 2 way sync of Kafka Streams Mirror Old DC New DC
- 16. Copying data across clusters ● Only copied raw data about 200TB compressed ○ All prepared and reports data generated from raw data ● Verification utilities to check correctness in data in both clusters ● Ran the full data platform stack in both places for over 2 weeks till all data publishers and consumers move
- 17. Thank You