DC Migration and Hadoop Scale For Big Billion Days
Download as PPTX, PDF0 likes460 views
The document presents the migration of Flipkart's data center (DC) to a new infrastructure with significant enhancements, including an increase in nodes from ~340 to ~1000 and storage from ~1.7 PB to ~30 PB. It discusses the challenges faced during data migration, such as managing ephemeral IP addresses and ensuring data consistency, along with utilities developed for scalable data transfer. The successful migration was achieved without downtime, leveraging tools like Blueshift and various strategies for data validation and replication.
DC Migration and Hadoop Scale For Big Billion Days
- 1. DC Migration and Hadoop Scale For Big Billion Days Presented By: Rahul Agarwal, Operation Engineer, Flipkart Dhiraj Kumar, Software Development Engineer, Flipkart Chetna Chaudhari, Software Development Engineer, Flipkart
- 2. ● DC Landscape Before Migration ● Data at Flipkart - FDP Stats ● Infrastructure Challenges ● Hadoop Clusters Iterations ● Data Migration - Challenges ● Data Migration - Utilities Considered and Developed ● Data Migration - Execution Agenda
- 3. 1 Gbps Shared 10 Gbps Shared Primary: All User, Order & FDP systems Secondary: Few batch processing systems (User Insights, Ads and Recommendations) New: All User and FDP systems DC Landscape
- 4. Flipkart Data Platform in the old DC ● ~340 nodes ● ~1.7 PB storage ● ~30TB RAM ● ~11000 cores
- 5. Flipkart Data Platform in the new DC ● ~1000 nodes ● ~30 PB storage ● ~75TB RAM ● ~32000 cores
- 6. Data at Flipkart - FDP Stats ● New data ingested daily o ~6 TB on a business-as-usual day o ~30 TB on sale days ● Number of raw data streams ~1000s ● Number of raw events in a day ~ 3 Billion ● Volume of data processed daily ~ 0.6 PB ● Number of Hadoop jobs run each day ~ 10K
- 7. InfraStructure Challenges C1: Validation of the Hardware in entirely new infrastructure S1: Ripper Utility based on MR framework doing native writes/reads on all disks on all datanodes. C2: Ephemeral IP Addresses and Lack of DNS S2: Managed Config Service: • Store Key/Value Pairs - HostName/IP pairs • ConfD implementation on all clients
- 8. InfraStructure Challenges - Contd. C3: Ephemeral Disks and Nodes S3.1: Quorum of 5 for ZK/JN S3.2: Isolated Deployments for Each Component C4: Lesser Memory on NameNode - 180mn FS Objects S4.1: Delete Zero Size Files. S4.2: Switching To G1 Helped Reduce Pause Times.
- 9. Hadoop Cluster Iterations - RedPill • Utility Around IaaS and Ambari: – Acquire Instance Types as requested – Setup MySQL, Ambari Server and Agents – Determine cluster configurations based on Instance Types – Co-Host master components for Dev/Test deployments and Isolated components for Prod setup. – Generate Blueprint and Cluster Templates – Deploy Cluster with Blueprint and batch of 50 nodes at first – Horizontally scale cluster by adding further batches to prevent Repo service related failures – TAT of ~20 mins for 100 node cluster
- 10. Data Migration
- 11. Data Migration Challenges • Data publishers/consumers not moving together – Data consumers could move earlier than the publishers or vice-versa. • Migrating PBs of data not feasible over network • Consistency for raw, prepared and reporting data
- 12. • Moving Disks from one center to another – Data centers in different states - Legal Challenges – Live data , 24/7 in use for analytics • Replicate the data (Copy, Mirror and Regenerate) – Files being created and deleted continuously – Build the supporting services for scale Solutions Considered | Data Migration
- 13. Data Migration Utilities- DistCp • Small Files Performance • Takes long to build index • Hard to Figure out Corruption/Copy Aborts • Content Based Data Validation is weak (CRC)
- 14. Data Migration Utilities - Transporter • Configurable Batch Sizes • Compression at Source • MD5 sum of the content • HAR to bundle small files • DistCP HAR in Binary Mode • UnHAR at Destination • MR Validation for MD5 sum of the content • Regenerate Production Hierarchy • File Counts Verification
- 15. Data Migration Utilities - BlueShift • OSS : https://github.com/flipkart-incubator/blueshift • Features: – On the fly compression – Bulk migration with batches of over 10 Million files – State management options, either HDFS or DB – Optimized task scheduling – Capable of using different protocols for source and destination. – MD5 based checksum to ensure no corruptions – Time based file filtering – filesize based filtering – Option to ignore exceptions and continue processing
- 16. Data Replication • Only copied raw data about O(100TB) compressed • Too many small files • some files were very large – All prepared and reports data generated from raw data • Propagated delta changes using an Apache Kafka mirror • Verification utilities to check correctness in data in both clusters • Ran the full data platform stack in both places for over 2 weeks till all data publishers and consumers move
- 17. 2 way sync of Kafka Streams Mirror Old DC New DC Kafka A Kafka C Kafka B
- 18. HBase Migration | Solutions • Copy Table – issues: • Full table scan - time consuming • Secure to UnSecure not supported • HBase Import/ Exports – issues: • Full table scan • Slower than copy table • Needs manual interventions • Extra space • Decompression while export • Use Blue-Shift and HBase Bulk loader
- 19. Blue-Shift + HBase Bulk Load HDFS FTP Disks Transferred Over Road - Trucks HBase HDFS
- 20. Blue-Shift + HBase Bulk Load • Moved snapshots of derived/computed data over wire (relatively small) • Used physical disks to move data (stored in HBase) • Avoided HBase export. Instead transferred HFiles into disks using blueshift – knapsack'ed ~50K files into dozens of physical hard disks • Disks shipped to new DC • Transferred HFiles into HDFS using Blueshift
- 21. Learnings • MD5 checksum - big win • Should have used workflow • Automated process is must • Having isolations per tenant is must Achievement :) • Migration without downtime
- 22. Thank You !! Questions ?? @cchaudhari11 @rahul67 @dhiraj2kumar