Large-scale Web Apps @ Pinterest
3 likes4,109 views
Pinterest, a visual discovery tool, faces significant scale challenges with hundreds of millions of pins and billions of requests, leveraging a storage stack that includes MySQL, Redis, and HBase. The document outlines the technical rationale for using HBase, such as high write throughput and seamless Hadoop integration, along with their architecture for data storage and processing. It also discusses features, fault tolerance, and optimization strategies to ensure high availability and low latency for various applications.
Large-scale Web Apps @ Pinterest
- 1. Large Scale Web Apps @Pinterest (Powered by Apache HBase) May 5, 2014
- 2. Pinterest is a visual discovery tool for collecting the things you love, and discovering related content along the way. What is Pinterest ?
- 4. Scale Challenges @scale • 100s of millions of pins/repins per month • Billions of requests per week • Millions of daily active users • Billions of pins • One of the largest discovery tools on the internet
- 5. Storage stack @Pinterest ! • MySQL • Redis (persistence and for cache) • MemCache (Consistent Hashing) App Tier Manual Sharding Sharding Logic
- 6. Why HBase ? ! • High Write throughput - Unlike MySQL/B-Tree, writes don’t ever seek on Disk • Seamless integration with Hadoop • Distributed operation - Fault tolerance - Load Balancing - Easily add/remove nodes ! Non-Technical Reasons • Large active community • Large scale online use cases
- 7. Outline ! • Features powered by HBase • SaaS (Storage as a Service) - MetaStore - HFile Service (Terrapin) • Our HBase setup - optimizing for High availability & Low latency
- 8. Applications/Features ! • Offline - Analytics - Search Indexing - ETL/Hadoop worklows • Online - Personalized Feeds - Rich Pins - Recommendations ! Why HBase ?
- 9. Personalized Feeds WHY HBASE ? Write Heavy load due to Pin fanout. Recommended Pins Users I follow
- 10. Rich Pins WHY HBASE ? Negative Hits with Bloom Filters
- 11. Recommendations HADOOP 1.0 HBASE + HADOOP 2.0 HADOOP 2.0 WHY HBASE ? Seamless Data Transfer from Hadoop Generate Recommendations DistCP Jobs Serving Cluster
- 12. SaaS • Large number of feature requests • 1 Cluster per feature • Scaling with organizational growth • Need for “defensive” multi tenant storage • Previous solutions reaching their limits
- 13. MetaStore I • Key Value store on top of HBase • 1 HBase Table per Feature with salted keys • Pre split tables • Table level rate limiting (online/offline reads/writes) • No Scan support • Simple client API! ! string getValue(string feature, string key, boolean online); void setValue(string feature, string key, string value, boolean online);
- 14. MetaStore II MetaStore Thrift Server Primary HBase Secondary HBase Clients Master/Master Replication Thrift Salting + Rate Limiting ZooKeeper Issue Gets/Sets Notifications Metastore Config - Rate Limits - Primary Cluster
- 15. HFile Service (Terrapin) • Solve the Bulk Upload problem • HBase backed solution - Bulk upload + major compact - Major compact to delete old data • Design solution from scratch using mashup of: - HFile - HBase BlockCache - Avoid compactions - Low latency key value lookups ! ! !
- 16. High Level Architecture I ! Client Library /Service ETL/Batch Jobs Load/Reload HFile Servers ! HFiles on Amazon S3 Key/Value Lookups Multiple HFiles/Server
- 17. High Level Architecture II • Each HFile server runs 2 processes - Copier: pulls HFiles from S3 to local disk - Supershard: serves multiple HFile shards to client • ZooKeeper - Detecting alive servers - Coordinating loading/swapping of new data - Enabling clients to detect availability of new data • Loader Module (replaces distcp) - Trigger new data copy - Trigger swap through zookeeper - Update ZooKeeper and notify client • Client library understands sharding • Old data deleted by background process !
- 18. Salient Features • Multi tenancy through namespacing • Pluggable sharding functions - modulus, range & more • HBase Block Cache • Multiple clusters for redundancy • Speculative execution across clusters for low latency ! ! !
- 19. Setting up for Success • Many online usecases/applications • Optimize for: - Low MTTR - high availability - Low latency (performance) ! !
- 20. MTTR - I DEADLIVE STALE 20sec 9min 40sec ! • Stale nodes avoided - As candidates for Reads - As candidate replicas for writes - During Lease Recovery • Copying of underreplicated blocks starts when a Node is marked as “Dead” DataNode States
- 21. MTTR - II Failure Detection Lease Recovery Log Split Recover Regions 30 sec ZooKeeper session timeout HDFS 4721 HDFS 3703 + HDFS 3912 < 2 min ! • Avoid stale nodes at each point of the recovery process • Multi minute timeouts ==> Multi second timeouts
- 22. Simulate, Simulate, Simulate Simulate “Pull the plug failures” and “tail -f the logs” • kill -9 both datanode and region server - causes connection refused errors • kill -STOP both datanode and region server - causes socket timeouts • Blackhole hosts using iptables - connect timeouts + “No Route to host” - Most representative of AWS failures
- 23. Performance Configuration tweaks • Small Block Size, 4K-16K • Prefix compression to cache more - when data is in the key, close to 4X reduction for some data sets • Separation of RPC handler threads for reads vs writes • Short circuit local reads • HBase level checksums (HBASE 5074) Hardware • SATA (m1.xl/c1.xl) and SSD (hi1.4xl) • Choose based on limiting factor - Disk space - pick SATA for max GB/$$ - IOPs - pick SSD for max IOPs/$$, clusters with heavy reads or heavy compaction activity
- 24. Performance (SSDs) HFile Read Performance • Turn off block cache for Data Blocks, reduce GC + heap fragmentation • Keep block cache on for Index Blocks • Increase “dfs.client.read.shortcircuit.streams.cache.size” from 100 to 10,000 (with short circuit reads) • Approx. 3X improvement in read throughput ! Write Performance • WAL contention when client sets AutoFlush=true • HBase 8755
- 25. In the Pipeline... ! • Building a graph database on HBase • Disaster recovery - snapshot + incremental backup + restore • Off Heap cache - reduce GC overhead and better use of hardware • Read path optimizations
- 26. And we are Hiring !!