SlideShare a Scribd company logo
Storage Infrastructure Behind
                 Facebook Messages
                        HBase/HDFS/ZK/MapReduce

                        Kannan Muthukkaruppan
                          Software Engineer, Facebook




Big Data Experiences & Scars, HPTS 2011
Oct 24th, 2011
The New Facebook Messages

 Messages   Chats    Emails   SMS
Why we chose HBase
▪   High write throughput
▪   Good random read performance
▪   Horizontal scalability
▪   Automatic Failover
▪   Strong consistency
▪   Benefits of HDFS
    ▪   Fault tolerant, scalable, checksums, MapReduce
    ▪   internal dev & ops expertise
What do we store in HBase
▪   HBase
    ▪   Small messages
    ▪   Message metadata (thread/message indices)
    ▪   Search index


▪   Haystack (our photo store)
    ▪   Attachments
    ▪   Large messages
HBase-HDFS System Overview
                             Database Layer

   HBASE
       Master      Backup
                   Master
   Region           Region                    Region    ...
   Server           Server                    Server
                                                              Coordination Service
            Storage Layer

HDFS                                                   Zookeeper Quorum
   Namenode          Secondary Namenode                ZK      ZK           ...
                                                       Peer    Peer
Datanode Datanode           Datanode          ...
Facebook Messages Architecture
                                                           User Directory Service
           Clients
   (Front End, MTA, etc.)
                            What’s the cell for
                               this user?         Cell 2
        Cell 1                                                   Cell 3

                                  Cell 1   Application Server
 Application Server                                  Application Server

                                                  Attachments
                                             HBase/HDFS/Z
  HBase/HDFS/Z                                Message, Metadata,
                                             K Search HBase/HDFS/Z
                                                        Index
  K                                                    K


                            Haystack
Typical Cluster Layout
 ▪   Multiple clusters/cells for messaging
     ▪   20 servers/rack; 5 or more racks per cluster
 ▪   Controllers (master/Zookeeper) spread across racks
 ZooKeeper Peer       ZooKeeper Peer     ZooKeeper Peer    ZooKeeper Peer     ZooKeeper Peer
 HDFS Namenode        Backup Namenode    Job Tracker       HBase Master     Backup HBase Master


Region Server        Region Server      Region Server     Region Server      Region Server
Data Node            Data Node          Data Node         Data Node          Data Node
Task Tracker         Task Tracker       Task Tracker      Task Tracker       Task Tracker



19x...                19x...            19x...            19x...             19x...


Region Server        Region Server      Region Server     Region Server      Region Server
Data Node            Data Node          Data Node         Data Node          Data Node
Task Tracker         Task Tracker       Task Tracker      Task Tracker       Task Tracker

         Rack #1          Rack #2            Rack #3           Rack #4            Rack #5
Facebook Messages: Quick Stats
▪   6B+ messages/day


▪   Traffic to HBase
     ▪ 75+ Billion R+W ops/day

     ▪ At peak: 1.5M ops/sec

     ▪ ~ 55% Read vs. 45% Write ops

     ▪ Avg write op inserts ~16 records across multiple

       column families.
Facebook Messages: Quick Stats (contd.)
▪   2PB+ of online data in HBase (6PB+ with replication;
    excludes backups)
    ▪ message data, metadata, search index

▪   All data LZO compressed
▪   Growing at 250TB/month
Facebook Messages: Quick Stats (contd.)
Timeline:
▪   Started in Dec 2009
▪   Roll out started in Nov 2010
▪   Fully rolled out by July 2011 (migrated 1B+ accounts from
    legacy messages!)


While in production:
▪   Schema changes: not once, but twice!
▪   Implemented & rolled out HFile V2 and numerous other
    optimizations in an upward compatible manner!
Shadow Testing: Before Rollout
▪   Both product and infrastructure were changing.
▪   Shadows the old messages product + chat while the new one was
    under development

            Messages                    Chat          Backfilled data from
           V1 Frontend                Frontend        MySQL for % of users and
                                                      mirrored their production
                                                      traffic


                                                       Application Servers



       Cassandra         MySQL           ChatLogger
                                                      HBase/HDFS Cluster


                         Production                        Shadow
Shadow Testing: After Rollout
▪   Shadows new version of the Messages product.
▪   All backend changes go through shadow cluster before prod push


                       New
                     Messages
                     Frontend



                 Application Servers               Application Servers




                                                 HBase/HDFS Cluster
                 HBase/HDFS Cluster


                                                       Shadow
                   Production
Backup/Recovery (V1)
•   During early phase, concerned about potential bugs in HBase.
•   Off-line backups: written to HDFS via Scribe
•   Recovery tools; testing of recovery tools

          Message
          Frontend


     Application Servers          Scribe/HDFS                      Local
                                                                 Datacenter
                                                                   HDFS          Remote
                                                                                Datacenter
                                                         • Merge & Dedup          HDFS
    HBase/HDFS Cluster                                   • Store locally, and
                                                         • Copy to Remote DC


                           Double log from AppServer & HBase
                           to reduce probability of data loss
Backups (V2)
▪   Now, does periodic HFile level backups.
▪   Working on:
    ▪   Moving to HFile + Commit Log based backups to be able to recover to
        finer grained points in time
        ▪   Avoid need to log data to Scribe.
    ▪   Zero copy (hard link based) fast backups
Messages Schema & Evolution
▪   “Actions” (data) Column Family the source of truth
    ▪   Log of all user actions (addMessage, markAsRead, etc.)

▪   Metadata (thread index, message index, search index) etc. in other
    column families
▪   Metadata portion of schema underwent 3 changes:
        ▪   Coarse grained snapshots (early development; rollout up to 1M users)
        ▪   Hybrid (up to full rollout – 1B+ accounts; 800M+ active)
        ▪   Fine-grained metadata (after rollout)
▪   MapReduce jobs against production clusters!
        ▪   Ran in throttled way
        ▪   Heavy use of HBase bulk import features
Write Path Overview
Region Server
                ....
            Region #2
       Region #1
                        ....
                  ColumnFamily #2

            ColumnFamily #1
                                    Memstore


                HFiles (in HDFS)



  Write Ahead Log ( in HDFS)
                                      Append/Sync
Flushes: Memstore -> HFile
Region Server
               ....
           Region #2
      Region #1
                        ....
                 ColumnFamily #2

           ColumnFamily #1
                                       Memstore


                       HFiles                               flush




           Data in HFile is sorted; has block index for efficient
           retrieval
Read Path Overview
Region Server
               ....
           Region #2
      Region #1
                         ....
                 ColumnFamily #2

           ColumnFamily #1
                                   Memstore


                       HFiles



                                         Get
Compactions
Region Server
               ....
           Region #2
      Region #1
                        ....
                 ColumnFamily #2

           ColumnFamily #1
                                   Memstore

               HFiles
Reliability: Early work
▪   HDFS sync support for durability of transactions
▪   Multi-CF transaction atomicity
▪   Several bug fixes in log recovery
▪   New block placement policy in HDFS
    ▪   To reduce probability of data loss
Availability: Early Work
▪   Common reasons for unavailability:
    ▪   S/W upgrades
        ▪   Solution: rolling upgrades
    ▪   Schema Changes
        ▪   Applications needs new Column Families
        ▪   Need to change settings for a CF
        ▪   Solution: online “alter table”
    ▪   Load balancing or cluster restarts took forever
        ▪   Upon investigation: stuck waiting for compactions to finish
        ▪   Solution: Interruptible Compactions!
Performance: Early Work
▪   Read optimizations:
    ▪   Seek optimizations for rows with large number of cells
    ▪   Bloom Filters
        ▪   minimize HFile lookups
    ▪   Timerange hints on HFiles (great for temporal data)
    ▪   Multigets
    ▪   Improved handling of compressed HFiles
Performance: Compactions                                          Was
▪   Critical for read performance
▪   Old Algorithm:
#1. Start from newest file (file 0); include next file if:
    ▪   size[i] < size[i-1] * C (good!)
#2. Always compact at least 4 files, even if rule #1 isn’t met.

Solution:
                                                                  Now
#1. Compact at least 4 files, but only if eligible files found.
#2. Also, new file selection based on summation of sizes.
           size[i] < (size[0] + size[1] + …size[i-1]) * C
Performance: Compactions
▪   More problems!
    ▪   Read performance dips during      Small Compaction Q   Big Compaction Q
        peak
    ▪   Major compaction storms               Compaction         Compaction

    ▪   Large compactions bottleneck          Compaction        Compaction

▪   Enhancements/fixes:                       Compaction
    ▪   Staggered major compactions
                                              Compaction
    ▪   Multi-thread compactions;
        separate queues for small & big
        compactions
    ▪   Aggressive off-peak compactions
Metrics, metrics, metrics…
▪   Initially, only had coarse level overall metrics (get/put latency/ops;
    block cache counters).
▪   Slow query logging
▪   Added per Column Family stats for:
    ▪   ops counts, latency
    ▪   block cache usage & hit ratio
    ▪   memstore usage
    ▪   on-disk file sizes
    ▪   file counts
    ▪   bytes returned, bytes flushed, compaction statistics
    ▪   stats by block type (data block vs. index blocks vs. bloom blocks, etc.)
    ▪   bloom filter stats
Metrics (contd.)
▪   HBase Master Statistics:
    ▪   Number of region servers alive
    ▪   Number of regions
    ▪   Load balancing statistics
    ▪   ..
▪   All stats stored in Facebook’s Operational Data Store (ODS).
▪   Lots of ODS dashboards for debugging issues
    ▪   Side note: ODS planning to use HBase for storage pretty soon!
Need to keep up as data grows on you!
▪   Rapidly iterated on several new features while in production:
    ▪   Block indexes upto 6GB per server! Cluster starts taking longer and
        longer. Block cache hit ratio on the decline.
        ▪   Solution: HFile V2
                ▪   Multi-level block index, Sharded Bloom Filters
    ▪   Network pegged after restarts
        ▪   Solution: Locality on full & rolling restart
    ▪   High disk utilization during peak
        ▪   Solution: Several “seek” optimizations to reduce disk IOPS
            ▪   Lazy Seeks (use time hints to avoid seeking into older HFiles)
            ▪   Special bloom filter for deletes to avoid additional seek
            ▪   Utilize off-peak IOPS to do more aggressive compactions during
Scares & Scars!
▪   Not without our share of scares and incidents:
    ▪   s/w bugs. (e.g., deadlocks, incompatible LZO used for bulk imported data, etc.)
        ▪   found a edge case bug in log recovery as recently as last week!
    ▪   performance spikes every 6 hours (even off-peak)!
            ▪   cleanup of HDFS’s Recycle bin was sub-optimal! Needed code and config fix.
    ▪   transient rack switch failures
    ▪   Zookeeper leader election took than 10 minutes when one member of the
        quorum died. Fixed in more recent version of ZK.
    ▪   HDFS Namenode – SPOF
    ▪   flapping servers (repeated failures)
Scares & Scars! (contd.)
▪   Sometimes, tried things which hadn’t been tested in dark launch!
        ▪   Added a rack of servers to help with performance issue
            ▪   Pegged top of the rack network bandwidth!
                ▪   Had to add the servers at much slower pace. Very manual .
                ▪   Intelligent load balancing needed to make this more automated.
▪   A high % of issues caught in shadow/stress testing
▪   Lots of alerting mechanisms in place to detect failures cases
    ▪   Automate recovery for a lots of common ones
    ▪   Treat alerts on shadow cluster as hi-pri too!
▪   Sharding service across multiple HBase cells also paid off
Future Work
▪   Reliability, Availability, Scalability!
▪   Lot of new use cases on top of HBase in the works.


      ▪   HDFS Namenode HA
      ▪   Recovering gracefully from transient issues
      ▪   Fast hot-backups
      ▪   Delta-encoding in block cache
      ▪   Replication
      ▪   Performance (HBase and HDFS)
      ▪   HBase as a service Multi-tenancy
      ▪   Features- coprocessors, secondary indices
Acknowledgements
Work of lot of people spanning HBase, HDFS, Migrations,
Backup/Recovery pipeline, and Ops!

   Karthik Ranganathan            Mikhail Bautin
   Nicolas Spiegelberg            Pritam Damania
   Aravind Menon                  Prakash Khemani
   Dhruba Borthakur               Doug Porter
   Hairong Kuang                  Alex Laslavic
   Jonathan Gray                  Kannan Muthukkaruppan
   Rodrigo Schmidt                + Apache HBase
   Amitanand Aiyer                + Interns
   Guoqiang (Jerry) Chen          + Messages FrontEnd Team
   Liyin Tang                     + Messages AppServer Team
   Madhu Vaidya                   + many more folks…
Thanks! Questions?
facebook.com/engineering

More Related Content

PPTX
Hadoop Summit 2012 | HBase Consistency and Performance Improvements
PDF
HBase @ Twitter
PDF
Hadoop Distributed File System Reliability and Durability at Facebook
PDF
Hadoop Successes and Failures to Drive Deployment Evolution
PDF
Facebook keynote-nicolas-qcon
PDF
Storage infrastructure using HBase behind LINE messages
PDF
[Hi c2011]building mission critical messaging system(guoqiang jerry)
PPTX
Moving from C#/.NET to Hadoop/MongoDB
Hadoop Summit 2012 | HBase Consistency and Performance Improvements
HBase @ Twitter
Hadoop Distributed File System Reliability and Durability at Facebook
Hadoop Successes and Failures to Drive Deployment Evolution
Facebook keynote-nicolas-qcon
Storage infrastructure using HBase behind LINE messages
[Hi c2011]building mission critical messaging system(guoqiang jerry)
Moving from C#/.NET to Hadoop/MongoDB

What's hot (20)

PDF
Meet HBase 1.0
PDF
Apache HBase for Architects
PPTX
HBaseCon 2013: Compaction Improvements in Apache HBase
PDF
Introduction to hadoop and hdfs
PPTX
MongoDB at eBay
PPTX
Operating and supporting HBase Clusters
PDF
Hadoop 101
 
PDF
HBase Advanced - Lars George
PPTX
Hadoop Demystified + MapReduce (Java and C#), Pig, and Hive Demos
PPTX
HBase Accelerated: In-Memory Flush and Compaction
PDF
Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?
PDF
Realtime Apache Hadoop at Facebook
PDF
Hadoop World 2011: The Hadoop Stack - Then, Now and in the Future - Eli Colli...
PPTX
HBase and HDFS: Understanding FileSystem Usage in HBase
PPTX
Zero-downtime Hadoop/HBase Cross-datacenter Migration
PDF
Cross-DC Fault-Tolerant ViewFileSystem @ Twitter
PDF
Apache HBaseの現在 - 火山と呼ばれたHBaseは今どうなっているのか
PDF
Apache Hadoop YARN, NameNode HA, HDFS Federation
PDF
Alfresco scalability and performnce
PPTX
Meet hbase 2.0
Meet HBase 1.0
Apache HBase for Architects
HBaseCon 2013: Compaction Improvements in Apache HBase
Introduction to hadoop and hdfs
MongoDB at eBay
Operating and supporting HBase Clusters
Hadoop 101
 
HBase Advanced - Lars George
Hadoop Demystified + MapReduce (Java and C#), Pig, and Hive Demos
HBase Accelerated: In-Memory Flush and Compaction
Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?
Realtime Apache Hadoop at Facebook
Hadoop World 2011: The Hadoop Stack - Then, Now and in the Future - Eli Colli...
HBase and HDFS: Understanding FileSystem Usage in HBase
Zero-downtime Hadoop/HBase Cross-datacenter Migration
Cross-DC Fault-Tolerant ViewFileSystem @ Twitter
Apache HBaseの現在 - 火山と呼ばれたHBaseは今どうなっているのか
Apache Hadoop YARN, NameNode HA, HDFS Federation
Alfresco scalability and performnce
Meet hbase 2.0
Ad

Similar to Storage Infrastructure Behind Facebook Messages (20)

PDF
Facebook Messages & HBase
PDF
支撑Facebook消息处理的h base存储系统
PDF
Realtime hadoopsigmod2011
PDF
Facebook's HBase Backups - StampedeCon 2012
PDF
Facebook Hadoop Usecase
PPTX
Storage Infrastructure Behind Facebook Messages
PDF
Hbase schema design and sizing apache-con europe - nov 2012
KEY
Near-realtime analytics with Kafka and HBase
PPTX
HBaseCon 2012 | Building a Large Search Platform on a Shoestring Budget
PPTX
Hadoop Backup and Disaster Recovery
PDF
Hadoop World 2011: Building Realtime Big Data Services at Facebook with Hadoo...
PDF
Managing Big Data (Chapter 2, SC 11 Tutorial)
PDF
HBaseCon 2012 | HBase Security for the Enterprise - Andrew Purtell, Trend Micro
PPTX
July 2012 HUG: Using Standard File-Based Applications and SQL-Based Tools wit...
PDF
Facebooks Petabyte Scale Data Warehouse using Hive and Hadoop
PPTX
Hic 2011 realtime_analytics_at_facebook
PPTX
Cluster Computing with Dryad
PPTX
Hadoop World 2011: Advanced HBase Schema Design - Lars George, Cloudera
PDF
Xldb2011 tue 0940_facebook_realtimeanalytics
PDF
Apache hbase for the enterprise (Strata+Hadoop World 2012)
Facebook Messages & HBase
支撑Facebook消息处理的h base存储系统
Realtime hadoopsigmod2011
Facebook's HBase Backups - StampedeCon 2012
Facebook Hadoop Usecase
Storage Infrastructure Behind Facebook Messages
Hbase schema design and sizing apache-con europe - nov 2012
Near-realtime analytics with Kafka and HBase
HBaseCon 2012 | Building a Large Search Platform on a Shoestring Budget
Hadoop Backup and Disaster Recovery
Hadoop World 2011: Building Realtime Big Data Services at Facebook with Hadoo...
Managing Big Data (Chapter 2, SC 11 Tutorial)
HBaseCon 2012 | HBase Security for the Enterprise - Andrew Purtell, Trend Micro
July 2012 HUG: Using Standard File-Based Applications and SQL-Based Tools wit...
Facebooks Petabyte Scale Data Warehouse using Hive and Hadoop
Hic 2011 realtime_analytics_at_facebook
Cluster Computing with Dryad
Hadoop World 2011: Advanced HBase Schema Design - Lars George, Cloudera
Xldb2011 tue 0940_facebook_realtimeanalytics
Apache hbase for the enterprise (Strata+Hadoop World 2012)
Ad

Recently uploaded (20)

PPTX
Big Data Technologies - Introduction.pptx
PDF
NewMind AI Weekly Chronicles - August'25 Week I
PDF
Advanced Soft Computing BINUS July 2025.pdf
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
PDF
KodekX | Application Modernization Development
PDF
solutions_manual_-_materials___processing_in_manufacturing__demargo_.pdf
PDF
GamePlan Trading System Review: Professional Trader's Honest Take
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
PDF
Unlocking AI with Model Context Protocol (MCP)
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
PDF
Approach and Philosophy of On baking technology
PDF
Spectral efficient network and resource selection model in 5G networks
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
PDF
Review of recent advances in non-invasive hemoglobin estimation
PDF
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
PPTX
Understanding_Digital_Forensics_Presentation.pptx
PDF
Machine learning based COVID-19 study performance prediction
Big Data Technologies - Introduction.pptx
NewMind AI Weekly Chronicles - August'25 Week I
Advanced Soft Computing BINUS July 2025.pdf
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
KodekX | Application Modernization Development
solutions_manual_-_materials___processing_in_manufacturing__demargo_.pdf
GamePlan Trading System Review: Professional Trader's Honest Take
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Mobile App Security Testing_ A Comprehensive Guide.pdf
Unlocking AI with Model Context Protocol (MCP)
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Approach and Philosophy of On baking technology
Spectral efficient network and resource selection model in 5G networks
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
Review of recent advances in non-invasive hemoglobin estimation
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
Understanding_Digital_Forensics_Presentation.pptx
Machine learning based COVID-19 study performance prediction

Storage Infrastructure Behind Facebook Messages

  • 1. Storage Infrastructure Behind Facebook Messages HBase/HDFS/ZK/MapReduce Kannan Muthukkaruppan Software Engineer, Facebook Big Data Experiences & Scars, HPTS 2011 Oct 24th, 2011
  • 2. The New Facebook Messages Messages Chats Emails SMS
  • 3. Why we chose HBase ▪ High write throughput ▪ Good random read performance ▪ Horizontal scalability ▪ Automatic Failover ▪ Strong consistency ▪ Benefits of HDFS ▪ Fault tolerant, scalable, checksums, MapReduce ▪ internal dev & ops expertise
  • 4. What do we store in HBase ▪ HBase ▪ Small messages ▪ Message metadata (thread/message indices) ▪ Search index ▪ Haystack (our photo store) ▪ Attachments ▪ Large messages
  • 5. HBase-HDFS System Overview Database Layer HBASE Master Backup Master Region Region Region ... Server Server Server Coordination Service Storage Layer HDFS Zookeeper Quorum Namenode Secondary Namenode ZK ZK ... Peer Peer Datanode Datanode Datanode ...
  • 6. Facebook Messages Architecture User Directory Service Clients (Front End, MTA, etc.) What’s the cell for this user? Cell 2 Cell 1 Cell 3 Cell 1 Application Server Application Server Application Server Attachments HBase/HDFS/Z HBase/HDFS/Z Message, Metadata, K Search HBase/HDFS/Z Index K K Haystack
  • 7. Typical Cluster Layout ▪ Multiple clusters/cells for messaging ▪ 20 servers/rack; 5 or more racks per cluster ▪ Controllers (master/Zookeeper) spread across racks ZooKeeper Peer ZooKeeper Peer ZooKeeper Peer ZooKeeper Peer ZooKeeper Peer HDFS Namenode Backup Namenode Job Tracker HBase Master Backup HBase Master Region Server Region Server Region Server Region Server Region Server Data Node Data Node Data Node Data Node Data Node Task Tracker Task Tracker Task Tracker Task Tracker Task Tracker 19x... 19x... 19x... 19x... 19x... Region Server Region Server Region Server Region Server Region Server Data Node Data Node Data Node Data Node Data Node Task Tracker Task Tracker Task Tracker Task Tracker Task Tracker Rack #1 Rack #2 Rack #3 Rack #4 Rack #5
  • 8. Facebook Messages: Quick Stats ▪ 6B+ messages/day ▪ Traffic to HBase ▪ 75+ Billion R+W ops/day ▪ At peak: 1.5M ops/sec ▪ ~ 55% Read vs. 45% Write ops ▪ Avg write op inserts ~16 records across multiple column families.
  • 9. Facebook Messages: Quick Stats (contd.) ▪ 2PB+ of online data in HBase (6PB+ with replication; excludes backups) ▪ message data, metadata, search index ▪ All data LZO compressed ▪ Growing at 250TB/month
  • 10. Facebook Messages: Quick Stats (contd.) Timeline: ▪ Started in Dec 2009 ▪ Roll out started in Nov 2010 ▪ Fully rolled out by July 2011 (migrated 1B+ accounts from legacy messages!) While in production: ▪ Schema changes: not once, but twice! ▪ Implemented & rolled out HFile V2 and numerous other optimizations in an upward compatible manner!
  • 11. Shadow Testing: Before Rollout ▪ Both product and infrastructure were changing. ▪ Shadows the old messages product + chat while the new one was under development Messages Chat Backfilled data from V1 Frontend Frontend MySQL for % of users and mirrored their production traffic Application Servers Cassandra MySQL ChatLogger HBase/HDFS Cluster Production Shadow
  • 12. Shadow Testing: After Rollout ▪ Shadows new version of the Messages product. ▪ All backend changes go through shadow cluster before prod push New Messages Frontend Application Servers Application Servers HBase/HDFS Cluster HBase/HDFS Cluster Shadow Production
  • 13. Backup/Recovery (V1) • During early phase, concerned about potential bugs in HBase. • Off-line backups: written to HDFS via Scribe • Recovery tools; testing of recovery tools Message Frontend Application Servers Scribe/HDFS Local Datacenter HDFS Remote Datacenter • Merge & Dedup HDFS HBase/HDFS Cluster • Store locally, and • Copy to Remote DC Double log from AppServer & HBase to reduce probability of data loss
  • 14. Backups (V2) ▪ Now, does periodic HFile level backups. ▪ Working on: ▪ Moving to HFile + Commit Log based backups to be able to recover to finer grained points in time ▪ Avoid need to log data to Scribe. ▪ Zero copy (hard link based) fast backups
  • 15. Messages Schema & Evolution ▪ “Actions” (data) Column Family the source of truth ▪ Log of all user actions (addMessage, markAsRead, etc.) ▪ Metadata (thread index, message index, search index) etc. in other column families ▪ Metadata portion of schema underwent 3 changes: ▪ Coarse grained snapshots (early development; rollout up to 1M users) ▪ Hybrid (up to full rollout – 1B+ accounts; 800M+ active) ▪ Fine-grained metadata (after rollout) ▪ MapReduce jobs against production clusters! ▪ Ran in throttled way ▪ Heavy use of HBase bulk import features
  • 16. Write Path Overview Region Server .... Region #2 Region #1 .... ColumnFamily #2 ColumnFamily #1 Memstore HFiles (in HDFS) Write Ahead Log ( in HDFS) Append/Sync
  • 17. Flushes: Memstore -> HFile Region Server .... Region #2 Region #1 .... ColumnFamily #2 ColumnFamily #1 Memstore HFiles flush Data in HFile is sorted; has block index for efficient retrieval
  • 18. Read Path Overview Region Server .... Region #2 Region #1 .... ColumnFamily #2 ColumnFamily #1 Memstore HFiles Get
  • 19. Compactions Region Server .... Region #2 Region #1 .... ColumnFamily #2 ColumnFamily #1 Memstore HFiles
  • 20. Reliability: Early work ▪ HDFS sync support for durability of transactions ▪ Multi-CF transaction atomicity ▪ Several bug fixes in log recovery ▪ New block placement policy in HDFS ▪ To reduce probability of data loss
  • 21. Availability: Early Work ▪ Common reasons for unavailability: ▪ S/W upgrades ▪ Solution: rolling upgrades ▪ Schema Changes ▪ Applications needs new Column Families ▪ Need to change settings for a CF ▪ Solution: online “alter table” ▪ Load balancing or cluster restarts took forever ▪ Upon investigation: stuck waiting for compactions to finish ▪ Solution: Interruptible Compactions!
  • 22. Performance: Early Work ▪ Read optimizations: ▪ Seek optimizations for rows with large number of cells ▪ Bloom Filters ▪ minimize HFile lookups ▪ Timerange hints on HFiles (great for temporal data) ▪ Multigets ▪ Improved handling of compressed HFiles
  • 23. Performance: Compactions Was ▪ Critical for read performance ▪ Old Algorithm: #1. Start from newest file (file 0); include next file if: ▪ size[i] < size[i-1] * C (good!) #2. Always compact at least 4 files, even if rule #1 isn’t met. Solution: Now #1. Compact at least 4 files, but only if eligible files found. #2. Also, new file selection based on summation of sizes. size[i] < (size[0] + size[1] + …size[i-1]) * C
  • 24. Performance: Compactions ▪ More problems! ▪ Read performance dips during Small Compaction Q Big Compaction Q peak ▪ Major compaction storms Compaction Compaction ▪ Large compactions bottleneck Compaction Compaction ▪ Enhancements/fixes: Compaction ▪ Staggered major compactions Compaction ▪ Multi-thread compactions; separate queues for small & big compactions ▪ Aggressive off-peak compactions
  • 25. Metrics, metrics, metrics… ▪ Initially, only had coarse level overall metrics (get/put latency/ops; block cache counters). ▪ Slow query logging ▪ Added per Column Family stats for: ▪ ops counts, latency ▪ block cache usage & hit ratio ▪ memstore usage ▪ on-disk file sizes ▪ file counts ▪ bytes returned, bytes flushed, compaction statistics ▪ stats by block type (data block vs. index blocks vs. bloom blocks, etc.) ▪ bloom filter stats
  • 26. Metrics (contd.) ▪ HBase Master Statistics: ▪ Number of region servers alive ▪ Number of regions ▪ Load balancing statistics ▪ .. ▪ All stats stored in Facebook’s Operational Data Store (ODS). ▪ Lots of ODS dashboards for debugging issues ▪ Side note: ODS planning to use HBase for storage pretty soon!
  • 27. Need to keep up as data grows on you! ▪ Rapidly iterated on several new features while in production: ▪ Block indexes upto 6GB per server! Cluster starts taking longer and longer. Block cache hit ratio on the decline. ▪ Solution: HFile V2 ▪ Multi-level block index, Sharded Bloom Filters ▪ Network pegged after restarts ▪ Solution: Locality on full & rolling restart ▪ High disk utilization during peak ▪ Solution: Several “seek” optimizations to reduce disk IOPS ▪ Lazy Seeks (use time hints to avoid seeking into older HFiles) ▪ Special bloom filter for deletes to avoid additional seek ▪ Utilize off-peak IOPS to do more aggressive compactions during
  • 28. Scares & Scars! ▪ Not without our share of scares and incidents: ▪ s/w bugs. (e.g., deadlocks, incompatible LZO used for bulk imported data, etc.) ▪ found a edge case bug in log recovery as recently as last week! ▪ performance spikes every 6 hours (even off-peak)! ▪ cleanup of HDFS’s Recycle bin was sub-optimal! Needed code and config fix. ▪ transient rack switch failures ▪ Zookeeper leader election took than 10 minutes when one member of the quorum died. Fixed in more recent version of ZK. ▪ HDFS Namenode – SPOF ▪ flapping servers (repeated failures)
  • 29. Scares & Scars! (contd.) ▪ Sometimes, tried things which hadn’t been tested in dark launch! ▪ Added a rack of servers to help with performance issue ▪ Pegged top of the rack network bandwidth! ▪ Had to add the servers at much slower pace. Very manual . ▪ Intelligent load balancing needed to make this more automated. ▪ A high % of issues caught in shadow/stress testing ▪ Lots of alerting mechanisms in place to detect failures cases ▪ Automate recovery for a lots of common ones ▪ Treat alerts on shadow cluster as hi-pri too! ▪ Sharding service across multiple HBase cells also paid off
  • 30. Future Work ▪ Reliability, Availability, Scalability! ▪ Lot of new use cases on top of HBase in the works. ▪ HDFS Namenode HA ▪ Recovering gracefully from transient issues ▪ Fast hot-backups ▪ Delta-encoding in block cache ▪ Replication ▪ Performance (HBase and HDFS) ▪ HBase as a service Multi-tenancy ▪ Features- coprocessors, secondary indices
  • 31. Acknowledgements Work of lot of people spanning HBase, HDFS, Migrations, Backup/Recovery pipeline, and Ops! Karthik Ranganathan Mikhail Bautin Nicolas Spiegelberg Pritam Damania Aravind Menon Prakash Khemani Dhruba Borthakur Doug Porter Hairong Kuang Alex Laslavic Jonathan Gray Kannan Muthukkaruppan Rodrigo Schmidt + Apache HBase Amitanand Aiyer + Interns Guoqiang (Jerry) Chen + Messages FrontEnd Team Liyin Tang + Messages AppServer Team Madhu Vaidya + many more folks…