Ramakrishnan Keynote Ladis2009
- 1. Data Serving in the Cloud Raghu Ramakrishnan Chief Scientist, Audience and Cloud Computing Brian Cooper Adam Silberstein Utkarsh Srivastava Yahoo! Research Joint work with the Sherpa team in Cloud Computing 1
- 2. Outline • Clouds • Scalable serving—the new landscape – Very Large Scale Distributed systems (VLSD) • Yahoo!’s PNUTS/Sherpa • Comparison of several systems – Preview of upcoming Y! Cloud Serving (YCS) benchmark 2
- 3. Types of Cloud Services • Two kinds of cloud services: – Horizontal (“Platform”) Cloud Services • Functionality enabling tenants to build applications or new services on top of the cloud – Functional Cloud Services • Functionality that is useful in and of itself to tenants. E.g., various SaaS instances, such as Saleforce.com; Google Analytics and Yahoo!’s IndexTools; Yahoo! properties aimed at end-users and small businesses, e.g., flickr, Groups, Mail, News, Shopping • Could be built on top of horizontal cloud services or from scratch • Yahoo! has been offering these for a long while (e.g., Mail for SMB, Groups, Flickr, BOSS, Ad exchanges, YQL) 3
- 4. Yahoo! Horizontal Cloud Stack EDGE YCS Horizontal Cloud Services YCPI Brooklyn … Monitoring/Metering/Security WEB Provisioning (Self-serve) VM/OS Horizontal Cloud Services yApache PHP App Engine APP Data Highway VM/OS Horizontal Cloud Services … Serving Grid OPERATIONAL STORAGE Horizontal Cloud Services PNUTS/Sherpa MOBStor … BATCH STORAGE Hadoop Horizontal Cloud Services … 5
- 5. Cloud-Power @ Yahoo! Content Search Optimization Index Machine Learning (e.g. Spam filters) Ads Optimization Attachment Storage Image/Video Storage & Delivery 6
- 6. Yahoo!’s Cloud: Massive Scale, Geo-Footprint • Massive user base and engagement – 500M+ unique users per month – Hundreds of petabyte of storage – Hundreds of billions of objects – Hundred of thousands of requests/sec • Global – Tens of globally distributed data centers – Serving each region at low latencies • Challenging Users – Downtime is not an option (outages cost $millions) – Very variable usage patterns 7
- 7. New in 2010! • SIGMOD and SIGOPS are starting a new annual conference (co-located with SIGMOD in 2010): ACM Symposium on Cloud Computing (SoCC) PC Chairs: Surajit Chaudhuri & Mendel Rosenblum GC: Joe Hellerstein Treasurer: Brian Cooper • Steering committee: Phil Bernstein, Ken Birman, Joe Hellerstein, John Ousterhout, Raghu Ramakrishnan, Doug Terry, John Wilkes 8
- 8. ACID or BASE? Litmus tests are colorful, but the picture is cloudy VERY LARGE SCALE DISTRIBUTED (VLSD) DATA SERVING 9
- 9. Databases and Key-Value Stores http://browsertoolkit.com/fault-tolerance.png 10
- 10. Web Data Management •Warehousing •CRUD •Scan •Point lookups oriented and short workloads scans Large data analysis Structured record •Focus on storage •Index sequential (Hadoop) organized (PNUTS/Sherpa) disk I/O table and •$ per cpu random I/Os cycle •$ per latency •Object retrieval and streaming Blob storage •Scalable file (MObStor) storage •$ per GB storage & bandwidth 11
- 11. The World Has Changed • Web serving applications need: – Scalability! • Preferably elastic – Flexible schemas – Geographic distribution – High availability – Reliable storage • Web serving applications can do without: – Complicated queries – Strong transactions • But some form of consistency is still desirable 12
- 12. Typical Applications • User logins and profiles – Including changes that must not be lost! • But single-record “transactions” suffice • Events – Alerts (e.g., news, price changes) – Social network activity (e.g., user goes offline) – Ad clicks, article clicks • Application-specific data – Postings in message board – Uploaded photos, tags – Shopping carts 13
- 13. Data Serving in the Y! Cloud FredsList.com application DECLARE DATASET Listings AS 1234323, 5523442, 32138, ( ID String PRIMARY KEY, camera, Category String, transportation, childcare, Description Text ) For sale: one Nanny Nikon bicycle, barely available in D40, used San Jose USD 300 ALTER Listings MAKE CACHEABLE Simple Web Service API’s Grid PNUTS / Foreign key MObStor memcached Vespa SHERPA photo → listing Storage Compute Database Search Caching Tribble Batch export Messaging 14
- 14. VLSD Data Serving Stores • Must partition data across machines – How are partitions determined? – Can partitions be changed easily? (Affects elasticity) – How are read/update requests routed? – Range selections? Can requests span machines? • Availability: What failures are handled? – With what semantic guarantees on data access? • (How) Is data replicated? – Sync or async? Consistency model? Local or geo? • How are updates made durable? • How is data stored on a single machine? 15
- 15. The CAP Theorem • You have to give up one of the following in a distributed system (Brewer, PODC 2000; Gilbert/Lynch, SIGACT News 2002): – Consistency of data • Think serializability – Availability • Pinging a live node should produce results – Partition tolerance • Live nodes should not be blocked by partitions 16
- 16. Approaches to CAP • “BASE” – No ACID, use a single version of DB, reconcile later • Defer transaction commit – Until partitions fixed and distr xact can run • Eventual consistency (e.g., Amazon Dynamo) – Eventually, all copies of an object converge • Restrict transactions (e.g., Sharded MySQL) – 1-M/c Xacts: Objects in xact are on the same machine – 1-Object Xacts: Xact can only read/write 1 object • Object timelines (PNUTS) http://www.julianbrowne.com/article/viewer/brewers-cap-theorem 17
- 17. “I want a big, virtual database” “What I want is a robust, high performance virtual relational database that runs transparently over a cluster, nodes dropping in and out of service at will, read-write replication and data migration all done automatically. I want to be able to install a database on a server cloud and use it like it was all running on one machine.” -- Greg Linden’s blog 18 18
- 18. Y! CCDI PNUTS / SHERPA To Help You Scale Your Mountains of Data 19
- 19. Yahoo! Serving Storage Problem – Small records – 100KB or less – Structured records – Lots of fields, evolving – Extreme data scale - Tens of TB – Extreme request scale - Tens of thousands of requests/sec – Low latency globally - 20+ datacenters worldwide – High Availability - Outages cost $millions – Variable usage patterns - Applications and users change 20 20
- 20. What is PNUTS/Sherpa? CREATE TABLE Parts ( ID VARCHAR, StockNumber INT, Status VARCHAR A 42342 E A 42342 E B 42521 W … B 42521 W C 66354 W C 66354 W ) D 12352 E D 12352 E E 75656 C E 75656 C F 15677 E F 15677 E Structured, flexible schema A 42342 E Parallel database B 42521 W Geographic replication C 66354 W D 12352 E E 75656 C F 15677 E Hosted, managed infrastructure 21 21
- 21. What Will It Become? A 42342 E A 42342 E B 42521 W B 42521 W C 66354 W C 66354 W D 12352 E D 12352 E E 75656 C E 75656 C F 15677 E F 15677 E Indexes and views A 42342 E B 42521 W C 66354 W D 12352 E E 75656 C F 15677 E 22
- 22. Design Goals Scalability Consistency • Thousands of machines • Per-record guarantees • Easy to add capacity • Timeline model • Restrict query language to avoid costly queries • Option to relax if needed Geographic replication Multiple access paths • Asynchronous replication around the globe • Hash table, ordered table • Low-latency local access • Primary, secondary access High availability and fault tolerance Hosted service • Automatically recover from failures • Applications plug and play • Serve reads and writes despite failures • Share operational cost 23 23
- 23. Technology Elements Applications PNUTS API Tabular API PNUTS • Query planning and execution • Index maintenance YCA: Authorization Distributed infrastructure for tabular data • Data partitioning • Update consistency • Replication YDOT FS YDHT FS • Ordered tables • Hash tables Tribble Zookeeper • Pub/sub messaging • Consistency service 24 24
- 24. PNUTS: Key Components • Maintains map from • Caches the maps from the TC database.table.key-to- • Routes client requests to tablet-to-SU correct SU • Provides load balancing • Stores records • Services get/set/delete requests 25 25
- 25. Detailed Architecture Local region Remote regions Clients REST API Routers Tribble Tablet Controller Storage units 26 26
- 26. DATA MODEL 27 27
- 27. Data Manipulation • Per-record operations – Get – Set – Delete • Multi-record operations – Multiget – Scan – Getrange • Web service (RESTful) API 28 28
- 28. Tablets—Hash Table Name Description Price 0x0000 Grape Grapes are good to eat $12 Lime Limes are green $9 Apple Apple is wisdom $1 Strawberry Strawberry shortcake $900 0x2AF3 Orange Arrgh! Don’t get scurvy! $2 Avocado But at what price? $3 Lemon How much did you pay for this lemon? $1 Tomato Is this a vegetable? $14 0x911F Banana The perfect fruit $2 Kiwi New Zealand $8 0xFFFF 29 29
- 29. Tablets—Ordered Table Name Description Price A Apple Apple is wisdom $1 Avocado But at what price? $3 Banana The perfect fruit $2 Grape Grapes are good to eat $12 H Kiwi New Zealand $8 Lemon How much did you pay for this lemon? $1 Lime Limes are green $9 Orange Arrgh! Don’t get scurvy! $2 Q Strawberry Strawberry shortcake $900 Tomato Is this a vegetable? $14 Z 30 30
- 30. Flexible Schema Posted date Listing id Item Price Color Condition 6/1/07 424252 Couch $570 Good 6/1/07 763245 Bike $86 6/3/07 211242 Car $1123 Red Fair 6/5/07 421133 Lamp $15 31
- 31. Primary vs. Secondary Access Primary table Posted date Listing id Item Price 6/1/07 424252 Couch $570 6/1/07 763245 Bike $86 6/3/07 211242 Car $1123 6/5/07 421133 Lamp $15 Secondary index Price Posted date Listing id 15 6/5/07 421133 86 6/1/07 763245 570 6/1/07 424252 1123 6/3/07 211242 32 Planned functionality 32
- 32. Index Maintenance • How to have lots of interesting indexes and views, without killing performance? • Solution: Asynchrony! – Indexes/views updated asynchronously when base table updated 33
- 33. PROCESSING READS & UPDATES 34 34
- 34. Updates 8 1 Sequence # for key k Write key k Routers Message brokers 3 Write key k 7 2 Write key k 4 Sequence # for key k 5 SUCCESS SU SU SU 6 Write key k 35 35
- 35. Accessing Data 1 4 Record for key k Get key k 2 3 Record for key k Get key k SU SU SU 36 36
- 36. Bulk Read 1 {k1, k2, … kn} Get k1 2 Get k2 Get k3 Scatter/ SU SU SU gather server 37 37
- 37. Range Queries in YDOT • Clustered, ordered retrieval of records Apple Storage unit 1 Avocado Grapefruit…Pear? Canteloupe Banana Storage unit 3 Grapefruit…Lime? Blueberry Lime Canteloupe Storage unit 2 Grape Strawberry Kiwi Storage unit 1 Lime…Pear? Lemon Lime Router Mango Orange Strawberry Apple Strawberry Lime Canteloupe Tomato Avocado Tomato Mango Grape Watermelon Banana Watermelon Orange Kiwi Blueberry Lemon Storage unit 1 Storage unit 2 Storage unit 3 38
- 38. Bulk Load in YDOT • YDOT bulk inserts can cause performance hotspots • Solution: preallocate tablets 39
- 39. ASYNCHRONOUS REPLICATION AND CONSISTENCY 40 40
- 40. Asynchronous Replication 41 41
- 41. Consistency Model • If copies are asynchronously updated, what can we say about stale copies? – ACID guarantees require synchronous updts – Eventual consistency: Copies can drift apart, but will eventually converge if the system is allowed to quiesce • To what value will copies converge? • Do systems ever “quiesce”? – Is there any middle ground? 42
- 42. Example: Social Alice West East Record Timeline User Status ___ (Alice logs on) Alice ___ User Status Busy (Network fault, Alice Busy updt goes to East) User Status User Status Free Alice Busy Alice Free User Status User Status Free Alice ??? Alice ??? 43
- 43. PNUTS Consistency Model • Goal: Make it easier for applications to reason about updates and cope with asynchrony • What happens to a record with primary key “Alice”? Record Update Update Update Update Update Delete inserted Update Update v. 1 v. 2 v. 3 v. 4 v. 5 v. 6 v. 7 v. 8 Generation 1 Time Time As the record is updated, copies may get out of sync. 44 44
- 44. PNUTS Consistency Model Write Stale version Stale version Current version v. 1 v. 2 v. 3 v. 4 v. 5 v. 6 v. 7 v. 8 Generation 1 Time Achieved via per-record primary copy protocol (To maximize availability, record masterships automaticlly transferred if site fails) Can be selectively weakened to eventual consistency (local writes that are reconciled using version vectors) 45 45
- 45. PNUTS Consistency Model Write if = v.7 ERROR Stale version Stale version Current version v. 1 v. 2 v. 3 v. 4 v. 5 v. 6 v. 7 v. 8 Generation 1 Time Test-and-set writes facilitate per-record transactions 46 46
- 46. PNUTS Consistency Model Read Stale version Stale version Current version v. 1 v. 2 v. 3 v. 4 v. 5 v. 6 v. 7 v. 8 Generation 1 Time In general, reads are served using a local copy 47 47
- 47. PNUTS Consistency Model Read up-to-date Stale version Stale version Current version v. 1 v. 2 v. 3 v. 4 v. 5 v. 6 v. 7 v. 8 Generation 1 Time But application can request and get current version 48 48
- 48. PNUTS Consistency Model Read ≥ v.6 Stale version Stale version Current version v. 1 v. 2 v. 3 v. 4 v. 5 v. 6 v. 7 v. 8 Generation 1 Time Or variations such as “read forward”—while copies may lag the master record, every copy goes through the same sequence of changes 49 49
- 49. OPERABILITY 50 50
- 50. Distribution 6/1/07 424252 Couch $570 Data shuffling for load balancing Distribution for parallelism 6/1/07 256623 Car $1123 6/2/07 636353 Bike $86 6/5/07 662113 Chair $10 6/7/07 121113 Lamp $19 6/9/07 887734 Bike $56 6/11/07 252111 Scooter $18 6/11/07 116458 Hammer $8000 Server 1 Server 2 Server 3 Server 4 51 51
- 51. Tablet Splitting and Balancing Each storage unit has many tablets (horizontal partitions of the table) Storage unit may become a hotspot Storage unit Tablet Overfull tablets split Tablets may grow over time Shed load by moving tablets to other servers 52 52
- 52. Consistency Techniques • Per-record mastering – Each record is assigned a “master region” • May differ between records – Updates to the record forwarded to the master region – Ensures consistent ordering of updates • Tablet-level mastering – Each tablet is assigned a “master region” – Inserts and deletes of records forwarded to the master region – Master region decides tablet splits • These details are hidden from the application – Except for the latency impact! 53
- 53. Mastering A 42342 E B 42521 E W C 66354 W D 12352 E E 75656 C F 15677 E A 42342 E B 42521 W E C 66354 W D 12352 E E 75656 C F 15677 E A 42342 E B 42521 W C 66354 W D 12352 E E 75656 C F 15677 E 54 54
- 54. Record vs. Tablet Master Record master serializes updates A 42342 E B 42521 E W C 66354 W D 12352 E E 75656 C F 15677 E A 42342 E Tablet master serializes inserts B 42521 W E C 66354 W D 12352 E E 75656 C F 15677 E A 42342 E B 42521 W C 66354 W D 12352 E E 75656 C F 15677 E 55 55
- 55. Coping With Failures X X A 42342 E B 42521 W C 66354 W D 12352 E E 75656 C OVERRIDE W → E F 15677 E A 42342 E B 42521 W C 66354 W D 12352 E E 75656 C F 15677 E A 42342 E B 42521 W C 66354 W D 12352 E E 75656 C F 15677 E 56 56
- 56. Further PNutty Reading Efficient Bulk Insertion into a Distributed Ordered Table (SIGMOD 2008) Adam Silberstein, Brian Cooper, Utkarsh Srivastava, Erik Vee, Ramana Yerneni, Raghu Ramakrishnan PNUTS: Yahoo!'s Hosted Data Serving Platform (VLDB 2008) Brian Cooper, Raghu Ramakrishnan, Utkarsh Srivastava, Adam Silberstein, Phil Bohannon, Hans-Arno Jacobsen, Nick Puz, Daniel Weaver, Ramana Yerneni Asynchronous View Maintenance for VLSD Databases (SIGMOD 2009) Parag Agrawal, Adam Silberstein, Brian F. Cooper, Utkarsh Srivastava and Raghu Ramakrishnan Cloud Storage Design in a PNUTShell Brian F. Cooper, Raghu Ramakrishnan, and Utkarsh Srivastava Beautiful Data, O’Reilly Media, 2009 Adaptively Parallelizing Distributed Range Queries (VLDB 2009) Ymir Vigfusson, Adam Silberstein, Brian Cooper, Rodrigo Fonseca 57
- 57. Green Apples and Red Apples COMPARING SOME CLOUD SERVING STORES 58
- 58. Motivation • Many “cloud DB” and “nosql” systems out there – PNUTS – BigTable • HBase, Hypertable, HTable – Azure – Cassandra – Megastore (behind Google AppEngine) – Amazon Web Services • S3, SimpleDB, EBS – And more: CouchDB, Voldemort, etc. • How do they compare? – Feature tradeoffs – Performance tradeoffs – Not clear! 59
- 59. The Contestants • Baseline: Sharded MySQL – Horizontally partition data among MySQL servers • PNUTS/Sherpa – Yahoo!’s cloud database • Cassandra – BigTable + Dynamo • HBase – BigTable + Hadoop 60
- 60. SHARDED MYSQL 61 61
- 61. Architecture • Our own implementation of sharding Client Client Client Client Client Shard Server Shard Server Shard Server Shard Server MySQL MySQL MySQL MySQL 62
- 62. Shard Server • Server is Apache + plugin + MySQL – MySQL schema: key varchar(255), value mediumtext – Flexible schema: value is blob of key/value pairs • Why not direct to MySQL? – Flexible schema means an update is: • Read record from MySQL • Apply changes • Write record to MySQL – Shard server means the read is local • No need to pass whole record over network to change one field Apache Apache Apache Apache … Apache (100 processes) MySQL 63
- 63. Client • Application plus shard client • Shard client – Loads config file of servers – Hashes record key – Chooses server responsible for hash range – Forwards query to server Client Application Q? Shard client Hash() Server map CURL 64
- 64. Pros and Cons • Pros – Simple – “Infinitely” scalable – Low latency – Geo-replication • Cons – Not elastic (Resharding is hard) – Poor support for load balancing – Failover? (Adds complexity) – Replication unreliable (Async log shipping) 65
- 65. Azure SDS • Cloud of SQL Server instances • App partitions data into instance-sized pieces – Transactions and queries within an instance SDS Instance Data Storage Per-field indexing 66
- 66. Google MegaStore • Transactions across entity groups – Entity-group: hierarchically linked records • Ramakris • Ramakris.preferences • Ramakris.posts • Ramakris.posts.aug-24-09 – Can transactionally update multiple records within an entity group • Records may be on different servers • Use Paxos to ensure ACID, deal with server failures – Can join records within an entity group – Reportedly, moving to ordered, async replication w/o ACID • Other details – Built on top of BigTable – Supports schemas, column partitioning, some indexing Phil Bernstein, http://perspectives.mvdirona.com/2008/07/10/GoogleMegastore.aspx 67
- 67. PNUTS 68 68
- 68. Architecture Clients REST API Routers Tablet controller Log servers Storage units 69
- 69. Routers • Direct requests to storage unit – Decouple client from storage layer • Easier to move data, add/remove servers, etc. – Tradeoff: Some latency to get increased flexibility Router Y! Traffic Server PNUTS Router plugin 70
- 70. Msg/Log Server • Topic-based, reliable publish/subscribe – Provides reliable logging – Provides intra- and inter-datacenter replication Log server Log server Pub/sub hub Pub/sub hub Disk Disk 71
- 71. Pros and Cons • Pros – Reliable geo-replication – Scalable consistency model – Elastic scaling – Easy load balancing • Cons – System complexity relative to sharded MySQL to support geo-replication, consistency, etc. – Latency added by router 72
- 72. HBASE 73 73
- 73. Architecture Client Client Client Client Client Java Client REST API HBaseMaster HRegionServer HRegionServer HRegionServer HRegionServer Disk Disk Disk Disk 74
- 74. HRegion Server • Records partitioned by column family into HStores – Each HStore contains many MapFiles • All writes to HStore applied to single memcache • Reads consult MapFiles and memcache • Memcaches flushed as MapFiles (HDFS files) when full • Compactions limit number of MapFiles HRegionServer writes Memcache Flush to disk HStore reads MapFiles 75
- 75. Pros and Cons • Pros – Log-based storage for high write throughput – Elastic scaling – Easy load balancing – Column storage for OLAP workloads • Cons – Writes not immediately persisted to disk – Reads cross multiple disk, memory locations – No geo-replication – Latency/bottleneck of HBaseMaster when using REST 76
- 76. CASSANDRA 77 77
- 77. Architecture • Facebook’s storage system – BigTable data model – Dynamo partitioning and consistency model – Peer-to-peer architecture Client Client Client Client Client Cassandra node Cassandra node Cassandra node Cassandra node Disk Disk Disk Disk 78
- 78. Routing • Consistent hashing, like Dynamo or Chord – Server position = hash(serverid) – Content position = hash(contentid) – Server responsible for all content in a hash interval Responsible hash interval Server 79
- 79. Cassandra Server • Writes go to log and memory table • Periodically memory table merged with disk table Cassandra node Update RAM Memtable (later) Log Disk SSTable file 80
- 80. Pros and Cons • Pros – Elastic scalability – Easy management • Peer-to-peer configuration – BigTable model is nice • Flexible schema, column groups for partitioning, versioning, etc. – Eventual consistency is scalable • Cons – Eventual consistency is hard to program against – No built-in support for geo-replication • Gossip can work, but is not really optimized for, cross-datacenter – Load balancing? • Consistent hashing limits options – System complexity • P2P systems are complex; have complex corner cases 81
- 81. Cassandra Findings • Tunable memtable size – Can have large memtable flushed less frequently, or small memtable flushed frequently – Tradeoff is throughput versus recovery time • Larger memtable will require fewer flushes, but will take a long time to recover after a failure • With 1GB memtable: 45 mins to 1 hour to restart • Can turn off log flushing – Risk loss of durability • Replication is still synchronous with the write – Durable if updates propagated to other servers that don’t fail 82
- 82. Thanks to Ryan Rawson & J.D. Cryans for advice on HBase configuration, and Jonathan Ellis on Cassandra NUMBERS 83 83
- 83. Overview • Setup – Six server-class machines • 8 cores (2 x quadcore) 2.5 GHz CPUs, RHEL 4, Gigabit ethernet • 8 GB RAM • 6 x 146GB 15K RPM SAS drives in RAID 1+0 – Plus extra machines for clients, routers, controllers, etc. • Workloads – 120 million 1 KB records = 20 GB per server – Write heavy workload: 50/50 read/update – Read heavy workload: 95/5 read/update • Metrics – Latency versus throughput curves • Caveats – Write performance would be improved for PNUTS, Sharded ,and Cassandra with a dedicated log disk – We tuned each system as well as we knew how 84
- 84. Results Read latency vs. actual throughput, 95/5 read/write 140 120 100 Latency (ms) 80 60 40 20 0 0 1000 2000 3000 4000 5000 6000 7000 Throughput (ops/sec) PNUTS Sharded Cassandra Hbase 85
- 85. Results Write latency vs. actual throughput, 95/5 read/write 180 160 140 120 Latency (ms) 100 80 60 40 20 0 0 1000 2000 3000 4000 5000 6000 7000 Throughput (ops/sec) PNUTS Sharded Cassandra HBase 86
- 86. Results 87
- 87. Results 88
- 88. Qualitative Comparison • Storage Layer – File Based: HBase, Cassandra – MySQL: PNUTS, Sharded MySQL • Write Persistence – Writes committed synchronously to disk: PNUTS, Cassandra, Sharded MySQL – Writes flushed asynchronously to disk: HBase (current version) • Read Pattern – Find record in MySQL (disk or buffer pool): PNUTS, Sharded MySQL – Find record and deltas in memory and on disk: HBase, Cassandra 89
- 89. Qualitative Comparison • Replication (not yet utilized in benchmarks) – Intra-region: HBase, Cassandra – Inter- and intra-region: PNUTS – Inter- and intra-region: MySQL (but not guaranteed) • Mapping record to server – Router: PNUTS, HBase (with REST API) – Client holds mapping: HBase (java library), Sharded MySQL – P2P: Cassandra 90
- 90. YCS Benchmark • Will distribute Java application, plus extensible benchmark suite – Many systems have Java APIs – Non-Java systems can support REST Command-line parameters • DB to use • Target throughput • Number of threads •… YCSB client Cloud DB Scenario file DB client • R/W mix Client • Record size Scenario threads • Data set executor •… Stats Extensible: define new scenarios Extensible: plug in new clients 91
- 91. Benchmark Tiers • Tier 1: Cluster performance – Tests fundamental design independent of replication/scale- out/availability • Tier 2: Replication – Local replication – Geographic replication • Tier 3: Scale out – Scale out the number of machines by a factor of 10 – Scale out the number of replicas • Tier 4: Availability – Kill system components If you’re interested in this, please contact me or cooperb@yahoo-inc.com 92
- 92. Hadoop Sherpa Shadoop Adam Silberstein and the Sherpa team in Y! Research and CCDI 93
- 93. Sherpa vs. HDFS • Sherpa optimized for low-latency record-level access: B-trees • HDFS optimized for batch-oriented access: File system • At 50,000 feet the data stores appear similar Is Sherpa a reasonable backend for Hadoop for Sherpa users? Client Client Router Name node Sherpa HDFS Shadoop Goals 1. Provide a batch-processing framework for Sherpa using Hadoop 2. Minimize/characterize penalty for reading/writing Sherpa vs. HDFS 94
- 94. Building Shadoop Input Output Sherpa Hadoop Tasks Hadoop Tasks Sherpa Record Map or Reduce scan(0x2-0x4) Reader Map scan(0xa-0xc) set set router scan(0x8-0xa) set set set scan(0x0-0x2) set set scan(0xc-0xe) 1. Split Sherpa table into hash ranges 1. Call Sherpa set to write output 2. Each Hadoop task assigned a range 1. No DOT range-clustering 3. Task uses Sherpa scan to retrieve 2. Record at a time insert records in range 4. Task feeds scan results and feeds records to map function 95
- 95. Use Cases HDFS Sherpa Sherpa Sherpa Bulk Load Sherpa Tables Migrate Sherpa Tables Source data in HDFS Between farms, code versions Map/Reduce, Pig Map/Reduce, Pig Output is servable content Output is servable content Sherpa HDFS Map/Reduce, Pig Output is input to further analysis Sherpa Sherpa reads Sherpa HDFS HDFS Validator Ops Tool Sherpa as Hadoop “cache” Ensure replicas are consistent Standard Hadoop in HDFS, using Sherpa as a shared cache 96
- 96. SYSTEMS IN CONTEXT 98 98
- 97. Application Design Space Get a few things Sherpa MObStor YMDB MySQL Oracle Filer BigTable Scan Hadoop Everest everything Records Files 99 99
- 98. Comparison Matrix Partitioning Availability Replication Storage Consistency Sync/async Local/geo Hash/sort Durability Dynamic Reads/ handled Failures Routing writes During failure Colo+ Read+ Local+ Timeline + Double Buffer PNUTS H+S Rtr Async Y server write geo Eventual WAL pages Local+ WAL Buffer MySQL H+S N Cli Colo+ Read Async ACID pages server nearby Read+ Local+ N/A (no Triple Y Colo+ Sync replication Files HDFS Other Rtr server write nearby updates) Colo+ Read+ Local+ Multi- Triple LSM/ BigTable S Y Rtr write Sync nearby version replication SSTable server Colo+ Read+ Local+ Buffer Dynamo H Y P2P write Async nearby Eventual WAL pages server Read+ Sync+ Local+ Triple LSM/ Cassandra H+S Y P2P Colo+ Eventual WAL SSTable server write Async nearby Megastore Colo+ Read+ Local+ Triple LSM/ S Y Rtr server write Sync nearby ACID/ replication SSTable other Azure S N Cli Server Read+ Buffer write Sync Local ACID WAL pages 100 100
- 99. HDFS Oracle Sherpa Y! UDB MySQL Dynamo BigTable Cassandra Elastic Operability Availability Global low latency Structured access Updates Consistency model Comparison Matrix SQL/ACID 101 101
- 100. QUESTIONS? 102 102