Membase East Coast Meetups
3 likes974 views
This document provides an overview and demonstration of Membase, a distributed database. It discusses how Membase can be deployed in five minutes or less on a single node or cluster. It is simple to develop applications using Membase's key-value interface without requiring a schema. The document also summarizes several use cases for Membase in domains like social games, advertising, and search/gaming portals. It provides a high-level overview of Membase's architecture, including its clustering functionality and TAP interface. Finally, it briefly previews Membase's NodeCode feature for extending the database with custom modules.
![Membase Datatypes
• byte[]
– Does your data have
1s and 0s?
26
“Any customer can have
a car painted any colour
that he wants so long as
it is black.”
• Items do have flags
– Many clients use flags
–Data type options
• Google protobuf
• Thrift
• Avro](https://image.slidesharecdn.com/membaseroadshow-101110151548-phpapp02/85/Membase-East-Coast-Meetups-26-320.jpg)
Membase East Coast Meetups
- 1. Boston
- 2. Tonight 2 • Membase Overview • Use Cases and Deployment Examples • Membase Architecture • Demo! • Developing with Membase • A Glimpse into the Future
- 4. Membase is a distributed database 4 Membase Servers In the data center Web application server Application user On the administrator console Web application serverWeb application server
- 5. Five minutes or less to a working cluster • Downloads for Linux and Windows • Start with a single node • One button press joins nodes to a cluster Easy to develop against • Just SET and GET – no schema required • Drop it in. 10,000+ existing applications already “speak membase” (via memcached) • Practically every language and application framework is supported, out of the box Easy to manage • One-click failover and cluster rebalancing • Graphical and programmatic interfaces • Configurable alerting Membase is Simple, Fast, Elastic 5
- 6. Membase is Simple, Fast, Elastic 6 Predictable • “Never keep an application waiting” • Quasi-deterministic latency and throughput Low latency • Built-in Memcached technology High throughput • Multi-threaded • Low lock contention • Asynchronous wherever possible • Automatic write de-duplication
- 7. Membase is Simple, Fast, Elastic 7 Zero-downtime elasticity • Spread I/O and data across commodity servers (or VMs) • Consistent performance with linear cost • Dynamic rebalancing of a live cluster All nodes are created equal • No special case nodes • Any node can replace any other node, online • Clone to grow Extensible • Filtered TAP interface provides hook points for external systems (e.g. full-text search, backup, warehouse) • Data bucket – engine API for specialized container types
- 8. Built-in Memcached Caching Layer 8 Memcached Membase Database Membase Cache Membase Database Memcached Mode Membase Mode Fact: Membase development team has also contributed over half of the code to the Memcached project.
- 9. Use Cases
- 10. Leading cloud service (PAAS) provider Over 65,000 hosted applications Membase Server serving over 1,200 Heroku customers (as of June 10, 2010) Deployments Leading Membase 10 Social game leader – FarmVille, Mafia Wars, Café World Over 230 million monthly users Membase Server is the 500,000 ops-per-second database behind FarmVille and Café World
- 11. Ad targeting 11 events profiles, campaigns profiles, real time campaign statistics 40 milliseconds to come up with an answer. 2 3 1
- 12. Search and Gaming Portal 12 Database
- 14. Clustering • Underlying cluster functionality based on erlang OTP • Have a custom, vector clock based way of storing and propagating... – Cluster topology – vBucket mapping • Collect statistics from many nodes of the cluster – Identify hot keys, resource utilization 14
- 16. TAP • A generic, scalable method of streaming mutations from a given server – As data operations arrive, they can be sent to arbitrary TAP receivers • Leverages the existing memcached engine interface, and the non-blocking IO interfaces to send data • Three modes of operation Working set Data Mutations Working set Data Mutations Working set 16
- 17. Membase data flow – under the hood 17 SET request arrives at KEY’s master server Listener-Sender Master server for KEY Replica Server 2 for KEYReplica Server 1 for KEY 3 3 1 SET acknowledgement returned to application2 DiskDisk Disk RAM membasestorageengine DiskDisk Disk 4
- 18. ns_server membase (memcached + membase engine) moxi ns_server vbucketmigrator TAP memcached operations with tap commands memcached operations Client port 11211 memcached operations moxi + Client port 11210 memcached operations REST/comet cluster topology and vbucket map Clients, nodes and other nodes 18
- 19. Data buckets are secure membase “slices” 19 Membase data servers In the data center Web application server Application user On the administrator console Bucket 1 Bucket 2 Aggregate Cluster Memory and Disk Capacity
- 21. Disk > Memory BucketConfiguration mem_high_wat mem_low_wat memory quota 21 Dataset may have many items infrequently accessed. However, memcached has different behavior (LRU) than wanted with membase. Still, traditional (most) RDBMS implementations are not 100% correct for us either. The speed of a miss is very, very important.
- 22. Membase Demo
- 23. 23 Thanks!
- 25. Key-Value Image courtesy http://www.flickr.com/photos/brenda-starr/3509344100/sizes/m/in/photostream/ (with a replica ) 25 Items have: Key Value Expiration Flags CAS (more on this later) Operations include: Get/Set Increment/Decrement Append/Prepend
- 26. Membase Datatypes • byte[] – Does your data have 1s and 0s? 26 “Any customer can have a car painted any colour that he wants so long as it is black.” • Items do have flags – Many clients use flags –Data type options • Google protobuf • Thrift • Avro
- 27. Transactions • Lock == slow me down • CAS operations – Optimistic locking • Very useful with complex datatypes – Imagine two clients trying to update a complex item • You’re likely using CAS already... if you use a CPU 27 User 1 Fail! User 2 Success
- 28. Common Use: Sessions • Web user sessions – Highly read, less writes in many case – Protocol advantage of memcached • Options already for PHP, Ruby and Java • Application state – Not necessarily “entity” style things – May be appropriate for a “cache” pool 28
- 29. Common Use (cache): Rate Limiting • Want to provide API calls into the system – Twitter search – Google search services • Use the atomic increment – Set an item with a unique ID – Upon API request, increment and check • HTTP 420: go away and come back later 29 Your Users Your App ¡Ouch!
- 30. Looking Ahead: NodeCode Frank Weigel, Membase
- 31. Beyond key-value • Indexing/Range Queries • Advanced Data Structures • Sub-object direct manipulation Validation and In-flight transformation • Block mutations failing validation • Enrich or transform objects Connectors (Integrate easily with other systems) • Solr • Hadoop • MySQL NodeCode – Motivation 31
- 32. NodeCode - What is it? Method for extending & customizing Membase Separate code modules Defined interface to datapath and cluster manager Notification on events • Synchronous • Asynchronous 32
- 33. Simple • Packaged modules for easy install and enable • Library of “off the shelf” modules • Module monitoring • Straight forward development and debugging Fast • Low latency/high-throughput • Per-bucket process isolation • Don’t break data manager performance/correctness Elastic • Automatically migrate and instantiate on rebalance • Provide support for migration of internal data • Leverage native Membase engine for internal data storage NodeCode – Drivers 33
- 35. Java only – jar format Must implement minimal module API • Initial module startup • Module removal • Association with bucket NodeCode library helper functions • Register synchronous & asynchronous listeners/callbacks • Register protocol extension/callbacks • Register rebalance callback • Register cluster manager event callbacks • Membase data access NodeCode 1.0 Plans 35
- 37. 37 Q&A
- 38. Attributions • http://commons.wikimedia.org/wiki/ File:Flag_of_China.png • http://commons.wikimedia.org/wiki/ File:Flag_of_South_Korea.svg • http://commons.wikimedia.org/wiki/ File:Flag_of_Japan.svg 38