Building a distributed Key-Value store with Cassandra
- 1. Building a Key-Value Store with Cassandra Kiwi PyCon 2010 Aaron Morton @aaronmorton Weta Digital 1
- 2. Why Cassandra? • Part of a larger project started earlier this year to build new systems for code running on the render farm of 35,000 cores • Larger project goals were Scalability, Reliability, Flexible Schema 2
- 3. How about MySQL ? • It works. But... • Schema changes • Write redundancy • Query language mismatch • So went looking for the right tool for the job 3
- 4. Redis ? • Fast, flexible. But... • Single core limit • Replication, but no cluster (itʼs coming) • Limited support options 4
- 5. Couch DB ? • Schema free, scalable (sort of), redundant (sort of). But... • Single write thread limit • Replication, but no cluster (itʼs coming) • Low consistency with asynchronous replication 5
- 6. Cassandra ? • Just right, perhaps. Letʼs see... • Highly available • Tuneable synchronous replication • Scalable writes and reads • Schema free (sort of) • Lots of new mistakes to be made 6
- 7. Availability • Row data is kept together and replicated around the cluster • Replication Factor is configurable • Partitioner determines the position of a row key in the distributed hash table • Replication Strategy determines where in the cluster to place the replicas 7
- 8. Consistency • Each read or write request specifies a Consistency Level • Individual nodes may be inconsistent with respect to others • Reads may give consistent results while some nodes have inconsistent values • The entire cluster will eventually mode to a state where there is one version of each 8
- 9. Consistency • R + W > N • R = Read Consistency • W = Write Consistency • N = Replication Factor 9
- 10. Scale • Distributed hash table • Scale throughput and capacity with more nodes, more disk, more memory • Adding or removing nodes is an online operation • Gossip based protocol for discovery 10
- 11. Data Model • Column orientated • Denormalise • Cassandra in an index building machine • Simple explanation: a row has a key and stores an ordered hash in one or more Column Families 11
- 12. Data Model • Keyspace • Row / Key • Column Family or Super Column Family • Column 12
- 13. Data Model User CF Posts SCF Fred email:fred@... dob:04/03 post_1:{ title: foo, body: bar} Bob email:bob post_100:{ title: monkeys, body: naughty} 13
- 14. API • Thrift • Avro (beta) • Auto generated bindings for many languages • Stateful connections • Python wrappers pycassa, Telephus (twisted) 14
- 15. API • Client supplied time stamp for all mutations • Client supplied Consistency Level for all mutations and reads 15
- 16. API • insert (key, column_family, super_column, column, value) • get(key, column_family, super_column, column) • remove(key, column_family, super_column, column) 16
- 17. API • Slicing columns or super columns • list of names • start, finish, count, reversed • get_slice() to slice one row • multiget_slice() to slice multiple rows • get_range_slices() to slice rows and columns 17
- 18. API • Slicing keys • start key, finish key, count • Partitioner effects key order • get_range_slices() to slice rows and columns 18
- 19. API • batch_mutate() • multiple rows and CFʼs • delete or insert / update • Individual mutations are atomic • Request is not atomic, no rollback 19
- 21. Our Application • Similar to Amazon S3. • REST API. • Databases, Buckets, Keys+Values. 21
- 22. Our Column Families • Database (super) • Bucket (super) • Bucket Index • Object • Object Index (super) 22
- 24. PUT Object • /bucket/object • batch_mutate() • one row in Objects CF with columns for meta and the body • one column in ObjectIndex CF row for the bucket 24
- 25. List Objects • /bucket_name?start=foo • get_slice() • for the bucket row in ObjectIndex CF • if needed, multiget_slice() to “join” to the Object CF 25
- 26. Delete Bucket • /bucket_name • get_slice() on ObjectIndex CF • batch_mutate() to delete Object CF and ObjectIndex CF • delete Bucket CF row 26