High order bits from cassandra & hadoop
- 1. High-order bits from Cassandra & Hadoopsrisatishambati@srisatish
- 2. Thank You! svccg in first page of search results for “cloud” on google!
- 4. pointsUsecasesWhy cassandra?Usecase: Hadoop, BriskFUD:Consistency Why facebook is not using Cassandra?Anti-patternsCommunity, Code, ToolsQ&A
- 5. Users. Netflix.Key by Customer, read-heavyKey by Customer:Movie, write-heavy
- 6. TimeSeries: (several customers)periodic readings: dev0, dev1…deviceID:metric:timestamp ->valueMetrics typically way larger dataset than users.
- 11. reads localdc1dc2Replication: Multi-datacenterMulti-region ec2, awsMulti-availability zones
- 12. 4.21.2011, Amazon Web Services outage:“Movie marathons on Netflix awaiting AWS to come back up.” #ec2disabled
- 13. 4.21.2011, Amazon Web Services outage:Netflix was running on AWS.
- 14. fast durable writes. fast reads.
- 16. WritesSequential, append-only.~1-5msOn cloud: ephemeral disks rock!
- 17. Reads LocalKey & row caches, (also, jna-based 0xffheap)indexes, materialized
- 18. Reads LocalKey & row caches, (also, jna-based 0xffheap)indexes, materializedssds: improved read performance!
- 19. Distribution between nodes GossipAnti-entropyFailure-detectorL i g h t w e i g h t
- 20. Clients: cql, thriftpycassa, phpcassa hector, pelops(scala, ruby, clojure)
- 21. Usecase #3: hadoopHdfs cassandra hiveLogs stats analytics
- 27. jobtracker, tasktrackerhdfs: namenode, datanode
- 28. clouderaamazon: elastic map reducehortonworksmapRbrisk
- 29. Tools & Analytics Hive, Pig, RKarmasphereDatameer… dozens of stealth startups!
- 33. Use column families (tables)inodesblock
- 34. near-real time hadoopLow latency: cassandra_dc nodesBatch Analytics: brisk_dc nodes
- 35. FUD, acronym: fear, uncertainty, doubt.
- 36. Consistency: R + W > N ORACLE, 2-node: R=1, W=2, N=2,(T=2)DNS* N is replication factor. Not to be confused with T=total #of nodes
- 37. Tune-able, flexibility.For High Consistency: read:quorum, write:quorumFor High Availability: high W, low R.
- 39. Inbox Search: 600+cores.120+TB (2008)Went from 100-500m users.Average NoSQL deployment size: ~6-12 nodes.
- 40. Usecase #5: searchApache Solr + Cassandra = SolandraOther inbox/file Searches:xobni, c3github.com/tjake/solandra
- 41. “Eventual consistency is harder to program.”mostly immutable data.complex systems at scale.
- 42. Miscellaneous,Myth: data-loss, partial rows.writes are durable.
- 44. Anti-Patterns for cloudebsjvm, virtualizedsingle region
- 45. Three good reasons for Cassandra...
- 46. ToolsAMIs, OpsCenter, DataStaxAppDynamicsNetflix just builds AMIs for deployment!
- 47. B e a u t i f u l C 0 d e= new code(); //less is more~90k.java.concurrent.@annotate. bloomfilters, merkletrees.non-blocking, staged-event-driven.bigtable, dynamo.
- 48. Current & Future Focus:Distributed Counters, CQL.Simple client.operational smoothening. compaction.
- 49. CommunityRobust. Rapid. #Professional support from DataStax.Filesysteminnovatin from Acunuengineers: independent,startups, large companies, Rackspace, Twitter, Netflix..Come join the efforts!
- 51. Usecase #4: first NoSQL, then scale!simpledb Cassandramongodb Cassandra
- 55. Copyright: xkcd
- 56. Copyright: plantoys… more than one way to do it!
- 57. Summary -high scale peer-to-peer datastorebest friend for multi-region, multi-zone availability.Hadoop – HDFS engulfing the DataWorld
- 58. Q&A@srisatish