Introduction to Hadoop, HBase, and NoSQL
15 likes7,475 views
The document is a presentation on NoSQL databases given by Nick Dimiduk. It begins with an introduction of the speaker and their background. The presentation then covers what NoSQL is not, the motivations for NoSQL databases, an overview of Hadoop and its components, and a description of HBase as a structured, distributed database built on Hadoop.
Introduction to Hadoop, HBase, and NoSQL
- 1. Nick Dimiduk - @xefyr Founder, Drawn to Scale nick@drawntoscalehq.com April 28, 2010
- 2. Agenda what NoSQL is not motivation Hadoop HBase
- 3. whoami Computer Science & Engineering at Ohio State: Artificial Intelligence, Programming Languages, Systems Engineering Applied Technical Systems: Hierarchical, non-relational data storage and analysis systems (no-sql before there was NoSQL). Information Retrieval, Wire Serialization/RPC (before there was Thrift/Avro), Data Visualization (GB's) Visible Technologies: Social Media Storage, Processing, Analytics. Monitoring, Engagement, Warehousing, and BI. (TB's) Drawn to Scale: Big Data Storage, Processing, Retrieval, Analytics (TB's, PB's)
- 4. Agenda what NoSQL is not motivation Hadoop HBase
- 5. What NoSQL is not. movement
- 6. What NoSQL is not. movement - no ANSI NoSQL-2010 one-size-fits-all
- 9. It’s about Choice! http://www.flickr.com/photos/zakh/337938459/
- 10. What NoSQL is not. movement - no ANSI NoSQL-2010 one-size-fits-all - it’s about choice silver bullet
- 11. What NoSQL is not. movement - no ANSI NoSQL-2010 one-size-fits-all - it’s about choice silver bullet - guarantees are hard
- 12. Agenda what NoSQL is not motivation Hadoop HBase
- 13. motivation more, More, MORE Data!
- 14. motivation more, More, MORE Data! ACID Burns
- 15. motivation more, More, MORE Data! ACID Burns BASE is good enough
- 16. motivation more, More, MORE Data! ACID Burns BASE is good enough Life’s too short
- 17. motivation more, More, MORE Data! ACID Burns BASE is good enough Life’s too short
- 19. “typical” application Data Server Village People App Server
- 20. growing pains Data Server Villages of People App Servers
- 21. vertical partitioning Data Server Villages of People App Servers Data Server Villages of People App Servers
- 22. vertical partitioning Data Server Villages of People Data Server Villages of People App Servers App Servers Data Server Villages of People Data Server Villages of People App Servers App Servers
- 23. vertical partitioning Data Server Villages of People App Servers Data Server Villages of People App Servers
- 25. growing pains Data Servers Villages of People App Servers
- 26. horizontal partitioning Villages of People
- 27. horizontal partitioning Villages of People
- 28. horizontal partitioning Villages of People Data Layer Application Layer
- 29. Agenda what NoSQL is not motivation Hadoop HBase
- 30. “open source, reliable, distributed computing”
- 31. “open source, reliable, distributed computing”
- 32. MapReduce - API for parallel computing
- 33. MapReduce - API for parallel computing HDFS - distributed, replicated file system
- 34. MapReduce - API for parallel computing HDFS - distributed, replicated file system ZooKeeper - distributed synchronization
- 35. MapReduce - API for parallel computing HDFS - distributed, replicated file system ZooKeeper - distributed synchronization Avro - Data Serialization / RPC
- 36. Agenda what NoSQL is not motivation Hadoop HBase
- 37. structured, distributed database for your horizontally scalable FS
- 38. structured, distributed database for your horizontally scalable FS
- 39. random access
- 42. random access real-time reads/writes simple API big table
- 43. references : http://www.nosql-database.org Eventually Consistent: http://www.allthingsdistributed.com/2007/12/ eventually_consistent.html Soft State: http://mercury.lcs.mit.edu/~jnc/tech/hard_soft.html Accuracy and Precision: http://en.wikipedia.org/wiki/Accuracy_and_precision Compare and Swap: http://en.wikipedia.org/wiki/Compare-and-swap Apache Hadoop: http://hadoop.apache.org Google MapReduce: http://labs.google.com/papers/mapreduce.html Google FS: http://labs.google.com/papers/gfs.html Apache Thrift: http://incubator.apache.org/thrift/ Protobuf: http://code.google.com/p/protobuf/ Google BigTable: http://labs.google.com/papers/bigtable.html Google Chubby: http://labs.google.com/papers/chubby.html
- 44. Questions? Nick Dimiduk - @xefyr Founder, Drawn to Scale nick@drawntoscalehq.com April 28, 2010
Editor's Notes
- #4: I’m Not an RDBMS Guy!
- #5: squish the FUD
- #6: no central point of organization no committee or standardizing body no plan/strategy/illuminati to take down the RDBMS; lots of "in-fighting"
- #7: central tenant - there IS NO one-size-fits-all unlike RDBMS assumptions, each engineering effort must be evaluated for data needs
- #8: is it “anti-RDBMS”?
- #9: not so much
- #11: will not magically solve all your data or performance problems applications won’t magically stop crashing, data corruption, etc. Big Data is still hard. These tools make it possible/affordable/approachable
- #12: data persistence comes down to garantees
- #13: why are we here?
- #14: "web scale" more users, content, connections more trends, insight, knowledge
- #15: Atomicity: fault-tolerance is moving to the application layer - smaller atomic units Consistency: yes! but not necessarily immediate - "availability" (latency, reads) is more important. Isolation: smaller atomic units (multi-step transaction vs. compare-and-swap), greater availability, denormalization => reduced dependency on isolation Durability: some things are more important that getting every last detail, i.e. latency of response, view in aggregate
- #16: Basically Available: is the data layer up or not? are we serving content to our users or not? Soft State: shifting burden of "correctness" up to application layer. availability is more important than precision. accuracy (correct) vs. precision (repeatable). Eventual Consistency: all operations are recorded and ordered. played back as resources permit.
- #17: agile dev moves too fast for schema and constraints - this isn’t waterfall data models change quickly up-front schema modeling is akin to waterfall development - not always practical/feasible/possible data is messy - record what you have and leave constraints up to the application
- #18: at scale, data services look like a DHT anyway! isolated independent services introduced caching layers partitioned data by logical and range boundaries.
- #19: webapp
- #21: app servers/session self-contained - load-balanced data’s in one spot - what do you do?
- #22: 37-signals approach - DHH “scaling is a good thing because scaling => users => $$$”
- #23: more users, more instances. easy!
- #24: doesn’t work for social applications: - users cannot interact - old MMO’s vs. new social games
- #26: redesign data server as “data services” separate independent logical components
- #27: knowing each service by name becomes “vexing”
- #28: configuration/logistical nightmare!
- #29: abstractions! wouldn’t it be nice if...
- #31: Distributed Computing Made Easy Less Hard
- #33: programming model/API for parallel computing Google's MapReduce paper
- #34: replicated, high throughput, fairly UNIX-y (not POSIX). Google FS Paper
- #35: Distributed Group Services - coordination, synchronization, configuration, naming. Google Chubby Paper
- #36: efficient, cross-language messaging Facebook/Apache Thrift Google Protobufs
- #38: Google BigTable
- #39: Addresses limitations of Raw M/R, HDFS access
- #40: request by key: vs. hdfs sequential reads
- #41: low-latency, ms response times vs. m/r high-latency
- #42: row/column concepts DHT semantics Java, ReST, thrift
- #43: Billions of rows, millions of columns