To SQL or NoSQL, That is the Question

Editor's Notes

• #3: * Lines are blurring, some of the common features of nosql either have existed for years (clustering) or are being added (HA, documents, in-memory) to traditional RDBMS systems. * new vs existing system, performance, dev efficiency, ability to reason about data, availability/reliability/durability, future growth (scaling), flexibility
• #4: Debates about SQL vs NoSQL often sound like the old Mac vs PC (or Vi vs Emacs, if you remember those days). It’s not about comparing, it’s about using the right tool for the job. Story: Company was determined to use the same database for storing application data and implementing a complex security mechanism. Without even testing SQL, went straight for NoSQL, performance benchmarked read/write throughput and picked Couchbase. “Why didn’t you use SQL?” But of course, I didn’t say that, because I like my job. Should have used SQL + Caching + Graph. As it is, every security change leads to potentially a million updates. It works, because high throughput compensates for a lot. But wasted a lot of dev effort.
• #5: Debates about SQL vs NoSQL often sound like the old Mac vs PC (or Vi vs Emacs, if you remember those days). It’s not about comparing, it’s about using the right tool for the job. Story: Company was determined to use the same database for storing application data and implementing a complex security mechanism. Without even testing SQL, went straight for NoSQL, performance benchmarked read/write throughput and picked Couchbase. “Why didn’t you use SQL?” But of course, I didn’t say that, because I like my job. Should have used SQL + Caching + Graph. As it is, every security change leads to potentially a million updates. It works, because high throughput compensates for a lot. But wasted a lot of dev effort.
• #6: Poll – how many people recognize 3, 5, 10 of these? How many people have actually used 3, 5, 10? Column = multiple proprietary databases: Sybase IQ, C-Store, etc. ColumnStore tables/indexes in MS SQL Server Row == partitioned row store. Some, like Hbase can also be used as a column store at the column family level, but that’s not the common use-case.
• #7: Poll – how many people recognize 3, 5, 10 of these? How many people have actually used 3, 5, 10? Column = multiple proprietary databases: Sybase IQ, C-Store, etc. ColumnStore tables/indexes in MS SQL Server Row == partitioned row store. Some, like Hbase can also be used as a column store at the column family level, but that’s not the common use-case.
• #9: Poll – how many people recognize 3, 5, 10 of these? How many people have actually used 3, 5, 10? Column = multiple proprietary databases: Sybase IQ, C-Store, etc. ColumnStore tables/indexes in MS SQL Server Row == partitioned row store. Some, like Hbase can also be used as a column store at the column family level, but that’s not the common use-case.
• #10: Data encapsulated in documents A document is Encoded in some common format, often JSON Can often contain other documents No limit in depth Flexible schema Can index and retrieve documents by content Works well with OOP (no O-R impedance mismatch) Use when Complex application object model You don’t know the schema in advance The schema likely to change Challenges Complex join-like queries Self-referencing documents and circular dependencies
• #11: Poll – how many people recognize 3, 5, 10 of these? How many people have actually used 3, 5, 10? Column = multiple proprietary databases: Sybase IQ, C-Store, etc. ColumnStore tables/indexes in MS SQL Server Row == partitioned row store. Some, like Hbase can also be used as a column store at the column family level, but that’s not the common use-case.
• #13: Poll – how many people recognize 3, 5, 10 of these? How many people have actually used 3, 5, 10? Column = multiple proprietary databases: Sybase IQ, C-Store, etc. ColumnStore tables/indexes in MS SQL Server Row == partitioned row store. Some, like Hbase can also be used as a column store at the column family level, but that’s not the common use-case.
• #14: In a table, data of the same column is stored together Storage is not wasted on null value as in row-based stores (RDBMS) Great for sparse tables Very fast column operation including aggregation Extreme compression for low-cardinality columns Use When Big Data (Excellent leverage of Map Reduce) Need compression or versioning Challenges You better know your access patterns before hand Keys design is not trivial
• #15: Poll – how many people recognize 3, 5, 10 of these? How many people have actually used 3, 5, 10? Column = multiple proprietary databases: Sybase IQ, C-Store, etc. ColumnStore tables/indexes in MS SQL Server Row == partitioned row store. Some, like Hbase can also be used as a column store at the column family level, but that’s not the common use-case.
• #16: Scaling – small example with 3 nodes, medium (10GB) example with a role based security graph, huge (1 trillion edges) example with facebook. Facebook blog about Apache Giraph - https://www.facebook.com/notes/facebook-engineering/scaling-apache-giraph-to-a-trillion-edges/10151617006153920 Sharding graphs is hard, so a compromise approach is to use cache sharding, i.e. consistent routing to make sure requests hit a warm cache. Neo4j query language (Cypher) emulates white-board graph sketches as ASCII art. Titan uses other databases for data storage and indexing (cassandra, hbase, berkleyDB) + (ES, Solr, Lucene). Query language (Gremlin) uses a chain of operations/functions (g.V().has('name', 'hercules').out('father').out('father').values('name’) ==> saturn Data is stored as a graph – nodes, edges, properties Perfect for highly interconnected data Allows for explicit relationships Fined grained graph traversal Flexible Often works well with OOP (object relationships) Use when You need to represent and query object relationships (e.g. Netflix movie suggestions) Challenges Harder to scale horizontally Harder to conceptualize
• #17: Properties often associated with NoSQL databases are also being developed for relational databases. Clustering, HA, Non-relational data model. MySQL Cluster (Fabric - HA, Sharding) PostgreSQL JSON (Documents) Oracle RAC (Clustering) However, any criticism that can be leveled at new NoSQL databases, like being half-baked, simple, have high admin overhead can be equally applied to all the new NoSQL-like features of relational databases. They’re relatively complicated solutions, so beyond a certain level of pain it’s time to consider moving to a dedicated NoSQL database.
• #18: What does CAP mean when the network is unreliable? Pick between CP and AP because CA assumes perfect network. In fact, several NoSQL databases claim to provide CA in their documentation – with an asterisk that assumes a “very stable” network. For a distributed computer system, it is impossible to provide all three of the following guarantees: Consistency (all components see the same data) Availability (every request receives a response) Partition tolerance (the system operates despite message loss or failure of some components) Relational databases generally provide C and sometimes A. For single-machine, P is irrelevant. For distributed SQL, same problems as with NoSQL apply. Aka. Brewster’s theorem, suggested by Eric Brewster from UC Berkley. The proof of the CAP theorem by Gilbert and Lynch is a bit narrower than that which Brewer had in mind. The theorem sets up a scenario in which a replicated service is presented with two conflicting requests arriving at distinct locations at a time when a link between them is failed. The obligation to provide availability despite partitioning failures leads the services to respond; at least one of these responses shall necessarily be inconsistent with what a service implementing a true one-copy replication semantic would have done. The researchers then go on to show that other forms of consistency are achievable, including a property they calleventual consistency. Thus the theorem doesn't rule out achieving consistency in a distributed system, and says nothing about cloud computing per se or about scalability.