I Have a NoSQL Toaster - Troy .NET User Group - July 2017

Editor's Notes

• #2: This session was originally conceived by Matthew Revell I have modified it, but it is largely based on his work, so I want to give him credit
• #3: We're here because "relational" is the norm So right away I can tell the age of my audience
• #6: We ‘always’ been doing SQL, anything else must be an aberration? Well I’m not so sure.
• #7: In 3200 B.C. we have the first evidence of human writing Then later the printing press The first commercial database is in 1960 (SABRE by American Airlines and IBM making a better way to handle airline ticket bookings) It was not relational, it was more like a filesystem hierarchy 1970 – EF Codd proposes the relational db model, and came up with SQL later 1979 – the necessary sacrifices to the dark forces were made and the Oracle database is released So maybe around 1990, relational databases become “the norm”, about 20-30 years to where we default to stick things in tables In 2005, people start looking at old ideas (because there’s nothing new underneath the sun), and a document database was created called CouchDB (which was a different spin on Lotus Notes) 2008 – explosion of NoSQL databases, influenced by two academic papers from Google and Amazon, who were losing money because they couldn’t keep relational DBs online – so Amazon created a massively distributed, fault tolerant, highly available, eventually consistent db called Dynamo, and you can use a version of that today, which is called DynamoDB So these papers started a lot of open source activity, and we’ll look at those in a bit more detail today 2010 – the heavens open and Couchbase decends to earth (disclaimer, I work for Couchbase)
• #8: That’s why I say this toaster is a NoSQL Toaster! I can operate it fully without using any SQL query on it It can store data, it’s not tables, it’s bread I can get that data back out (as toast) It’s a good marketing shorthand to say “these databases are different than what we’ve been doing for 30 years” But defining something by what it’s not – only useful for a little while So let’s dive in and figure out what NoSQL means?
• #9: Is it because they’re scalable? Well… not really… not all of them are scalable (and sometimes they aren’t even if their creators say they are)
• #10: So what is it that defines a NoSQL database? Is it because they’re easy to develop against? Well… not really… if you try Cassandra, it’s not straightforward totally easy, can still be a bit inflexible
• #11: Does it have to do with ACID vs BASE? Atomic Consistent Isolated Durable Vs Basic Availability Soft-state Eventual Consistency Well, no… FoundationDB is a NoSQL database that is ACID compliant (Apple purchased them) Many NoSQL databases tradeoff some level of ACIDity for scalability or performance reasons
• #12: Is it that there is no Schema? Maybe… kinda… sorta… even if your database isn’t imposing a rigid schema on you This doesn’t mean there isn’t a schema, it’s just that the database isn’t enforcing one You as a developer need to impose a schema on it
• #13: Is it that they are denormalized? This is pretty close. Data doesn’t need to be normalized, and there may be some benefit to non-normalized data (performance or flexibility or both) Doesn’t mean you can’t split data up, you just do it differently/broader
• #14: Before I get into some of the details about NoSQL databases Here’s a disclaimer If you’re starting out on a new project or refactoring an existing project, don’t use NoSQL Because it’s cool, because there’s peer pressure, because you want more stuff on your resume A lot of problems are SOLVED WELL by existing technology that you understand well and have a good toolset around So if you have a problem that’s not going to scale massively, or it fits nicely into a relational model, and you can deal with object impedance mismatch and OR/Ms and you’re comfortable with that, then you might as well use a relational database and leave the road less traveled for another time
• #15: This is the Blobfish, some call it the ugliest fish in the world But this is a picture at surface level. This is actually a deep sea fish When it’s in its normal high-pressure environment it looks more like this It’s still not going to win any beauty pageants, but it definitely looks more like a standard fish My point is this: whatever data store you choose should be the right one for the job. There’s a term “Polyglot persistence” which just means that. So maybe if part of your app is just saving log data, maybe that’s a good place to start with a nosql db. If another part is highly relational, then use a relational db. Maybe stick with relational for the “database of record”, but use nosql for caching, or for session, or for shopping cart. Wherever it makes sense.
• #16: So maybe SQL is right for your project But there are some cases where maybe relational ISN’T a good fit or at least not the best fit
• #17: With ANY system really Vertical scaling is the easiest: faster server, more ram, beefier machine, but that gets expensive, impractical Horizontal scaling: with relational databases, I’ve got two words for you “sharding scheme” With SQL Server there are a variety of sharding methods (vertical, range, hash, directory), but you have to worry about joining across shards, denormalizing, rebalancing, what if you change the schema? People do this, and people get it to work. Depending on your use case, it can be reasonable for certain use cases where the data can be easily divided, it can be a nightmare otherwise.
• #18: This is where distributed databases can help. You can do this with relational databases, but many NoSQL databases are built with this in mind If you have a database cluster, and you need additional capacity, you can rack up another cheap server and add it to the cluster. Now: You don’t have to upgrade your hardware licensing may cost you less Your capacity is flexible There are some other benefits!
• #19: This is always hard with a relational database. Keeping things online. Especially for writes. CAP Theorum – Consistent Available Partition-tolerant, applies to when nodes go down or there’s a network split. When operating normally, there is no such tradeoff.
• #20: CAP theorem states that a distributed system can have three attributes And that you can only guarantee two of them at a time That word “guarantee” is in the academic sense If your network and servers are working as expected, you get all three. You might get all three to 99999 but you can't guarantee it We’ll see more of this graph later
• #21: Something simple may not be so bad But something complex… Coming up with good schemas in the first place And then maintaining them across the lifetime of your app Has anyone had to take a database offline (or put into read-only) in order to do a schema migration? In many cases, this just isn’t acceptable: to tell a user that you can’t use this because we’re doing schema maintenance
• #22: There are lots of types of NoSQL databases These 4 seem to be the main categories that people are interested in, esp in open source world And these will be the ones you’ll likely encounter in your research about what to use Graph is still a bit of the odd one out
• #23: NoSQL databases are converging, and databases overall are converging In Couchbase, we’ve introduced N1QL, which is a full SQL implementation PLUS SOME to query documents Riak has introduced indexing and other features to help with querying SQL Server and PostgreSQL have introduced some great JSON features DocumentDB for Azure has a SQL implementation So it’s an exciting time where databases are adapting and changing and converging
• #25: Logos for: MongoDB, CouchDB, MarkLogic, Couchbase, Lotus Notes, RethinkDB, OrientDB, Xbase, RavenDB, Existdb These are the top two most relevant document databases MongoDB has huge traction, Couchbase is easily the runner up (I’d say that even if they didn’t pay my mortgage)
• #26: It understands the format of the document you’re dealing with It’s not just a bag you throw things in, it has understanding of the data These are often the best general replacement for relational databases
• #27: MarkLogic is XML, I learned recently it can also do JSON I guess there could be a YAML database or something else like that… But no one likes XML (animation to cross it off or whatever) BSON is a binary implementation of JSON that Mongo uses, can be exported to “Extended JSON” so really what we’re talking about here is JSON
• #28: So what we’re talking about in document databases is storing JSON It’s human readable, compact (compared to XML), can contain arrays, objects, hierarchies
• #30: It can do something to the document (or documents) without you having to -pull it out -do something to it -put it back
• #31: Take a quick run through how couchbase handles documents Here are some new terms in NoSQL Buckets – is a collections of documents, or equivalent of a database Documents – Kinda like tables, but not Views – map/reduce functions to do interesting queries N1QL – Couchbase has a query language that you can use to query/operate on documents as well
• #32: Here is a CSV of data for Swag There’s a code, title/description, and quantity
• #33: Here’s some C# that pulls that in and puts it into couchbase
• #34: There’s a URL to one or more nodes The bucket name
• #35: Taking the values from the CSV I’m making the swag code into the document key And I’m taking the other values, stuffing them into an anonymous object (which will be converted to JSON)
• #36: This is a screenshot of the Couchbase Console, it’s the admin view And this is what it would look like if you imported into Couchbase
• #37: Pull out a single document with a GET statement
• #39: Document databases have these operations, typically simple crud operations This is (initially) why these databases are called “nosql”, because you aren’t using SQL to do these core operations Do more interesting stuff with M/R or with N1QL queries in Couchbase Often what we’re doing with relational databases is starting with an object that has hierarchy, and twisting it around to get it to fit into database tables If we can avoid it, why go through that pain of an impedence mismatch? Instead just put the whole thing into a document
• #41: Some things that couchbase is good for Great for dev, less impedence mismatch Scales well, works as a cluster, easy to ramp up – just rack up another server, join the cluster, and it redistributes data Fast, because there is a built-in cache so you’ll often be reading/writing to RAM and not having to wait on the disk Drawback: When nodes go down or network splits: highly consistent, but eventually available
• #42: Consistency means that even during a network partition, there is only a single up to date copy of the data. Couchbase works this way. So, if a node goes down, you can no longer access that document. That sounds dire, but with Couchbase, there are replica documents stored on other nodes. So, when the nodes goes down, you can get read-only copies. And, Couchbase has an “auto failover” feature. When a node is detected as being down, it can compensate by promoting the replicas to active documents. This does take time, so during that period you do not get availability.
• #44: Similar to document databases Except the value is an opaque blob, the database doesn’t care what the value is and doesn’t reason about it
• #45: Couchbase can act as a key/value store Amazon dynamodb is a service from amazon Redis is an in-memory data store Riak, which is a faithful implementation of the Dynamo paper from Amazon
• #46: Mix of in-memory and distributed data stores Let’s focus on distributed data stores (cross out in-memory caches)
• #47: It’s “content agnostic” It’s very flexible in what you can store, usually very fast, not terribly flexible for querying
• #48: I’m not a Riak expert, I’m not even a Riak user, I’ll try to represent it fairly
• #49: Connects to riak Create a bucket called ‘actors’ Let’s store actors as the value and the role as the key “James Bond” => “Sean Connery”
• #51: This is the gotcha with dynamo-style systems
• #52: You’ve got a number of servers and a number of apps writing to them and reading from them at the same time Riak tries its best not to ever refuse a write So let’s say you lose a bunch of servers, someone tripped over the power cord And riak, dynamodb, etc, will carry on receiving the writes and taking over the workload for now Or maybe there’s a network split; some apps are writing to one half, the other are writing to the other half So you might have conflicting values With a strongly consistent database, writes could be refused. This is trade-off with something like Riak [animation] So app 1 writes Sean Connery, app 2 writes Timothy Dalton, and maybe app 3 writes in Roger Moore
• #53: But the truth is: there can be only one!
• #54: So when we’re talking about distributed databases, there’s really only two options, Availability or Consistency Availability means that when there is a network partition, you can still perform reads and writes on any piece of data Even if that data lived on a server that can no longer be accessed The question is, what happens if that node comes back? There’s potential for a conflict (also known as “siblings”) You can resolve this conflict in a number of ways: at the database level with timestamp or last-write-wins Or bubble it up to a user Or write some sort of custom merging code If availability is more important that consistency, this is the system you go with.
• #55: Riak has introduced a lot of cool stuff to help with this problem: Version Vectors, explicit siblings, and other stuff But in many cases it’s going to be something that you have to think about, figure out, and fix I would recommend you try out Riak
• #57: These come from google’s BigTable paper These are the hard ones Store data in columns instead of rows
• #58: These are some Columnar databases For web devs working with open source tools, we focus on these two Let’s have a very quick look at cassandra
• #59: It’s taken the data model idea from BigTable But it’s taken data distribution from the dynamo paper So you’ve got eventual consistency of something like Riak with data model that kinda looks relational but absolutely isn’t Keyspaces are like databases Column Families are a bit like tables Rows are rows, but different than you know them Columns are key-value pairs associated with a row
• #60: So what does that actually mean This image borrowed from DataStax website The keyspace is ‘blog’ (but please don’t use Cassandra to run a blog)
• #61: Zoom into column families ‘Users’ is a column family (kinda looks like a table) Each row has a key So if we get jbellis, we get name and state, which are columns (k/v pairs associated with the row) They can be anything, each row doesn’t have to have same columns You can do foreign keys, but manually It looks relational, but it really has a key/value approach Why do this? Google likes it for analytics. Instead of updating something in place, you can just add another column (there is an upper limit to columns, but it’s large) Let’s say we had a gas meter instead of a blog. Gas reader takes a reading every 15 minutes So a column family could be a gas meter A row could be a day Each column would be every 15-minute reading It’s very quick to do an aggregate of row, get an average reading per day, something like that
• #62: This is an example I took from the datastax.com site on getting started This looks like SQL, but it’s CQL, and it’s more like a subset of SQL: it’s lacking things like JOINs and subqueries
• #63: Here I’m connecting to the cluster Picking a keyspace Executing an insert And then selecting out that user and printing it to console
• #65: Impedance mismatch – don’t use it for a blog, use it where you need to throw a lot of data quickly, do range queries across large bits of data Scales well – but apparently is an Ops headache, since it uses the JVM
• #66: So when we’re talking about distributed databases, there’s really only two options, Availability or Consistency Availability means that when there is a network partition, you can still perform reads and writes on any piece of data Even if that data lived on a server that can no longer be accessed The question is, what happens if that node comes back? There’s potential for a conflict (also known as “siblings”) You can resolve this conflict in a number of ways: at the database level with timestamp or last-write-wins At the client level, do a merge or even let the user resolve the conflict, you cannot guarantee that a merge will never have to be handled manually If availability is more important that consistency, this is the system you go with.
• #67: Really big data: Hbase says “billions of rows times millions of columns”
• #68: Graph is kinda the odd one out, but worth mentioning The others are trying to store run-of-the-mill data With a graph, you’re storing data, but you’re also storing rich connections between the data
• #69: Graph databases are based on the ideas of Euler who a polymath in the 1700s, contributed to math, physics, astronomy, logic, etc The “7 bridges of…” problem So Euler came up with a whole area of math to deal with that problem
• #70: The problem is this: figure out a walk through the city that would cross each bridge once (and only once). You don’t have to start or stop at the same spot. He proved that there was no way to do that, but in doing so Euler used this problem to establish the entire field of graph theory
• #71: Graph databases are all about the WAY that things are connected together CosmosDb is an azure offering from Microsoft, it’s “multi-model”, so it can act as all the types we’ve talked about as well as graph OrientDb is also multi-model, it can act as document database or graph
• #72: Graph databases are all about the WAY that things are connected together So this image has some random-looking things Each of the items are all different things, database only cares about the relationships between them These relationships are directional So you could ask “give me all the enemies of Luke Skywalker who have appeared in The Force Awakens” Supermarket: “give me all the people who bought baby food and who also bought beer”, decide to give them discounts or coupons, etc Images: https://www.flickr.com/photos/mlemos/581278438 https://www.flickr.com/photos/pmiaki/5351186496 https://commons.wikimedia.org/wiki/File:Starwars-tatooine.jpg https://www.flickr.com/photos/stevetroughton/16889877817 https://commons.wikimedia.org/wiki/File:Star_Wars_The_Force_Awakens.jpg https://commons.wikimedia.org/wiki/File:MCM_London_May_15_-_Stormtrooper_(18246218651).jpg https://commons.wikimedia.org/wiki/File:Empirestrikesback2.png
• #73: Gremlin, CosmosDB Neo4j supports Cypher query language
• #75: NoSQL is a term that I have a love/hate relationship with Groups together a bunch of different databases into one big tent "NoSQL" may mean something different to someone you're talking to It's more helpful to drill into specifics: document, key/value, etc
• #76: So, for your next web project, take a minute to think about the database you’re going to use Don’t use something just because it’s fashionable Don’t use something just because you’ve always used it Consider using multiple data stores Think about what you’re going to need long term Polyglot persistence
• #79: Mongo: Features N1QL, XDCR, Full Text Search, Mobile & Sync. Memory-first architecture and proven, easy scaling. CouchDb: Couchbase started as a whole new piece of software that was basically a combination of memcache and CouchDb a long time ago, but has grown far beyond that. Couchbase isn’t a fork or vice versa. They share an acronym and they are both NoSQL. Like MySQL and SQL Server, for instance. Open source apache license for community edition, enterprise edition on a faster release schedule, some advanced features, and support license. Couchbase is software you can run in the cloud on a VM or on your own data center. CosmosDb is a manage cloud service, but there is a emulator you can run locally.