JSON Data Modeling - GDG Indy - April 2020

Editor's Notes

• #3: Most developers are probably already familiar with the relational way of modeling data But if you want the benefits of a non-relational database, you have to think differently about modeling
• #7: Spend just a little time on why people are using NoSQL Talk about how data is modeled differently in JSON Let’s talk about why SQL is good and why SQL for JSON is needed Talk about accessing data, since that has an effect on modeling Maybe we'll get to migrating/syncing data from relational to nosql
• #8: SQL (relational) databases are great. They give you LOT OF functionality. Great set of abstractions (tables, columns, data types, constraints, triggers, SQL, ACID TRANSACTIONS, stored procedures and more) at a highly reasonable cost. Change is inevitable One thing RDBMS does not handle well is CHANGE. Change of schema (both logical and physical), change of hardware, change of capacity. NoSQL databases ESPECIALLY ONES DESIGNED TO BE DISTRIBUTED tend to help solve problems with: agility, scalability, performance, and availability
• #9: Let’s talk about what NoSQL is, first. NoSQL generally refers to databases which lack SQL or don’t use a relational model Once the SQL language, transaction became optional, flurry of databases were created using distinct approaches for common use-cases. KEY-Value simply provided quick access to data for a given KEY. Wide Column databases can store large number of arbitrary columns in each row Graph databases store data and relationships as first class concepts Document databases aggregate data into a hierarchical structure. With JSON is a means to the end. Document databases provide flexible schema,built-in data types, rich structure, implicit relationships using JSON.
• #10: When we look at document databases, they originally came with a Minimal set of APIs and features But as they continue to mature, we’re seeing more features being added And generally I’m seeing a convergent trend between relational and NoSQL But anyway, this set of minimal features, lacking a SQL language and tables gives us the buzzword “nosql”
• #11: Think of a document database at the simplest as a type of a key/value store, where the value is in a known format You write code where you start with a key, and you ask the database to return the document That corresponds to that key. And the same with creating/updating
• #12: If you are using a cloud based DBaaS, this is basically what's going on behind the scenes Elastic scaling Size your cluster for today Scale out on demand Cost effective scaling Commodity hardware On premise or on cloud Scale OUT instead of Scale UP [example: changing the channel to a soccer game or Game of Thrones, everyone makes the same API request in the same 5 minutes] [example: TV show lets watchers vote during some period of the week, so you can scale up during that period of time] [example: black Friday]
• #13: Schema flexibility Easier management of change in the business requirements Easier management of change in the structure of the data Sometimes you're pulling together data, integrating from different sources (e.g. ELT) and that flexibility helps Document database means that you have no rigid schema. You can do whatever the heck you want. That being said, you SHOULDN’T. You should still have discipline about your data.
• #14: If one machine goes down, customers can still use the other. Or if you need to perform maintenance, upgrade, etc, you don't have to take the whole system down This is related to scaling Built-in replication and fail-over No application downtime when hardware fails Online maintenance & upgrade No application downtime
• #15: NoSQL systems are optimized for specific access patterns Low response time for web & mobile user experience Millisecond latency Consistently high throughput to handle growth [perf measures can be subjective – talk about architecture, integrated cache, maybe mention MDS too]
• #16: NoSQL is very versatile and can be used for a wide variety of use cases Including, but NOT limited to these If you're exploring NoSQL, make sure you have the right project or right use case. Using a NoSQL database does NOT mean you have to abandon relational databases. Most large websites use a combination. And it's also worth pointing out that plenty of companies are doing (most) of these use cases with relational as well. Relational usually is at least mediocre, NoSQL may be BETTER But usually the catalyst is one of the earlier reasons: performance, flexibility, scale.
• #18: Let’s talk about data modeling a bit, because storing data in JSON Is different that storing in tables.
• #19: Let’s look at modeling Customer data. This is an example of what a customer might look like. You might do this as part of a proof of concept, discovery, requirements gathering, planning, etc There is a rich structure: attributes, potentially sub-attributes (first name and last name) Relationships: to other data (other customers, to products perhaps) Value evolution: Maybe we’d start with one purchase, add more as Helen makes more purchases Structure evolution: Maybe we start will billing information being properties of Helen, then evolve later to be multiple billing options
• #20: Let’s look at modeling Customer data. This is an example of what a customer might look like There is a rich structure: attributes, potentially sub-attributes (first name and last name) Relationships: to other data (other customers, to products perhaps) Value evolution: Maybe we’d start with one purchase, add more as Helen makes more purchases Structure evolution: Maybe we start will billing information being properties of Helen, then evolve later to be multiple billing options
• #21: Let’s see how to represent customer data in JSON. The primary (CustomerID) becomes the DocumentKey Column name-Column value becomes KEY-VALUE pair.
• #22: We aren’t normal form anymore Rich Structure & Relationships Billing information is stored as a sub-document There could be more than a single credit card. So, use an array.
• #23: Value evolution Simply add additional array element or update a value.
• #24: Structure evolution Simply add new key-value pairs No downtime to add new KV pairs Applications can validate data Structure evolution over time. Relations via Reference
• #25: So, finally, you have a JSON document that represents a CUSTOMER. In a single JSON document, relationship between the data is implicit by use of sub-structures and arrays and arrays of sub-structures.
• #26: So, finally, you have a JSON document that represents a CUSTOMER. In a single JSON document, relationship between the data is implicit by use of sub-structures and arrays and arrays of sub-structures.
• #29: Hackolade supports couchbase, mongo, elastic, Cassandra, dynamo, firebase/firestore Erwin supports mongodb and couchbase Idera supports just mongodb
• #30: The way you plan to get data out can also affect the way you model your data
• #35: What types of relationships are being modeled? How are the relationships accessed?
• #36: I've mostly been talking about key/value access Most NoSQL databases will have at least one other way to access data besides key/value. What does this have to do with modeling? Because modeling doesn't exist in a vacuum. You have to think about how you are going to interact with your data. I'm going to show you some examples from Couchbase. In Couchbase, we have N1QL, which is ANSI SQL for JSON I'm also going to briefly cover FTS today. There are other options I'm not going to cover today, including Analytics and Views/MapReduce.
• #37: Just to reiterate on key/value If you know the key already, it's really simple and extremely fast to access that piece of data.
• #38: Since key/value is so fast and easy, it would benefit us to use it as much as possible. Here are some tips to maximize your key/value usage.
• #39: Starting from "matt" you can walk through this chain of documents With ONLY key/value access.
• #40: Another thing to consider is whether or not your nosql database has a "subdocument" API If they do, then you have even more flexibility Not all of them do, and some databases may call this something different But the idea is this: if you only need "address", without a subdocument API You'd have to pull the entire document over the wire when doing reads/writes With a subdocument API, you can specify just a specific part to read/write This can be very helpful is you have large documents, or if you are doing a lot of reads and writes that only need a small portion of the data Firestore: there is a different concept of "subcollection". You have collections that contain documents, and then documents themselves can contain collections. Those documents have their own keys. If you DELETE a parent, then this subcollection will stick around. But when it comes to modeling considerations, with Firestore you can access these subcollections and look up individual keys in them. This is where we diverge a little bit from JSON modeling. But the idea is that you don't have to get the ENTIRE document, you can target individual pieces.
• #41: In firestore, you have a collection of documents (rooms, in this case). An individual document (roomA) can itself have a collection (messages) And so on. You can do this in plain JSON in just about every other document database You can nest as deep as you want in JSON. Much like the subdocument I mentioned before, you can do a similar thing here. If you want to make a change To one message, you can address it directly with "rooms/roomA/messages/message1" EXCEPT that Firestore treats the documents in the subcollection kinda like separate documents. Gotcha: "Deleting a document does not delete its subcollections" So you should be careful when using this, you could end up with something kinda like orphan documents
• #42: N1QL is powerful in it's flexibility, declarative nature, familiar to developers, JOINs, etc. But note, once we step out of key/value access, we need to involve other processes: We gotta parse the query, most likely use an index service, and in the end we'll get a bunch of keys to lookup the data. There is overhead involved, but sometimes this is a necessity.
• #43: When you're outside of key/value access, you must understand the query plan. This is true for ANY database, relational or NoSQL. As an example, here's a Couchbase SQL query. I execute this and it ran in 1.2 seconds. It's using AN index on the TYPE field, but notice that name field on there. I can bring up a visualization of the query plan to see which parts are taking up the most time. In Couchbase, there is an index advisor. It suggested an index for me. After creating the index, the same query went to 146ms (about 8 times faster) Covering index could make this even faster (note the *)
• #44: This is a search that revolves around text. Things like stemming, language awareness, facets, ranking, etc. This is, again, a very simple example. I'm searching for the keyword "submarine". In my application, this query may be limited to a certain search radius, or it may be limited to a certain facet, etc. But the end results are language aware, ranked matches. You would use this INSTEAD OF a sql 'like' for instance
• #45: Build a proof of concept, which will help you see if NoSQL is the right fit And it will also help you understand the access patterns better Just to sum up Also note that in Couchbase, you can combine FTS and N1QL
• #46: Migrating or syncing Because often Couchbase and relational are complimentary Couchbase can be used for engagement, Relational for transactions
• #47: Are you going to take the time to clean up the data? Do you need to? Do you need to enrich or restructure the data to take advantage of Json? Also I'm using the term MIGRATION, but it might NOT be the case that you are abandoning a database in favor of another. You might want to sync data, you might want to make a copy of data into a more suitable database for your use case, etc.
• #48: informatica, Talend, DART, ODBC, Kafka, Debezium, Spark, Kafka, Nifi
• #49: Node / Python / bash / Powershell / curl / REST etc GoldenGate / DTS / SSIS CLI: cbimport, mongoimport, etc
• #50: Again, proof of concept KISS: export to CSV and use N1QL to do any ETL that’s required Export to CSV Import as documents into a 'staging' bucket Use N1QL to transform Insert into new bucket
• #51: Align with your data model The modeling step is vital. If you don't model your data to take advantages of JSON, then you are not going to see the advantages of using JSON. Don't treat a JSON database as if it were a relational database. Basically keep it as simple as you can and plan for failure. Developers often think of the migration process as “One and Done”, but the reality is that data migration is often an ongoing headache that DevOps needs to monitor and manage in a production environment. Make everyone’s life easier by thinking about the long game as much as possible.
• #52: Plan for failure Bad source data Hardware failure Resource limitations (proof of concept vs MVP) Developers often think of the migration process as “One and Done”, but the reality is that data migration is often an ongoing headache that DevOps needs to monitor and manage in a production environment.
• #53: Ensure: Interruptible, restartable, logged -> predictable Make everyone’s life easier by thinking about the long game as much as possible.
• #54: From NoSQL to relational You can also turn this around and use Kafka in the other direction
• #55: From relational to NoSQL: Goldendate is from oracle Cdata for SSIS and Couchbase https://github.com/mahurtado/CouchbaseGoldenGateAdapter https://www.cdata.com/drivers/couchbase
• #56: Make it part of your application directly May or may not be reusable This is a lot of work, so make sure you have a good reason
• #58: Focus on SOA, microservices, application/use case specific
• #59: Modeling, Focus, Success Criteria, Review Architecture consider using a tool like Hackolade to define models rigorously and collaboratively
• #60: Use Document type, Versionid Create optimized, understandable keys Weigh nested, referenced or mixed designs Add indexes: Simple, Compound, Functional, Partial, Array, Covering, Memory Optimized N1QL, Key-value, Views
• #62: This is my family My enormous head barely fits in the picture
• #64: Couchbase Cloud is currently in limited beta
• #65: Memory first: integrated cache, you don't need to put redis on top of couchbase Master-master: easier scaling, better scaling Auto-sharding: we call vBuckets, you don't have to come up with a sharding scheme, it's done by crc32 N1QL: SQL, mongo has a more limited query language and it's not SQL-like Full Text Search: Using the bleve search engine, language aware FTS capabilities built in Mobile & sync: Mongo has nothing like the offline-first and sync capabilities couchbase offers Mongo DOES have a DbaaS cloud provider
• #66: Everything I've shown you today is available in Community edition The only N1QL feature I can think of not in Community is INFER and Query Plan Visualizer The Enterprise features you probably don't need unless you are Enterprise developer.