JSON Data Modeling - July 2018 - Tulsa Techfest

Editor's Notes

• #3: Spend just a little time on why people are using NoSQL Talk about how data is modeled differently in JSON Let’s talk about why SQL is good and why SQL for JSON is needed Let’s talk about the exciting stuff happening in the database ecosystem Including but not limited to the stuff Couchbase is doing If we have time, we’ll look at how a .NET developer (or Java developer, etc) would interact with SQL for JSON
• #7: What’s also interesting is that we’re seeing the use of NoSQL expand inside many of these companies. Orbitz, the online travel company, is a great example – they started using Couchbase to store their hotel rate data, and now they use Couchbase in many other ways. Same with ebay, they recently presented at the Couchbase conference with a chart tracking how many instances of various nosql databases are in use, and we see growth in Cassandra, mongo, and couchbase has actually surpassed them within ebay
• #8: SQL (relational) databases are great. They give you LOT OF functionality. Great set of abstractions (tables, columns, data types, constraints, triggers, SQL, ACID TRANSACTIONS, stored procedures and more) at a highly reasonable cost. Change is inevitable One thing RDBMS does not handle well is CHANGE. Change of schema (both logical and physical), change of hardware, change of capacity. NoSQL databases ESPECIALLY ONES DESIGNED TO BE DISTRIBUTED tend to help solve problems with: agility, scalability, performance, and availability
• #9: Let’s talk about what NoSQL is, first. NoSQL generally refers to databases which lack SQL or don’t use a relational model Once the SQL language, transaction became optional, flurry of databases were created using distinct approaches for common use-cases. KEY-Value simply provided quick access to data for a given KEY. Wide Column databases can store large number of arbitrary columns in each row Graph databases store data and relationships as first class concepts Document databases aggregate data into a hierarchical structure. With JSON is a means to the end. Document databases provide flexible schema,built-in data types, rich structure, implicit relationships using JSON.
• #10: When we look at document databases, they originally came with a Minimal set of APIs and features But as they continue to mature, we’re seeing more features being added And generally I’m seeing a convergent trend between SQL and NoSQL But anyway, this set of minimal features, lacking a SQL language and tables gives us the buzzword “nosql”
• #11: Elastic scaling Size your cluster for today Scale out on demand Cost effective scaling Commodity hardware On premise or on cloud Scale OUT instead of Scale UP [example: changing the channel to a soccer game or Game of Thrones, everyone makes the same API request in the same 5 minutes] [example: TV show lets watchers vote during some period of the week, so you can scale up during that period of time] [example: black Friday]
• #12: Schema flexibility Easier management of change in the business requirements Easier management of change in the structure of the data Sometimes you're pulling together data, integrating from different sources (e.g. ELT) and that flexibility helps Document database means that you have no rigid schema. You can do whatever the heck you want. That being said, you SHOULDN’T. You should still have discipline about your data.
• #13: If one machine goes down, customers can still use the other. Or if you need to perform maintenance, upgrade, etc, you don't have to take the whole system down This is related to scaling Built-in replication and fail-over No application downtime when hardware fails Online maintenance & upgrade No application downtime
• #14: NoSQL systems are optimized for specific access patterns Low response time for web & mobile user experience Millisecond latency Consistently high throughput to handle growth [perf measures can be subjective – talk about architecture, integrated cache, maybe mention MDS too]
• #15: Let’s talk about data modeling a bit, because storing data in JSON Is different that storing in tables.
• #16: So I want to compare the approaches over 4 key areas. I’m going to fill in this table, traditional SQL on the left and JSON on the right
• #17: Let’s look at modeling Customer data. This is an example of what a customer might look like There is a rich structure: attributes, potentially sub-attributes (first name and last name) Relationships: to other data (other customers, to products perhaps) Value evolution: Maybe we’d start with one purchase, add more as Helen makes more purchases Structure evolution: Maybe we start will billing information being properties of Helen, then evolve later to be multiple billing options
• #18: Let’s look at modeling Customer data. This is an example of what a customer might look like There is a rich structure: attributes, potentially sub-attributes (first name and last name) Relationships: to other data (other customers, to products perhaps) Value evolution: Maybe we’d start with one purchase, add more as Helen makes more purchases Structure evolution: Maybe we start will billing information being properties of Helen, then evolve later to be multiple billing options
• #19: Let’s see how to represent customer data in JSON. The primary (CustomerID) becomes the DocumentKey Column name-Column value becomes KEY-VALUE pair.
• #20: We aren’t normal form anymore Rich Structure & Relationships Billing information is stored as a sub-document There could be more than a single credit card. So, use an array.
• #21: Value evolution Simply add additional array element or update a value.
• #22: Structure evolution Simply add new key-value pairs No downtime to add new KV pairs Applications can validate data Structure evolution over time. Relations via Reference
• #23: So, finally, you have a JSON document that represents a CUSTOMER. In a single JSON document, relationship between the data is implicit by use of sub-structures and arrays and arrays of sub-structures.
• #24: So, finally, you have a JSON document that represents a CUSTOMER. In a single JSON document, relationship between the data is implicit by use of sub-structures and arrays and arrays of sub-structures.
• #25: So I want to compare the approaches over 4 key areas. I’m going to fill in this table, traditional SQL on the left and JSON on the right
• #28: Hackolade supports couchbase, mongo, elastic, Cassandra, dynamo Erwin supports mongodb, couchbase soon? Idera supports mongodb, couchbase soon?
• #29: The way you plan to get data out can also affect the way you model your data
• #34: What types of relationships are being modeled? How are the relationships accessed?
• #35: Another thing to consider is whether or not your nosql database has a "subdocument" API Not all of them do, and some databases may call this something different But the idea is this: if you only need "address", without a subdocument API You'd have to pull the entire document over the wire when doing reads/writes With a subdocument API, you can specify just a specific part to read/write This can be very helpful is you have large documents, or if you are doing a lot of reads and writes that only need a small portion of the data
• #36: I've mostly been talking about key/value access But document databases have a wide range of ways to access data other than key/value Other databases like mongo have their own javascript query API, but I find N1QL to be very compelling because I'm used to writing SQL to interact with data In Couchbase, we have N1QL, which is ANSI SQL for JSON Most document databases have MapReduce capabilities, including Couchbase. We're kinda leaning away from M/R these days in favor of N1QL, but it can still be useful in some cases There are other ways to access data like FTS and Geospatial which I'm not going to cover today
• #37: Notice I’m using Guid That may not be a good idea This is C#, and you already saw how to do this with Go Also supported: Java, Node, Python, PHP, C, and many more
• #39: Starting from "matt" you can walk through this chain of documents Without having to do a query You can access them with key/value operations only if you'd like
• #40: N1QL is powerful in it's flexibility, declarative nature, familiar to developers, JOINs, etc. Indexing is very important, as it's not as performant as key/value or map/reduce (Maybe talk about indexing on a SQL table vs indexing on a whole bucket)
• #41: Couchbase 5.0 has introduced some tools for analyzing query performance So you can see what indexes are being used, where the biggest costs are in the query And so on. There are a lot of different types of indexes for N1QL
• #42: This is kinda like a materialized view It's powerful in that it can be run in parallel, can use JavaScript to do filtering/mapping, great for aggregation. It's limited in that it can't do anything like a JOIN, can't get input from other views, can only order lexigraphically
• #44: Migrating or syncing Because often Couchbase and relational are complimentary Couchbase can be used for engagement, Relational for transactions
• #45: Are you going to take the time to clean up the data? Do you need to? Do you need to enrich or restructure the data to take advantage of Json?
• #46: Duration v resources: how long is it going to take? What tools and resources are available to you? What’s your biggest constraint – time or resources? Do you need to get the migration done in 1 hr (and have it use as many parallel resources as needed) or do you need to minimize/manage the resource impact on the existing system and it doesn’t matter how long it takes?
• #47: You must obey the claw Data governance: what are the rules for moving data, auditing, etc? Do you need to keep track of where the data came from and who is allowed to access it? Many newer systems need to track where sensitive data originated. 
• #48: A whole bunch at a time, or one at a time Single threaded – easier Multi-threaded – faster, complicated is the migration a one-time event or does it need to happen incrementally (every day or over a 2-3 month period where both the old system and new system are both operating in parallel)? Do you plan to do the data migration as a single thread (read all the data, write all of the data) or using a multi-threaded or multi-process approach where each thread or process reads some percentage of the data.
• #49: If you're writing your own, Entity Framework can be helpful, because it can do the mapping of aggregate root C# objects for you, which you can then write to a document database So if you already have EF mappings created, you're part way there.
• #50: KISS: Either export to CSV and use N1QL to do any ETL that’s required (assuming that it’s Simple) or use SQL to do simple ETL on export and then just import into CB.
• #51: Basically keep it as simple as you can and plan for failure. Developers often think of the migration process as “One and Done”, but the reality is that data migration is often an ongoing headache that DevOps needs to monitor and manage in a production environment. Make everyone’s life easier by thinking about the long game as much as possible.
• #52: From NoSQL to relational
• #53: From relational to NoSQL: Goldendate is from oracle Cdata for SSIS and Couchbase https://github.com/mahurtado/CouchbaseGoldenGateAdapter https://www.cdata.com/drivers/couchbase
• #54: A tool I recently discovered that helps you move, process, transform data all around your enterprise Called Nifi Can connect to Couchbase, SQL Server, just about any source or destination I wrote a blog post showing how to move data from SQL Server to Couchbase using NiFi I did this as a proof of concept for the Cincinnati Reds I'm far from a Nifi expert, but I really like what I've seen It allows you to make a flow that's repeatable, auditable, debuggable, can be easily stopped, started, monitored, notifications, error handling, and more And it's visual Great for migration or syncing or both https://blog.couchbase.com/nifi-processing-flow-couchbase-server/
• #55: Make it part of your application directly Maybe use Akka May or may not be reusable This is a lot of work, so make sure you have a good reason
• #57: Focus on SOA, application/use case specific
• #58: Use Document type, Versionid Create optimized, understandable keys Weigh nested, referenced or mixed designs Add indexes: Simple, Compound, Functional, Partial, Array, Covering, Memory Optimized
• #59: N1QL, Key-value, Views
• #60: Focus, Success Criteria, Review Architecture consider using a tool like Hackolade to define models rigorously and collaboratively
• #62: All I ask is that you give Couchbase a chance Free download You can also take it for a free test drive on the major cloud providers Also, this is something new for me this year, please go to this URL to enter to win a $100 gift card. It is literally a 1 question survey and it helps me out a lot.
• #63: This is my family My enormous head barely fits in the picture
• #65: Not yet. We've been talking about it at least as long as I've been with Couchbase. It's partly a technical problem, may need additional features for multi-tenant. It's partly (mostly) a business problem. Would this be worth it? Couchbase IS in the Azure and AWS marketplaces, and there are some wizards to make config easy, but it runs on your VMs.
• #66: Memory first: integrated cache, you don't need to put redis on top of couchbase Master-master: easier scaling, better scaling Auto-sharding: we call vBuckets, you don't have to come up with a sharding scheme, it's done by crc32 N1QL: SQL, mongo has a more limited query language and it's not SQL-like Full Text Search: Using the bleve search engine, language aware FTS capabilities built in Mobile & sync: Mongo has nothing like the offline-first and sync capabilities couchbase offers Mongo DOES have a DbaaS cloud provider
• #67: Everything I've shown you today is available in Community edition The only N1QL feature I can think of not in Community is INFER and Query Plan Visualizer The Enterprise features you probably don't need unless you are Enterprise developer.