Relational data model in Cassandra: Will it fit?
Download as PPTX, PDF1 like215 views
This document discusses options for modeling relational data in Cassandra, including denormalization, user-defined types (UDTs), and secondary indexes. It presents a use case of modeling highly nested data with UDTs in Cassandra and accessing it from Spark. While the Cassandra data model is not a traditional relational model, this document shows that with techniques like UDTs and external indexing with Solr, many relational patterns can still be supported.
![Cassandra data model
It’s simple
Map[k, Map[k, v]]](https://image.slidesharecdn.com/distributeddatadays2018sf-180914234143/85/Relational-data-model-in-Cassandra-Will-it-fit-9-320.jpg)
![Cassandra data model
It’s simple
Map[k, Map[k, v]]
It sucks](https://image.slidesharecdn.com/distributeddatadays2018sf-180914234143/85/Relational-data-model-in-Cassandra-Will-it-fit-10-320.jpg)
![Cassandra data model
It’s simple
Map[k, Map[k, v]]
It sucks
Or not...](https://image.slidesharecdn.com/distributeddatadays2018sf-180914234143/85/Relational-data-model-in-Cassandra-Will-it-fit-11-320.jpg)
Relational data model in Cassandra: Will it fit?
- 2. Relational model in Cassandra: Will it fit? Distributed Data Days SF, September 2018 Matija Gobec matija.gobec@smartcat.io @mad_max0204
- 4. Agenda Cassandra data model Options and alternatives UDT use case and Apache Spark
- 5. Who is using Cassandra?
- 6. What are 3 major Cassandra issues Data model Over-expectations Poor resource planning
- 7. What are 3 major Cassandra issues Data model Over-expectations Poor resource planning
- 8. Cassandra data model It’s simple
- 9. Cassandra data model It’s simple Map[k, Map[k, v]]
- 10. Cassandra data model It’s simple Map[k, Map[k, v]] It sucks
- 11. Cassandra data model It’s simple Map[k, Map[k, v]] It sucks Or not...
- 12. Data model
- 13. Cassandra data model Primary key DATA Slim row
- 14. Cassandra data model Partition key DATA Wide row Clustering key DATA Clustering key DATA Clustering key ...
- 15. But my data model looks like this
- 16. Or even...
- 17. What are my options?
- 18. Denormalization Query based data model Employee EmployeeID OrganizationID Name OrganizationID Name Employee name EmployeeID 1. Select all employees for a given organizationID EmployeeID Name 2. Select employee for a given employeeID OrganizationID Organization OrganizationID Name Relational model
- 19. Denormalization Application level joins Organization OrganizationID Name Employee EmployeeID OrganizationID Firstname Lastname Email ... Relational model OrganizationID Name 1. Select all employees for a given organization 2. Select employee for a given employeeID EmployeeID Name OrganizationID Results in multiple select statements EmployeeID EmployeeID ...
- 20. Denormalization Secondary indexes 1. Select all employees for a given organization 2. Select employee for a given employeeID EmployeeID Name OrganizationID Performance impact ... CREATE SECONDARY INDEX Organization OrganizationID Name Employee EmployeeID OrganizationID Firstname Lastname Email ... Relational model
- 21. Denormalization PROS Fast reads One query per request (usually) Scalable (probably) CONS Complex data management Can be extremely hard and complex on insert/update/delete Need to know all queries upfront
- 22. UDTs CREATE TABLE keyspace.organization ( organizationid bigint PRIMARY KEY, name text, employees list<frozen<employee>> ); CREATE TYPE test.employee ( employeeid bigint, firstname text, lastname text, email text ); OrganizationID Name Employees Employee Employee ...
- 23. UDTs PROS Fast(er) reads One query per request Scalable (should be!!) Indexing? CONS Complex data management No partial updates Need to know all queries upfront Indexing?
- 24. Blob data CREATE TABLE keyspace.organization ( organizationid bigint PRIMARY KEY, name text, employees text / blob ); OrganizationID Name Employees Employees list as a JSON text or a serialized objects blob JSON text or serialized objects
- 25. Blob data PROS Fast reads One query per request No need to serialize into JSON CONS Complex data management No partial updates Need to know all queries upfront No indexing option
- 26. Relational database PROS It’s made for relational data CONS Scaling Availability Fault tolerance Performance
- 28. Leveraging UDTs
- 29. Use case Highly nested data model Impossible to denormalize Fairly simple access patterns Top level (root) entity
- 30. Data model Root entity Child entity Child entity Child entity Child entityChild entity Child entity Child entity Child entity Child entity Child entity Child entity Child entity Child entity Child entity Child entity Child entity Child entity Child entity Child entity Child entity Child entity Child entity
- 31. How to insert data Insert AS JSON (2.2+) Inserted as string, stored as a column type Easy to manage and debug Keep track of the data size!!!
- 32. Spark dataframe UDT mapping dataframe.as("parent").join( child.groupBy(seq.map(col): _*) .agg(collect_list(struct(columns.map(col): _*)) .alias(alias)), seq, joinType ) dataframe.join(child .withColumn(alias, struct(child.columns.map(col): _*)) .select(joinColumn, alias), Seq(joinColumn), joinType) One to many One to one
- 33. Inserting from Spark // Save to cassandra dataframe.write .format("org.apache.spark.sql.cassandra") .options(Map( "keyspace" -> s"$keyspace", "table" -> s"$table" )) .mode(SaveMode.Append) .save
- 34. Indexing UDTs Not possible with just Cassandra Lucene/Solr based secondary index Indexing of fields on nested UDTs Field analyzers
- 35. Solr schema example <?xml version="1.0" encoding="UTF-8" standalone="no"?> <schema name="solrSchema" version="1.5"> <types> <fieldType class="org.apache.solr.schema.TrieIntField" name="TrieIntField"/> <fieldType class="org.apache.solr.schema.TrieDateField" name="TrieDateField"/> ... <fields> <field docValues="true" indexed="true" multiValued="false" name="partition_key" stored="true" type="TrieIntField"/> <field docValues="true" indexed="true" multiValued="false" name="clustering_key" stored="true" type="TrieDateField"/> <field docValues="true" indexed="true" multiValued="false" name="some_type.id" stored="true" type="TrieIntField" /> <field docValues="true" indexed="true" multiValued="false" name="some_type.some_other_type.id" stored="true" type="TrieIntField" /> ... </fields> <uniqueKey>(partition_key,clustering_key)</uniqueKey> </schema>
- 36. But will it blend?
- 37. Closing notes Cassandra data model supports a lot of use cases Data modeling skills are required Relational model is hard but not impossible Additional tools in the ecosystem Don’t be stubborn
- 38. Q&A