Schema less table & dynamic schema
Download as PPTX, PDF3 likes4,215 views
The document explores the concepts of schema, schemaless, and implicit schemas in data storage, highlighting the pros and cons of each approach. It discusses the challenges of maintaining data integrity with a schemaless design and offers solutions like NoSQL databases and the use of custom columns in relational databases. The conclusion emphasizes the importance of defining schemas for data integrity and suggests using schemaless structures only when absolutely necessary.
Schema less table & dynamic schema
- 1. Schema-less table & Dynamic Schema Davide Mauri dmauri@solidq.com @mauridb
- 2. Davide Mauri • Microsoft SQL Server MVP • Works with SQL Server from 6.5, on BI from 2003 • Specialized in Data Solution Architecture, Database Design, Performance Tuning, High-Performance Data Warehousing, BI, Big Data • President of UGISS (Italian SQL Server UG) • Regular Speaker @ SQL Server events • Consulting & Training, Mentor @ SolidQ • E-mail: dmauri@solidq.com • Twitter: @mauridb • Blog: http://sqlblog.com/blogs/davide_mauri/default.aspx
- 3. Agenda • Schema, Schemaless & Implicit Schemas • Possible solutions • Conclusion
- 4. Schema • “A priori” definition of data structures • Allows data to be inserted if and only if it is compatible with the schema • Es: RDBMS Table, XML Schema, Class, Struct
- 5. Schemaless (?) • No definition at all on the data you expect to have. • Unstructured data. • For example: text files, binary files • with no metadata and no position-based format • In one word: chaos
- 6. Implicit Schema • In reality a schema always exists, albeit implicit • Otherwise it would be impossible to handle data
- 7. Implicit Schema Any data that doesn't fit this implicit schema will not be manipulated properly, leading to errors. (Schemaless data structures, Martin Fowler)
- 8. Pros • Flexibility • Easy to manage • actually, almost no management at all • Easy to be extended • Just add a new element and you’re done • Easy to be used • No mismatch between OOP and other models
- 9. Cons • Schema information are hidden somewhere • Scattered all across the codebase • It’s really difficult to keep under control the chaos that can emerge • For example two different element that contains the same information • CustomerName and Customer_Name • You still need to have a sort of «First Normal Form» in order to avoid inconsistency and code inefficiencies
- 10. Cons • It’s really difficult to define and maintain integrity constraints • Data Integrity is a value that must be preserved! • Otherwise we’ll have data, not information • XML Schema were born for that specific reason • Without Data Integrity, the process of extracting information from data becomes • Difficult • Expensive • Untrustable
- 11. Words of Wisdom «Schemaless => implicit schema = bad. Prefer an explicit schema» (Schemaless data structures, Martin Fowler)
- 12. But if we need it anyway? • What if my use case is one that perfectly fits the need for a implicit schema? • The only possible solution are the so-called «No-SQL» databases • Document Database or Key-Value store? • How can I integrate it into already existing database? • Integration does not come for free!
- 13. Schemaless & RDBMS • (Usually) Are the exact opposite extremes • Still is a very common request • CRM, eCommerce, ERPs…. • Schemaless is used not only for pure data persistence
- 14. Solution within an RDBMS • «Custom» columns • Custom1, Custom2 • In-Table Data Structures • BLOB, XML, JSON, «Complex» columns • Entity-Attribute-Value Models
- 15. «Custom» Columns • A problem until SQL Server 2008 • Space is still used for fixed length column even if they contain a NULL value • With SQL Server 2008 the «Sparse Column» feature comes to help • Helps to make the schema easily modifiable, even in presence of existing data • Changes to the schema must still be done with «ALTER TABLE»
- 16. «Custom» Columns • Sparse Columns • Are Columns at 100% • Optionally you can have *all* the Sparse Columns returned as a single XML column • «Column Set» • Make development easier • Do not take space if not used • But use more space when used
- 17. Demo Dynamic Schema & Sparse Columns
- 18. In-Table Data Structures • Complete support for XML • XPath/XQuery • XML Index • Performance «Good Enough» • But not optimal (compared with the equivalent relational approach) • Use a lot of space
- 19. In-Table Data Structures • XML Sometimes needs some help to boost performance • Would be nice to be able to «promote» elements to turn them into real columns • Must be done manually using a choice of • Triggers • Stored Procedure • Data Access Layer • Service Broker
- 20. In-Table Data Structures • JSON support is still missing in SQL Server • But others database like PostgreSQL already have it… • …so we can see it coming to MS Platform too • Right now one solution is to use SQLCLR • Solutions available surfing the web: • http://www.sqlservercentral.com/articles/SQLCLR/74160/ • http://www.json4sql.com/examples.html • There is also a pure T-SQL solution • https://www.simple-talk.com/sql/t-sql-programming/consuming-json-strings- in-sql-server/
- 21. In-Table Data Structures • Blob is an option if you just need to do persistence • Blob can be stored in different way • «Classic» blob inside SQL Server pages & extents • Blob in a filestream • Blob in a filetable
- 22. Demo Dynamic Schema & In-Table Data Structures
- 23. Entity-Attribute-Values • Old and very common technique to store attribute-value pairs • Some well-known samples: Wordpress • Works on any RDBMS • No «special» features required • There’s a huge debate around it • http://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_mo del • But until SQL 2005 no true alternative
- 25. Entity-Attribute-Values • Offers maximum flexibility • No real control over data types. • Options to deal with data types • All strings • SQL Variant • One-Column-Per-Type • Complex query pattern for «AND» predicates between attributes • «Return all the entities that have «CPU=i7» and «Display=15.4’»
- 26. Entity-Attribute-Values • Queries requires the implementation of a relational operator not implemented in common RDMBS • «Relational Division» • Document and well explained in theory • It is quite easy to implement it. Follow theory + add some pepper to boost performances
- 27. Relational Division • Let’s get back to theory a little bit, in order to see the problem from a more open perspective: Dividend Divisor Result Remainder 𝛼 𝛽
- 28. Relational Division • How do we implement the division? • Thanks to Codd and the relational theory we already have the solution 28
- 29. Relational Division • Thanks to relational algebra we know that the division is expressed as • Generate all possibile pairings • Remove existing pairing • (Now we’ve found all pairings that are NOT answers) • Remove the non-answers from the dividend 29
- 30. Demo Dynamic Schema & EAV
- 31. Conclusions • It works! • Performance more than good • Choose the solution that better fits your use-case • Search for attributes only? • Persistence only? • Search for attributes & values? • Performance read, write, read/write?
- 32. Conclusions • Use it if and only if when really needed • Always remeber the «Words of Wisdom» • If you can define and use a schema. • It may seem «not cool» and convoluted but in the long term is the best solution. • *data* *must* *be* *turned* *into* *information* • Sooner or later • Without metadata (a schema) it’s really really really hard!
- 33. Questions?
- 34. Thanks! • If you want to rate this session on my SpeakerScore page: • www.speakerscore.com • Feedback Key: TZQL
- 35. Demo Material • Can be found here • http://1drv.ms/1Av5mb5 • Everything is release under the Creative Common Attribution- NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) • http://creativecommons.org/licenses/by-nc-sa/4.0/
Editor's Notes
- #7: http://martinfowler.com/articles/schemaless/