The Key to Big Data Modeling: Collaboration
3 likes1,091 views
The document discusses key aspects of big data modeling, emphasizing the need for data modeling in an era dominated by vast and complex data sets. It highlights the differences between traditional and NoSQL data modeling approaches and outlines principles for effective collaboration in data modeling efforts. Furthermore, it advocates for understanding data semantics and leveraging agile methodologies to enhance data model quality and sustainability.
![© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 6
New Landscape - NoSQL
KEY VALUE
DATABASES
GRAPH
DATABASES
DOCUMENT
STORES
MongoDB
MUMPS Datab
ase
ObjectDatabas
e++
OrientDB
PostgreSQL
Qizx
RethinkDB
Rocket U2
Sedna
SimpleDB
Solr
TokuMX
OpenLink
Virtuoso
OpenLinkVirtuoso
Oracle Spatial and
Graph
Oracle NoSQL
Database
OrientDB
OQGRAPH
Profium Sense
R2DF
ROIS
Semblent
Lionsgate
sones GraphDB
SPARQLCity
Sqrrl Enterprise
Stardog
Teradata Aster
Titan
TripleBit
VelocityGraph
VertexDB
VivaceGraph
Weaver
WhiteDB
OhmDB
Redis
XAP
KV - solid-state drive or
rotating disk[edit]
Aerospike
BigTable
CDB
Clusterpoint Database
Server
Couchbase Server
FairCom c-treeACE
GT.M
Hibari
Keyspace
LevelDB
LMDB
MemcacheDB (using
Berkeley DB or LMDB)
MongoDB
NoSQLz
Coherence
Oracle NoSQL Database
OpenLink Virtuoso
Tarantool
Tokyo Cabinet
Tuple space
KV - eventually
consistent
Apache Cassandra
Dynamo
Oracle NoSQL Database
Project Voldemort
Riak
OpenLink Virtuoso
KV – ordered
Berkeley DB
FairCom c-treeACE/c-
treeRTG
FoundationDB
HyperDex
IBM Informix C-ISAM
InfinityDB
LMDB
MemcacheDB
NDBM
KV - RAM[edit]
Aerospike
Coherence
Hazelcastmemcached
OpenLink Virtuoso
BaseX
Cloudant
Clusterpoint
Database
Couchbase
Server
CouchDB
CrateIO
DocumentDB
Elasticsearch
eXist
HyperDex
Informix
Jackrabbit
Lotus
Notes (IBM
Lotus Domino)
MarkLogic
AllegroGraph
ArangoDB
Blazegraph
Bitsy
BrightstarDB
Cayley
DEX/Sparksee[2]
Filament
GraphBase
Graphd
Graph Engine[3]
Grapholytic
Horton
HyperGraphDB
IBM System G Native
Store
InfiniteGraph
InfoGrid
jCoreDB Graph
Neo4j
OntotextGraphDB
Orly](https://image.slidesharecdn.com/bigdatamodelingandcollaboration082617-150831165827-lva1-app6892/85/The-Key-to-Big-Data-Modeling-Collaboration-6-320.jpg)
The Key to Big Data Modeling: Collaboration
- 1. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 1 Aug 26th, 2015 Webinar, By Len Silverston, Universal Data Models, LLC Sponsored by Embarcadero Technologies
- 2. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 2 Purpose Share Keys to Big Data Modeling and How to Collaborate
- 3. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 3 Agenda • Big Data Overview • Data Modeling in Big Data • Collaboration Principles
- 4. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 4 Big Data Big data is a broad term for data sets so large or complex that traditional data processing applications are inadequate. Wikipedia 3Vs – Volume, Velocity, Variety By 2020 - 44 zettabytes! Mostly unstructured
- 5. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 5 Unstructured Data Information that either does not have a pre- defined data model or is not organized in a pre- defined manner. Wikipedia How can data have no structure? Is "unstructured" data merely unmodeled?* * Structure, Models and Meaning’ Seth Grimes, Information Week,
- 6. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 6 New Landscape - NoSQL KEY VALUE DATABASES GRAPH DATABASES DOCUMENT STORES MongoDB MUMPS Datab ase ObjectDatabas e++ OrientDB PostgreSQL Qizx RethinkDB Rocket U2 Sedna SimpleDB Solr TokuMX OpenLink Virtuoso OpenLinkVirtuoso Oracle Spatial and Graph Oracle NoSQL Database OrientDB OQGRAPH Profium Sense R2DF ROIS Semblent Lionsgate sones GraphDB SPARQLCity Sqrrl Enterprise Stardog Teradata Aster Titan TripleBit VelocityGraph VertexDB VivaceGraph Weaver WhiteDB OhmDB Redis XAP KV - solid-state drive or rotating disk[edit] Aerospike BigTable CDB Clusterpoint Database Server Couchbase Server FairCom c-treeACE GT.M Hibari Keyspace LevelDB LMDB MemcacheDB (using Berkeley DB or LMDB) MongoDB NoSQLz Coherence Oracle NoSQL Database OpenLink Virtuoso Tarantool Tokyo Cabinet Tuple space KV - eventually consistent Apache Cassandra Dynamo Oracle NoSQL Database Project Voldemort Riak OpenLink Virtuoso KV – ordered Berkeley DB FairCom c-treeACE/c- treeRTG FoundationDB HyperDex IBM Informix C-ISAM InfinityDB LMDB MemcacheDB NDBM KV - RAM[edit] Aerospike Coherence Hazelcastmemcached OpenLink Virtuoso BaseX Cloudant Clusterpoint Database Couchbase Server CouchDB CrateIO DocumentDB Elasticsearch eXist HyperDex Informix Jackrabbit Lotus Notes (IBM Lotus Domino) MarkLogic AllegroGraph ArangoDB Blazegraph Bitsy BrightstarDB Cayley DEX/Sparksee[2] Filament GraphBase Graphd Graph Engine[3] Grapholytic Horton HyperGraphDB IBM System G Native Store InfiniteGraph InfoGrid jCoreDB Graph Neo4j OntotextGraphDB Orly
- 7. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 7 In this age of Big Data, is there ‘less of a need’ or ‘more of a need’ for data modeling? (or ‘no need’ or the ‘same need’)
- 8. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 8 Why Model? 8 DATA Understand Design? Common semantics?
- 9. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 9 UNDERSTAND OVERSTAND What are customers saying about our products? What exactly do we mean by a customer? Is a prospect that has signed a contract but not paid yet, a customer? Is a person that only bought from us over 10 years ago a customer? Is an organization that bought a minor item from us a customer? Is sales volume based on orders, invoices, payments, or GL posts? What are we predicting our sales volume to be this quarter?
- 10. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 10 REQUIRES TEXTCON TEXT
- 11. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 11 Traditional MODEL (and DESIGN) LOAD EXPLORE/ QUERY DATA EXPLORE ‘Schema on write’
- 12. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 12 Big Data LOAD QUERY MODEL NoSQL STORE EXPLORE But Fast and Agile! ‘Schema on read’
- 13. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 13 Data modeling in Big Data Customer - - - NoSQL DATABASE Documents - - - Product - - - Key values - - - Conceptual/ business data model Understanding Logical/physical data model Architecture/Design RELATIONAL DATABASE (i.e., Data warehouse/data mart) May transfer into structured database (using models)
- 14. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 14 Big Data Modeling Considerations • Changes nature of modeling – Later – Modeling for understanding • Design considerations - performance and scalable • Changes where physical structures reside: in code • Shifting functions to programming –Performance –Security –Integrity • Lately, SQL interfaces over NoSQL
- 15. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 15 When to Model First, When To Explore First Explore First When format cannot be predicted in advance (Rapidly changing data structures) When you need to keep ‘data as is’ Continually new sources of data Don’t know if valuable (exploratory) Huge amounts of information (e.g. streaming terabytes per minute) E.g. Cyber terrorism, Sentiment Analysis Model First More predictable data structure When there is some flexibility to modify/conform data Stable and known sources Know that it’s valuable Reasonable amount of information for relational E.g. Customer demographics, Product info, Sales History
- 16. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 16 What Does ‘Agile’ Mean? • Customer solution – deliver value • Flexible • Fast • Iterative • Sustainable – constant pace • Quality design (and efficient) • Human Factors – Communication - face to face – Collaborative – Trust – Motivation – Ongoing reflecting and adjusting Quotes from principles behind the Agile Manifesto can be found at http://agilemanifesto.org/principles.html “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” “Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage” “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale” “Business people and developers must work together daily throughout the project.” “Working software is the primary measure of progress.” “Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.” “Continuous attention to technical excellence and good design enhances agility.” “The most efficient and effective method of conveying information to and within a development team is face-to- face conversation.” “Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.” “Simplicity--the art of maximizing the amount of work not done--is essential.” “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.” “"The best architectures, requirements, and designs emerge from self-organizing teams.” Can we do this in data modeling?
- 17. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 17 What is Agile - What is NOT Agile? • Agile data modeling – Deliver flexible, quality data models that facilitate sustainability and deliver value, in a quick, iterative, collaborative way, building trust and motivation • NOT agile data modeling – Quick & dirty – Excuse to develop more silos – Without quality or understanding PERSON ORGANIZATION PARTY PARTYROLE PARTYRELATIONSHIP PARTYCONTACTMECHANISM FACILITY CASE WORKEFFORT ROLETYPEWORKEFFORTROLE CONTACTMECHANISM SUPPLIERCUSTOMER WORKER PARTNER CONTACTMECHANISMTYPE PROJECTPROGRAM TASK FIXEDASSET ASSIGNMENT WORKEFFORT ASSOCATION COMMUNICATIONEVENT FIXEDASSET PRODUCT GOOD SERVICE
- 18. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 18 How can we perform agile data modeling? 1. Re-use 2. Quick broad brush data model 3. Correct model for correct purpose 4. Prioritize 5. Deliver 6. Understand motivations 7. Have lots of choices available See Article: “Data Modeling’s Role in Agile Development” http://tdwi.org/articles/2010/07/07/data-modeling-agile-development.aspx RE-USE!
- 19. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 19 EPISODE TYPE EPISODE TYPE ID DESCRIPTION HEALTH CARE EPISODE HEALTH CARE EPISODE ID PATIENT PARTY ID (FK) PATIENT ROLE TYPE ID (FK) INCIDENT ID (FK) EPISODE TYPE ID (FK) EPISODE CREATE DATE HEALTH CARE VISIT HEALTH CARE VISIT ID PATIENT PARTY ID (FK) PATIENT ROLE TYPE ID (FK) CONTACT MECHANISM ID (FK) FACILITY ID (FK) FROM DATE THRU DATE INCIDENT INCIDENT ID INCIDENT TYPE ID (FK) INCIDENT DATE DESCRIPTION EMPL RELATED IND INCIDENT TYPE INCIDENT TYPE ID DESCRIPTION PATIENT PARTY ID (FK) ROLE TYPE ID (FK) SYMPTOM SYMPTOM ID HEALTH CARE EPISODE ID (FK) SYMPTOM TYPE ID (FK) DESCRIPTION SYMPTOM TYPE SYMPTOM TYPE ID DESCRIPTION VISIT REASON VISIT REASON ID HEALTH CARE VISIT ID (FK) SYMPTOM ID (FK) HEALTH CARE EPISODE ID (FK) DESCRIPTION HEALTH CARE DELIVERY HEALTH CARE DELIVERY ID HEALTH CARE VISIT ID (FK) HEALTH CARE EPISODE ID (FK) HEALTH CARE OFFERING ID (FK) FROM DATE THRU DATE DELIVERY NOTES Re-use to understand Universal Data Models
- 20. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 20 We must understand the data and therefore continue to develop data models, even in this 'Big Data' era. Come on! Get into the new mindset of today’s Big Data! We need to do things differently today! How can you use the data without first understanding it? Also, it’s important that we all use common semantics. Are you trying to slow us down and continue to try to enforce bureaucracy?! Data Modeler Data Scientist
- 21. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 21 Biggest Issue: “Mine” Data ‘Mine’ing
- 22. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 22 Key to Big Data Modeling: Data ‘Ours’ing
- 23. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 23 Keys To Collaboration • Shared Purpose • Understand Motivations • Develop Trust • Listen • Manage Conflict
- 24. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 24 Vision? Mission? 1. Shared Purpose
- 25. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 25 Can you state the exact mission statement of your organization? (without looking it up first!) Be Honest BE HONEST
- 26. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 26 Position Versus Interest INTERESTS A POSITION A INTERESTS B POSITION B
- 27. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 27 2. Understand Motivations
- 28. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 28 Motivational Model - Sponsorship Map
- 29. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 29 What Does The Business Really Need? • Insight? • Buying behavior? • Assessment? • Prescriptions? • Predictions?
- 30. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 30 Who is the MOST important person to know their motivations? A. The Most Influential Sponsor? B. Your Boss? C. Your Most Difficult Person Who Is the Greatest Obstacle in Your Effort? D. Yourself?
- 31. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 31 Core Elements of TrustKeys to Trust3. Trust
- 32. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 32 • Character –Integrity –Intent – Vulnerability/openness • Competence –Capabilities –Results From “The Speed Of Trust” By Stephen M. R. Covey Keys to Trust
- 33. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 33 To Listen, ACCEPT (A) ware, attention, alert (C) are (C) onfirm, check (E) mpathize (P) urpose (T) otally (with all senses) 4. Listen and ACCEPT
- 34. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 34 Keys to Trust5. Conflict Management
- 35. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 35 What is the first thing to do in a conflict? A. Define your strategy for winning? B. Understand their perspective? C. Don’t react? D. Figure out a win-win? E. Something else?
- 36. © 2014 Universal Data Models, LLC - All Rights Reserved 36 From “Getting Past No: Negotiating with Difficult People”, William Ury Step 1. Don’t React - Observe
- 37. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 37 Don’t React Event Feelings/ Thoughts Emotional Physical Stories Reaction Freeze Flight Fight Mess Step 1. Don’t react
- 38. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 38 Respond Event Feelings/ Thoughts Emotional, Physical, Stories Data Stop – observe. Data? Questions? Response Intelligent Actions Step 1. Don’t react
- 39. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 39 Big Data Modeling Model To Understand – Even if after viewing data Collaboration is the key Find Common Purpose Understand Motivations Develop Trust Listen Conflict Management
- 40. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 40 What Will You Do With This?
- 41. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 41 Questions or More Info? www.universaldatamodels.com lsilverston@univdata.com Twitter: @lensilverston For info on template Models: www.embarcadero.com/products /er-studio-universal-data-models
- 42. © 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 42 Creative Commons Image Attributions Much thanks to those who provided the creative common images in this presentation. Thanks for: Stars https://www.flickr.com/photos/tom_hall_nz/17317951241/sizes/sq/ All rights reserved by Kiwi Tom Sky https://www.flickr.com/photos/cubagallery/9679210392 © All rights reserved by ►CubaGallery License