Everything has changed except us
4 likes1,599 views
The document discusses the evolving challenges and methodologies in data management and architecture, emphasizing the limitations of traditional data warehousing approaches in the context of complex and varied data types. It argues for a shift towards more adaptive data architectures that accommodate real-time analytics and diverse data structures without relying on rigid upfront modeling. The paper concludes that solutions to current data issues require a rethink of architecture over technology alone, advocating for flexible and dynamic data management frameworks.
Everything has changed except us
- 3. Copyright Third Nature, Inc. I never said the “E” in EDW meant “everything”… What do you mean, “Just tables?”
- 5. © Third Nature Inc.© Third Nature Inc. The good news is: we solved the bigness problem Source: Noumenal, Inc.
- 7. © Third Nature Inc.© Third Nature Inc. What makes data “big”? Aside from very large amounts: Hierarchical structures Nested structures Linked structures Encoded values Non‐standard (for a database) types Deep structure Human authored text “big” is better off being defined as “complex” or “hard to manage” Copyright Third Nature, Inc.
- 8. Copyright Third Nature, Inc. Datasets today: Interconnection and Dependency Dynamic models are missing from most data systems today. These drive new workloads, generate different data, need new techniques. Hierarchical Edge Bundles: Visualization of Adjacency Relations in Hierarchical Data, Danny Holten
- 9. Copyright Third Nature, Inc. It’s not the number of genes that determine complexity, it’s the interactions between them. Source: M. Pertea and S. Salzberg/Genome Biology 2010
- 10. Copyright Third Nature, Inc. It’s not the number of genes that determine complexity, it’s the interactions between them. Source: M. Pertea and S. Salzberg/Genome Biology 2010
- 11. Copyright Third Nature, Inc. Categorizing the measurement data we collect The convenient data is the transactional data. ▪ Goes in the DW and is used, even if it isn’t the right measurement. The inconvenient data is observational data. ▪ It’s not neat, clean, or designed into most systems of operation. The difficult and misleading data is declarative data. ▪ What people say and what they do require ground truth. We need an architecture that supports all three categories. Copyright Third Nature, Inc.
- 12. Copyright Third Nature, Inc. Observations Sensor data doesn’t fit well with current methods of collection and storage, or with the technology to process and analyze it. Copyright Third Nature, Inc.
- 14. Copyright Third Nature, Inc. Unstructured is Not Really Unstructured Slide 14 Unstructured data isn’t really unstructured: objects have structure, language has structure. Text can contain traditional structured data elements. The problem is that the content is unmodeled. Our real problem is making implicit structure explicit. Conclusion: the data warehouse must cope with more complex data structures, storage and processing.
- 15. Copyright Third Nature, Inc. The creation, flow and use of data is different for transactions and machine‐generated events Data entry Extract Cleanse Load UseStore Transactions MDM Generate Store Use UseCleanse Program Capture This runs at human speed This runs at machine speed, with slower feedback cycle
- 17. Copyright Third Nature, Inc. The architecture we’ve been using. The general concept of a separate architecture for BI has been around longer, but this paper by Devlin and Murphy is the first formal data warehouse architecture and definition published. 17 “An architecture for a business and information system”, B. A. Devlin, P. T. Murphy, IBM Systems Journal, Vol.27, No. 1, (1988) Slide 17Copyright Third Nature, Inc.
- 18. Copyright Third Nature, Inc. Origins: in 1988 there was only big hair. ▪ No real commercial email, public internet barely started ▪ Storage state of the art: 100MB, cost $10,000/GB ▪ Oracle Applications v1 GL released; SAP goes public, enters US market ▪ Unix is mostly run by long‐haired freaks ▪ Mobile was this This is the context: scarcity of data, of system resources, of automated systems outside core financials, of money to pay for storage.
- 20. Copyright Third Nature, Inc. Data strategy means understanding the context of data use so we can build the right infrastructure Collect new data Monitor Analyze Exceptions Analyze Causes Decide Act Act on the process Act within the process We need to focus on what people do with information as the primary task, not on the data or the technology.
- 21. Copyright Third Nature, Inc. The usage models for conventional BI Collect new data Monitor Analyze Exceptions Analyze Causes Decide Act No problem No idea Do nothing Act on the process Usually days/longer timeframe Act within the process Usually real-time to daily This is what we’ve been doing with BI so far: static reporting, dashboards, ad-hoc query, OLAP
- 22. Copyright Third Nature, Inc. The usage models for analytics and “big data” Collect new data Monitor Analyze Exceptions Analyze Causes Decide Act No problem No idea Do nothing Act on the process Usually days/longer timeframe Act within the process Usually real-time to daily Analytics and big data is focused on new use cases: deeper analysis, causes, prediction, optimizing decisions This isn’t ad-hoc, reporting, or OLAP.
- 28. Copyright Third Nature, Inc. Centralize: that solves all problems! Creates bottlenecks Causes scale problems Enforces a single model
- 32. Copyright Third Nature, Inc. Which is best, 3NF or dimensional? The core assumption that there can be just one big schema model on one big platform is flawed. Answer: neither. We think we can model all the data before use, but that’s a bottleneck. Current techniques for modeling and managing data are too rigid and incapable of describing all the possible relationships.
- 35. Copyright Third Nature, Inc. Why is the choice no schema or hard schema? Simple key‐value files give you flexibility in some areas. Tables give you flexibility in other areas. Which area do you need flexibility in and why? Programs writing data? Files Tables Programs processing data? Programs reading data? Why not flexible schemas instead of either-or?
- 37. Copyright Third Nature, Inc. With too much data the approach has to be inverted The process we still use: 1. Model 2. Collect 3. Analyze The new process is: 1. Collect 2. Analyze 3. Model 4. Promote This is a shift from planned design to evolutionary design for the data warehouse
- 39. Copyright Third Nature, Inc. Workloads OLTP BI Analytics Access Read‐Write Read‐only Read‐mostly Predictability Predictable Unpredictable Fixed path Selectivity High Low Low Retrieval Low Low High Latency Milliseconds < seconds msecs to days Concurrency Huge Moderate 1 to huge Model 3NF, nested object Dim, denorm BWT Task size Small Large Small to huge
- 41. Copyright Third Nature, Inc. Decoupled Data Architecture The core of the data warehouse isn’t the database, it’s the data architecture that the database and tools implement. We need a data architecture that is not limiting: ▪ Deals with data and schema change easily ▪ Does not always require up front modeling ▪ Does not limit the format or structure of data ▪ Assumes a full range of data latencies, from streaming to one‐time bulk loads, both in and out,
- 43. Integrate Manage Decouple data architecture layers Use This implies a new warehouse architecture and data modeling approaches Collect Transactions Observations Declarations
- 45. The technology architecture must change, based on work done with the data: ▪ Collection separate from ▪ Data management separate from ▪ Data delivery and use Data may live in more than one place because it may have more than one model, for more than one use, using more than one engine
- 46. Copyright Third Nature, Inc. Reinforcing relationships keep architectures from changing, despite radical technology shifts Note how only one third is tech Architectural Regime MethodologyTechnology Organization Organization defines where the work is done and the roles. Technology defines what work can be done in a given area. Methodology defines how work is done and what that work is. Slide 49Copyright Third Nature, Inc.
- 49. Copyright Third Nature, Inc. The geography we have been using is out of date The box we created: • not any data, rigidly typed data • not any form, tabular rows and columns of typed data • not any latency, persist what the DB can keep up with • not any process, only queries The digital world was diminished to only what’s inside the box until we forgot the box was there.
- 50. Copyright Third Nature, Inc. Data infrastructure is a platform ▪ Any data – structures, forms ▪ Any latency –in motion, at rest ▪ Any process – query, algorithm, transform ▪ Any access – SQL, API, queue, file movement
- 54. Copyright Third Nature, Inc. CC Image Attributions Thanks to the people who supplied the creative commons licensed images used in this presentation: round hole square peg ‐ https://www.flickr.com/photos/epublicist/3546059144 firemen not noticing fire.jpg ‐ http://flickr.com/photos/oldonliner/1485881035/ pyramid_camel_rider.jpg ‐ http://www.flickr.com/photos/khalid‐almasoud/1528054134/ House on fire ‐ http://flickr.com/photos/oldonliner/1485881035/ glass_buildings.jpg ‐ http://www.flickr.com/photos/erikvanhannen/547701721 Circos, Hierarchical Edge Bundles:Visualization of Adjacency Relations in Hierarchical Data, Danny Holten text composition ‐ http://flickr.com/photos/candiedwomanire/60224567/ Building demolition ‐ https://www.flickr.com/photos/gregpc/4429888820 peek_fence_dog.jpg ‐ http://www.flickr.com/photos/webwalker/114998078/ donuts_4_views.jpg ‐ http://www.flickr.com/photos/le_hibou/76718773/ shady_puppy_sales.jpg ‐ http://www.flickr.com/photos/brizzlebornandbred/5001120150 subway dc metro ‐ http://flickr.com/photos/musaeum/509899161/
- 56. About Third Nature Third Nature is a research and consulting firm focused on new and emerging technology and practices in analytics, business intelligence, information strategy and data management. If your question is related to data, analytics, information strategy and technology infrastructure then you‘re at the right place. Our goal is to help organizations solve problems using data. We offer education, consulting and research services to support business and IT organizations as well as technology vendors. We fill the gap between what the industry analyst firms cover and what IT needs. We specialize in product and technology analysis, so we look at emerging technologies and markets, evaluating technology and hw it is applied rather than vendor market positions.