The Analytics Data Store: Information Supply Framework
7 likes6,004 views
The document discusses the evolution of data warehousing, particularly focusing on the analytics data store (ADS) and its integration into modern data architectures, referred to as DW 3.0. It outlines various types of data—including enterprise operational data, process data, and complex data—and highlights the importance of capturing and analyzing extensive datasets for improved decision-making. The content emphasizes the significance of statistical analysis and the ability to manage data flows effectively within an organization.
The Analytics Data Store: Information Supply Framework
- 1. The Analytics Data Store Martyn Jones Extending the Data Warehouse Architecture Cambriano Energy 2015 -
- 3. Data Warehousing Big Data Business Intelligence Statistics Analytics
- 4. Confused by Big Data? Martyn Richard Jones 2015 – martynjones.eu Cambriano Energy 2015 - Published by goodstrat.com You should be!
- 5. Martyn Richard Jones 2015 – martynjones.eu Cambriano Energy 2015 - Published by goodstrat.com
- 6. NOW! Build me an ADS... Martyn Richard Jones 2015 – martynjones.eu Cambriano Energy 2015 - Published by goodstrat.com
- 9. Enterprise Operational Data – This is data that is used in applications that support the day to day running of an organisation’s operations. Typical data items in this space are sales transactions, purchase transactions, product information, client and contact information. Enterprise Operational Data may also include complexly structured data, such as contracts and other business documents. Applications in this space may include production control, logistics and stock control, as well as purchase order, supply chain management, management accounting and human resource modules.
- 10. Enterprise Process Data – This is measurement and management data collected to show how the operational systems are performing. In the past the recording of events went down to the level of a completed transaction – with a start and an end and nothing in between, and as transactions were kept as simple as possible, to maximize performance and throughput and minimise the risk of failure, very little process data was captured. Now, especially with the advent of Business Process Management and Web Logs, we collect a whole array of transaction and process performance data that was never previously captured.
- 11. Enterprise Process Data – This is measurement and management data collected to show how the operational systems are performing. In the past the recording of events went down to the level of a completed transaction – with a start and an end and nothing in between, and as transactions were kept as simple as possible, to maximize performance and throughput and minimise the risk of failure, very little process data was captured. Now, especially with the advent of Business Process Management and Web Logs, we collect a whole array of transaction and process performance data that was never previously captured.
- 12. Internal digital data DW 3.0 Information Supply Framework External digital data Data logistics Operational Data Store Data Warehouse Analytics Data Store Data Marts Statistical Analysis Business Intelligence Scenarios Data logistics Primary data flow Secondary data flow Operational applications Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu Cambriano Energy 2015 -
- 13. EDW ADS DM DM DM Statistical analysis ETL T/ETL ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Message Adapter Message Queue OLTP Staging ODS ETLT/ETL Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 TL DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter Message Adapter Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu Cambriano Energy 2015 -
- 14. EDW ADS DM DM DM Statistical analysis ETL T/ETL ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Message Adapter Message Queue OLTP Staging ODS ETLT/ETL Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 TL DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter Message Adapter Data Sources – This element covers all the current sources, varieties and volumes of data available which may be used to support processes of 'challenge identification', 'option definition', decision making, including statistical analysis and scenario generation. Cambriano Energy 2015 - Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu
- 15. EDW ADS DM DM DM Statistical analysis ETL T/ETL ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Message Adapter Message Queue OLTP Staging ODS ETLT/ETL Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 TL DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter Message Adapter Core Data Warehousing – This is a suggested evolution path of the DW 2.0 model. It faithfully extends the Inmon paradigm to not only include unstructured and complex data but also the information and outcomes derived from statistical analysis performed outside of the Core Data Warehousing landscape. Cambriano Energy 2015 - Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu
- 16. EDW ADS DM DM DM Statistical analysis ETL T/ETL ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Message Adapter Message Queue OLTP Staging ODS ETLT/ETL Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 TL DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter Message Adapter Core Statistics – This element covers the core body of statistical competence, especially but not only with regards to evolving data volumes, data velocity and speed, data quality and data variety. Cambriano Energy 2015 - Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu
- 17. ADS Statistical analysis ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter Cambriano Energy 2015 - Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu INTO THE ZONE!
- 18. ADS Statistical analysis ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter Complex Data – This is unstructured or highly complexly structured data contained in documents and other complex data artefacts, such as multimedia documents. Cambriano Energy 2015 - Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu
- 19. ADS Statistical analysis ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter Event Data – This is an aspect of Enterprise Process Data, and typically at a fine-grained level of abstraction. Here are the business process logs, the internet web activity logs and other similar sources of event data. The volumes generated by these sources will tend to be higher than other volumes of data, and are those that are currently associated with the Big Data term, covering as it does that masses of information generated by tracking even the most minor piece of 'behavioural data' from, for example, someone casually surfing a web site. Cambriano Energy 2015 - Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu
- 20. ADS Statistical analysis ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter Infrastructure Data – This aspect includes data which could well be described as signal data. Continuous high velocity streams of potentially highly volatile data that might be processed through complex event correlation and analysis components. Cambriano Energy 2015 - Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu
- 21. ADS Statistical analysis ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter Event Applicance – This puts the dynamic data collation, selection and reduction functionality as close to the point of event data generation as physically possible. Cambriano Energy 2015 - Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu
- 22. ADS Statistical analysis ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter Signal Applicance – This puts the dynamic data collation, selection and reduction functionality as close to the point of continuous streaming data generation as physically possible. Cambriano Energy 2015 - Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu
- 23. ADS Statistical analysis ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter Distributed Inter Process Communication – Different forms of messaging allow high volumes of data to be transmitted in near real time. Cambriano Energy 2015 - Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu
- 24. ADS Statistical analysis ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter Staging and Reduction – Traditional data staging combined with in-line data reduction. Cambriano Energy 2015 - Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu
- 25. ADS Statistical analysis ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter ET(A)L – Extending ETL to include data analytics components tightly integrated into parallel ETL job streams. Cambriano Energy 2015 - Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu
- 26. ADS Statistical analysis ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter ADS – The Analytics Data Store. 1. Statistics oriented 2. Integrated by focus area 3. Variable volatility 4. Time variant Cambriano Energy 2015 - Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu
- 27. ADS Statistical analysis ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter Statistical Analysis – Qualitative analysis. Diagnostic analysis, predictive analysis, speculative analysis, data mining, data exploration, modelling. Cambriano Energy 2015 - Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu
- 28. ADS Statistical analysis ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 DW 3.0 Information Supply Framework Core Data Warehousing Core Statistics Data Source s Message Adapter Scenarios and outcomes – 1. Snapshots of outcomes of scenario analysis as the process of analyzing possible future events by generating alternative possible outcomes. 2. Captured outcomes of statistical analysis. Cambriano Energy 2015 - Published by goodstrat.com Martyn Richard Jones 2015 – martynjones.eu
- 29. ADS Statistical analysis ET(A)L Staging & Reduction Signal Appliance Message Adapter Message Queue Infrastructur e Data Write back Complex data Event Data Event Appliance Scenario 1 Scenario 2 Scenario 3 DW 3.0 Information Supply Framework Martyn Richard Jones 2015 – martynjones.eu Core Data Warehousing Core Statistics Data Source s Message Adapter Write back – The ability to append data, update data and enrich data within the Analytics Data Store, and to provide scenario data to the Core Data Warehousing. Cambriano Energy 2015 - Published by goodstrat.com
- 30. UNCOVER UNDERSTAND USE KNOWLEDGE INFORMATION DATA The Iniciativa Information Management Pyramid Copyright © 2000-2015 Iniciativa Org, S.L. Martyn Richard Jones 2015 – martynjones.eu Cambriano Energy 2015 - Published by goodstrat.com STEP STEP BY
- 31. UNCOVER UNDERSTAND USE KNOWLEDGE INFORMATION DATA The Iniciativa Information Management Pyramid Copyright © 2000-2015 Iniciativa Org, S.L. DW 3.0 Aligning with Statistics Martyn Richard Jones 2015 – martynjones.eu Cambriano Energy 2015 - Published by goodstrat.com DW 1.0 Building the Data Warehouse DW 2.0 Adding Unstructured Data
- 32. The Analytics Data Store Martyn Jones Extending the Data Warehouse Architecture Cambriano Energy 2015 - Have any questions about the Analytics Data Store and DW 3.0? Feel free to connect via Twitter, Facebook and the Cambriano Energy website. Alternatively you can contact me via my personal web-site at martynjones.eu Also, please checkout my blog, Good Strat, which deals with organisational strategy and information management.