![Data Warehousing/Mining
Warehousing and Industry
Warehousing is big business
– $2 billion in 1995
– $3.5 billion in early 1997
– Predicted: $8 billion in 1998 [Metagroup]
WalMart has largest warehouse
– 900-CPU, 2,700 disk, 23 TB Teradata system
– ~7TB in warehouse
– 40-50GB per day](https://image.slidesharecdn.com/evolutionofdatawarehouse-250105171732-f70b1818/85/Evolution-of-Data-Warehouse-and-Data-mining-7-320.jpg)
Evolution of Data Warehouse and Data mining
- 1. Data Warehousing/Mining Data Warehouse Evolution TIME 2000 1995 1990 1985 1980 1960 1975 Information- Based Management Data Revolution “Middle Ages” “Prehistoric Times” Relational Databases PC’s and Spreadsheets End-user Interfaces 1st DW Article DW Confs. Vendor DW Frameworks Company DWs “Building the DW” Inmon (1992) Data Replication Tools
- 2. Data Warehousing/Mining What is a Data Warehouse? A Practitioners Viewpoint “A data warehouse is simply a single, complete, and consistent store of data obtained from a variety of sources and made available to end users in a way they can understand and use it in a business context.” -- Barry Devlin, IBM Consultant
- 3. Data Warehousing/Mining A Data Warehouse is... Stored collection of diverse data – A solution to data integration problem – Single repository of information Subject-oriented – Organized by subject, not by application – Used for analysis, data mining, etc. Optimized differently from transaction- oriented db User interface aimed at executive
- 4. Data Warehousing/Mining A Data Warehouse is... (continued) Large volume of data (Gb, Tb) Non-volatile – Historical – Time attributes are important Updates infrequent May be append-only Examples – All transactions ever at WalMart – Complete client histories at insurance firm – Stockbroker financial information and portfolios
- 5. Data Warehousing/Mining Summary Operational Systems Enterprise Modeling Business Information Guide Data Warehouse Catalog Data Warehouse Population Data Warehouse Business Information Interface
- 6. Data Warehousing/Mining Warehouse is a Specialized DB Standard DB Mostly updates Many small transactions Mb - Gb of data Current snapshot Index/hash on p.k. Raw data Thousands of users (e.g., clerical users) Warehouse Mostly reads Queries are long and complex Gb - Tb of data History Lots of scans Summarized, reconciled data Hundreds of users (e.g., decision-makers, analysts)
- 7. Data Warehousing/Mining Warehousing and Industry Warehousing is big business – $2 billion in 1995 – $3.5 billion in early 1997 – Predicted: $8 billion in 1998 [Metagroup] WalMart has largest warehouse – 900-CPU, 2,700 disk, 23 TB Teradata system – ~7TB in warehouse – 40-50GB per day
- 8. Data Warehousing/Mining Types of Data Business Data - represents meaning – Real-time data (ultimate source of all business data) – Reconciled data – Derived data Metadata - describes meaning – Build-time metadata – Control metadata – Usage metadata Data as a product* - intrinsic meaning – Produced and stored for its own intrinsic value – e.g., the contents of a text-book
- 9. Data Warehousing/Mining Data Warehouse Architectures: Conceptual View Single-layer – Every data element is stored once only – Virtual warehouse Two-layer – Real-time + derived data – Most commonly used approach in industry today “Real-time data” Operational systems Informational systems Derived Data Real-time data Operational systems Informational systems
- 10. Data Warehousing/Mining 1 Three-layer Architecture: Conceptual View Transformation of real-time data to derived data really requires two steps Derived Data Real-time data Operational systems Informational systems Reconciled Data Physical Implementation of the Data Warehouse View level “Particular informational needs”