Data warehouse 11 introduction to data transformation
- 1. Data Warehouse: 11 Data Transformation Prof Neeraj Bhargava Vaibhav Khanna Department of Computer Science School of Engineering and Systems Sciences Maharshi Dayanand Saraswati University Ajmer
- 2. Warehouse is a Specialized DB Standard (Operational) DB âą Mostly updates âą Many small transactions âą Mb - Gb of data âą Current snapshot âą Index/hash on p.k. âą Raw data âą Thousands of users (e.g., clerical users) Warehouse (Informational) âą Mostly reads âą Queries are long and complex âą Gb - Tb of data âą History âą Lots of scans âą Summarized, reconciled data âą Few users (e.g., decision-makers, analysts)
- 3. Typical OLTP Data Model
- 4. Typical Data Warehouse Data Model
- 5. Other Data Warehouse Changes âą New descriptive attributes âą New business activity attributes âą New classes of descriptive attributes âą Descriptive attributes become more refined âą Descriptive data are related to one another âą New source of data
- 6. The Reconciled Data Layer âą Typical operational data is: â Transient â Not historical â Restricted in scopeânot comprehensive â Sometimes poor qualityâinconsistencies and errors âą After ETL, data should be: â Detailedânot summarized yet â Historicalâperiodic â Comprehensiveâenterprise-wide perspective â Timelyâdata should be current enough to assist decision-making â Quality controlledâaccurate with full integrity
- 7. Types of Data in DW âą Business Data - represents meaning â Real-time data (ultimate source of all business data) â Reconciled data â Derived data âą Metadata - describes meaning â Build-time metadata â Control metadata â Usage metadata âą Data as a product* - intrinsic meaning â Produced and stored for its own intrinsic value â e.g., the contents of a text-book
- 8. The ETL Process âą Capture/Extract âą Scrub or data cleansing âą Transform âą Load and Index âą ETL = Extract, transform, and load
- 9. Capture/ExtractâŠobtaining a snapshot of a chosen subset of the source data for loading into the data warehouse Static extract = capturing a snapshot of the source data at a point in time Incremental extract = capturing changes that have occurred since the last static extract
- 10. Thank You