Data warehouse testing

Er. Nawaraj Bhandari
Data Warehouse/Data Mining
Chapter 4:

Data Warehouse Back End Tools
Data warehousing systems use various data extraction and cleaning tools, and
load and refresh utilities for populating data warehouses.
 Data Cleaning
 Load
 Refresh

Data Cleaning
The data warehouse involves large volumes of data from multiple sources, which
can lead to a high probability of errors and anomalies in the data.
Inconsistent field lengths, inconsistent descriptions, inconsistent value
assignments, missing entries and violation of integrity constraints are some of the
examples.
The three classes of data cleaning tools are popularly used to help detect data
anomalies and correct them:
 Data migration tools allow simple transformation rules to be specified.
 Data scrubbing tools use domain-specific knowledge to do the scrubbing of
data. Such tools help to maintain integrity of an entity.
 Data auditing tools make it possible to discover rules and relationships by
scanning data. Thus, such tools may be considered variants of data mining
tools.

Load
 After extracting, cleaning, and transforming, data will be loaded into the data
warehouse. A load utility has to allow the system administrator to monitor
status, to cancel, suspend and resume a load, and to restart after failure with no
loss of data integrity.
 Sequential loads can take a very long time to complete especially when it deals
with terabytes of data. Therefore, pipelined and partitioned parallelism are
typically used.
 Also incremental loading over full load is more popularly used with most
commercial utilities since it reduces the volume of data that has to be
incorporated into the data warehouse.

Refresh
 Refreshing a warehouse consists in propagating updates on source data to
correspondingly update the base data and derived data stored in the
warehouse.
 There are two sets of issues to consider: when to refresh, and how to refresh.
Usually, the warehouse is refreshed periodically (e.g., daily or weekly).
 Only if some OLAP queries need current data, it is necessary to propagate every
update. The refresh policy is set by the warehouse administrator, depending on
user needs and may be different for different sources.

Data Warehouse Tuning
The process of applying different strategies in performing different operations of
data warehouse such that performance measures will enhance is called data
warehousing tuning.
For this, it is very important to have a complete knowledge of data warehouse.
We can tune the different aspects of a data warehouse such as performance, data
load, queries, etc

Data Warehouse Tuning
Tuning a data warehouse is a difficult procedure due to following:
 Data warehouse is dynamic; it never remains constant.
 It is very difficult to predict what query the user is going to post in the future.
 Business requirements change with time.
 Users and their profiles keep changing.
 The user can switch from one group to another.
 The data load on the warehouse also changes with time.

Computation of Data Cubes
Data warehouses contain huge volumes of data.
 OLAP servers demand that decision support queries to answered in the order of
seconds.
 It is crucial for data warehouse systems to support highly efficient cube
computation techniques, access methods and query processing techniques.

Efficient Computation of Data
Cubes
 At the core of multidimensional data analysis is the efficient computation of
aggregations across many sets of dimensions.
 In SQL’s terms, these aggregations are referred to as group-by’s.
 Each group-by can be represented by a cuboid.
 Where the set of group-by’s forms a lattice of cuboids defining a data cube.

Data Warehouse - Testing
Testing is very important for data warehouse systems to make them work
correctly and efficiently. There are three basic levels of testing performed on a
data warehouse:
 Unit testing
 Integration testing
 System testing

Unit Testing
 In unit testing, each component is separately tested.
 Each module, i.e., procedure, program, SQL Script, Unix shell is tested.
 This test is performed by the developer.

Integration Testing
 In integration testing, the various modules of the application are brought
together and then tested against the number of inputs.
 It is performed to test whether the various components do well after integration.

System Testing
 In system testing, the whole data warehouse application is tested together.
 The purpose of system testing is to check whether the entire system works
correctly together or not.
 System testing is performed by the testing team.
 Since the size of the whole data warehouse is very large, it is usually possible to
perform minimal system testing before the test plan can be enacted.

References
1. Sam Anahory, Dennis Murray, “Data warehousing In the Real World”, Pearson
Education.
2. Kimball, R. “The Data Warehouse Toolkit”, Wiley, 1996.
3. Teorey, T. J., “Database Modeling and Design: The Entity-Relationship Approach”,
Morgan Kaufmann Publishers, Inc., 1990.
4. “An Overview of Data Warehousing and OLAP Technology”, S. Chaudhuri,
Microsoft Research
5. “Data Warehousing with Oracle”, M. A. Shahzad
6. “Data Mining Concepts and Techniques”, Morgan Kaufmann J. Han, M Kamber
Second Edition ISBN : 978-1-55860-901-3

Data warehouse testing

More Related Content

What's hot (20)

Similar to Data warehouse testing (20)

More from Er. Nawaraj Bhandari (20)

Recently uploaded (20)

Data warehouse testing