Data Ware Housing And Data Mining

Chanderprabhu Jain College of Higher Studies & School of Law
Plot No. OCF, Sector A-8, Narela, New Delhi – 110040
(Affiliated to Guru Gobind Singh Indraprastha University and Approved by Govt of NCT of Delhi & Bar Council of India)
Semester: SIXTH Semester
SUBJECT CODE:BCA -302
Name of the Subject:
DATA WARE HOUSING AND DATA MINING

What is Data Mining?
Data Mining is defined as extracting information from
huge sets of data. In other words, we can say that data
mining is the procedure of mining knowledge from data.
The information or knowledge extracted so can be used
for any of the following applications −
•Market Analysis
•Fraud Detection
•Customer Retention
•Production Control
•Science Exploration

Data Mining Applications
Data mining is highly useful in the following domains
−
Market Analysis and Management
Corporate Analysis & Risk Management
Fraud Detection
Apart from these, data mining can also be used in
the areas of production control, customer retention,
science exploration, sports, astrology, and Internet
Web Surf-Aid

Market Analysis and Management
Listed below are the various fields of market where data mining is used −
Customer Profiling − Data mining helps determine what kind of people buy
what kind of products.
Identifying Customer Requirements − Data mining helps in identifying the
best products for different customers. It uses prediction to find the factors that
may attract new customers.
Cross Market Analysis − Data mining performs Association/correlations
between product sales.
Target Marketing − Data mining helps to find clusters of model customers who
share the same characteristics such as interests, spending habits, income, etc.
Determining Customer purchasing pattern − Data mining helps in
determining customer purchasing pattern.
Providing Summary Information − Data mining provides us various
multidimensional summary reports.

Fraud Detection
Data mining is also used in the fields of credit
card services and telecommunication to detect
frauds. In fraud telephone calls, it helps to find
the destination of the call, duration of the call,
time of the day or week, etc. It also analyzes
the patterns that deviate from expected norms.

Data Mining - Systems
There is a large variety of data mining systems available.
Data mining systems may integrate techniques from the
following −
• Spatial Data Analysis
• Information Retrieval
• Pattern Recognition
• Image Analysis
• Signal Processing
• Computer Graphics
• Web Technology
• Business
• Bioinformatics

Data Mining System Classification

Classification Based on the kind of
Knowledge Mined
We can classify a data mining system according to the kind of
knowledge mined. It means the data mining system is
classified on the basis of functionalities such as −
• Characterization
• Discrimination
• Association and Correlation Analysis
• Classification
• Prediction
• Outlier Analysis
• Evolution Analysis

Integrating a Data Mining System with a DB/DW System
If a data mining system is not integrated with a database or a data warehouse
system, then there will be no system to communicate with. The list of
Integration Schemes is as follows −
No Coupling − In this scheme, the data mining system does not utilize any of
the database or data warehouse functions. It fetches the data from a particular
source and processes that data using some data mining algorithms. The data
mining result is stored in another file.
Loose Coupling − In this scheme, the data mining system may use some of
the functions of database and data warehouse system. It fetches the data from
the data respiratory managed by these systems and performs data mining on
that data.
Semi−tight Coupling − In this scheme, the data mining system is linked with a
database or a data warehouse system and in addition to that, efficient
implementations of a few data mining primitives can be provided in the
database.
Tight coupling − In this coupling scheme, the data mining system is smoothly
integrated into the database or data warehouse system. The data mining
subsystem is treated as one functional component of an information system.

Data Preprocessing
Data preprocessing is a data mining technique that involves
transforming raw data into an understandable format. Real-
world data is often incomplete, inconsistent, and/or lacking
in certain behaviors or trends, and is likely to contain many
errors. Data preprocessing is a proven method of resolving
such issues. Data preprocessing prepares raw data for
further processing.
Data preprocessing is used database-driven applications
such as customer relationship management and rule-based
applications (like neural networks).

UNIT 2

Data Warehousing - Overview
The term "Data Warehouse" was first coined by Bill Inmon in
1990. According to Inmon, a data warehouse is a subject
oriented, integrated, time-variant, and non-volatile collection
of data. This data helps analysts to take informed decisions
in an organization.
An operational database undergoes frequent changes on a
daily basis on account of the transactions that take place.
Suppose a business executive wants to analyze previous
feedback on any data such as a product, a supplier, or any
consumer data, then the executive will have no data
available to analyse because the previous data has been
updated due to transactions.

Understanding a Data Warehouse
A data warehouse is a database, which is kept separate from
the organization's operational database.
There is no frequent updating done in a data warehouse.
It possesses consolidated historical data, which helps the
organization to analyze its business.
A data warehouse helps executives to organize, understand,
and use their data to take strategic decisions.
Data warehouse systems help in the integration of diversity
of application systems.
A data warehouse system helps in consolidated historical
data analysis.

Data Warehouse Features
The key features of a data warehouse are discussed below −
Subject Oriented − A data warehouse is subject oriented because it provides
information around a subject rather than the organization's ongoing operations.
These subjects can be product, customers, suppliers, sales, revenue, etc. A data
warehouse does not focus on the ongoing operations, rather it focuses on
modelling and analysis of data for decision making.
Integrated − A data warehouse is constructed by integrating data from
heterogeneous sources such as relational databases, flat files, etc. This
integration enhances the effective analysis of data.
Time Variant − The data collected in a data warehouse is identified with a
particular time period. The data in a data warehouse provides information from the
historical point of view.
Non-volatile − Non-volatile means the previous data is not erased when new data
is added to it. A data warehouse is kept separate from the operational database
and therefore frequent changes in operational database is not reflected in the data
warehouse.

Data Warehouse Applications
As discussed before, a data warehouse helps business
executives to organize, analyze, and use their data for
decision making. A data warehouse serves as a sole part of
a plan-execute-assess "closed-loop" feedback system for
the enterprise management. Data warehouses are widely
used in the following fields −
• Financial services
• Banking services
• Consumer goods
• Retail sectors
• Controlled manufacturing

Types of Data Warehouse
Information processing, analytical processing, and data mining are the
three types of data warehouse applications that are discussed below −
Information Processing − A data warehouse allows to process the
data stored in it. The data can be processed by means of querying,
basic statistical analysis, reporting using crosstabs, tables, charts, or
graphs.
Analytical Processing − A data warehouse supports analytical
processing of the information stored in it. The data can be analyzed by
means of basic OLAP operations, including slice-and-dice, drill down,
drill up, and pivoting.
Data Mining − Data mining supports knowledge discovery by finding
hidden patterns and associations, constructing analytical models,
performing classification and prediction. These mining results can be
presented using the visualization tools.

What is Data Warehousing?
Data warehousing is the process of constructing
and using a data warehouse. A data warehouse is
constructed by integrating data from multiple
heterogeneous sources that support analytical
reporting, structured and/or ad hoc queries, and
decision making. Data warehousing involves data
cleaning, data integration, and data consolidations.

Functions of Data Warehouse Tools and Utilities
The following are the functions of data warehouse tools and
utilities −
Data Extraction − Involves gathering data from multiple
heterogeneous sources.
Data Cleaning − Involves finding and correcting the errors in
data.
Data Transformation − Involves converting the data from
legacy format to warehouse format.
Data Loading − Involves sorting, summarizing, consolidating,
checking integrity, and building indices and partitions.
Refreshing − Involves updating from data sources to
warehouse.

Business Analysis Framework
The business analyst get the information from the data
warehouses to measure the performance and make critical
adjustments in order to win over other business holders in
the market. Having a data warehouse offers the following
advantages −
Since a data warehouse can gather information quickly and
efficiently, it can enhance business productivity.
A data warehouse provides us a consistent view of
customers and items, hence, it helps us manage customer
relationship.
A data warehouse also helps in bringing down the costs by
tracking trends, patterns over a long period in a consistent
and reliable manner.

•Three-Tier Data Warehouse Architecture
•Generally a data warehouses adopts a three-tier architecture. Following are the three
tiers of the data warehouse architecture.
•Bottom Tier − The bottom tier of the architecture is the data warehouse database
server. It is the relational database system. We use the back end tools and utilities to
feed data into the bottom tier. These back end tools and utilities perform the Extract,
Clean, Load, and refresh functions.
•Middle Tier − In the middle tier, we have the OLAP Server that can be implemented in
either of the following ways.
– By Relational OLAP (ROLAP), which is an extended relational database
management system. The ROLAP maps the operations on multidimensional data
to standard relational operations.
– By Multidimensional OLAP (MOLAP) model, which directly implements the
multidimensional data and operations.
•Top-Tier − This tier is the front-end client layer. This layer holds the query tools and
reporting tools, analysis tools and data mining tools.

Data Warehouse Models
From the perspective of data warehouse
architecture, we have the following data
warehouse models −
• Virtual Warehouse
• Data mart
• Enterprise Warehouse

Query Manager Architecture
The following screenshot shows the architecture of a query manager. It includes
the following:
Query redirection via C tool or RDBMS
Stored procedures
Query management tool
Query scheduling via C tool or RDBMS
Query scheduling via third-party software

Data Warehousing - OLAP
Online Analytical Processing Server (OLAP) is
based on the multidimensional data model. It
allows managers, and analysts to get an insight of
the information through fast, consistent, and
interactive access to information. This chapter
cover the types of OLAP, operations on OLAP,
difference between OLAP, and statistical
databases and OLTP.

Types of OLAP Servers
We have four types of OLAP servers −
•Relational OLAP (ROLAP)
•Multidimensional OLAP (MOLAP)
•Hybrid OLAP (HOLAP)
•Specialized SQL Servers

UNIT 3

Data Mining - Classification & Prediction
There are two forms of data analysis that can be used for
extracting models describing important classes or to predict
future data trends. These two forms are as follows −
• Classification
• Prediction
Classification models predict categorical class labels; and
prediction models predict continuous valued functions. For
example, we can build a classification model to categorize
bank loan applications as either safe or risky, or a prediction
model to predict the expenditures in dollars of potential
customers on computer equipment given their income and
occupation.

What is classification?
Following are the examples of cases where the data
analysis task is Classification −
• A bank loan officer wants to analyze the data in order to
know which customer (loan applicant) are risky or which
are safe.
• A marketing manager at a company needs to analyze a
customer with a given profile, who will buy a new
computer.
• In both of the above examples, a model or classifier is
constructed to predict the categorical labels. These labels
are risky or safe for loan application data and yes or no
for marketing data.

What is prediction?
Following are the examples of cases where the data
analysis task is Prediction −
Suppose the marketing manager needs to predict how
much a given customer will spend during a sale at his
company. In this example we are bothered to predict a
numeric value. Therefore the data analysis task is an
example of numeric prediction. In this case, a model or a
predictor will be constructed that predicts a continuous-
valued-function or ordered value.

Classification and Prediction Issues
The major issue is preparing the data for Classification and Prediction. Preparing the
data involves the following activities −
Data Cleaning − Data cleaning involves removing the noise and treatment of
missing values. The noise is removed by applying smoothing techniques and the
problem of missing values is solved by replacing a missing value with most
commonly occurring value for that attribute.
Relevance Analysis − Database may also have the irrelevant attributes. Correlation
analysis is used to know whether any two given attributes are related.
Data Transformation and reduction − The data can be transformed by any of the
following methods.
Normalization − The data is transformed using normalization. Normalization
involves scaling all values for given attribute in order to make them fall within a
small specified range. Normalization is used when in the learning step, the
neural networks or the methods involving measurements are used.
Generalization − The data can also be transformed by generalizing it to the
higher concept. For this purpose we can use the concept hierarchies.

Comparison of Classification and Prediction Methods
Here is the criteria for comparing the methods of Classification and Prediction −
Accuracy − Accuracy of classifier refers to the ability of classifier. It predict the
class label correctly and the accuracy of the predictor refers to how well a given
predictor can guess the value of predicted attribute for a new data.
Speed − This refers to the computational cost in generating and using the
classifier or predictor.
Robustness − It refers to the ability of classifier or predictor to make correct
predictions from given noisy data.
Scalability − Scalability refers to the ability to construct the classifier or predictor
efficiently; given large amount of data.
Interpretability − It refers to what extent the classifier or predictor understands.

Data Mining - Decision Tree Induction
A decision tree is a structure that includes a root node, branches, and leaf nodes.
Each internal node denotes a test on an attribute, each branch denotes the
outcome of a test, and each leaf node holds a class label. The topmost node in the
tree is the root node.

Data Mining - Bayesian Classification
Bayesian classification is based on Bayes' Theorem. Bayesian classifiers
are the statistical classifiers. Bayesian classifiers can predict class
membership probabilities such as the probability that a given tuple
belongs to a particular class.
Baye's Theorem
Bayes' Theorem is named after Thomas Bayes. There are two types of
probabilities −
Posterior Probability [P(H/X)]
Prior Probability [P(H)]
where X is data tuple and H is some hypothesis.
According to Bayes' Theorem,
P(H/X)= P(X/H)P(H) / P(X)

What is Clustering?
Clustering is the process of making a group of abstract
objects into classes of similar objects.
Points to Remember
• A cluster of data objects can be treated as one group.
• While doing cluster analysis, we first partition the set of
data into groups based on data similarity and then assign
the labels to the groups.
• The main advantage of clustering over classification is that,
it is adaptable to changes and helps single out useful
features that distinguish different groups.

Applications of Cluster Analysis
• Clustering analysis is broadly used in many applications such as market
research, pattern recognition, data analysis, and image processing.
• Clustering can also help marketers discover distinct groups in their
customer base. And they can characterize their customer groups based
on the purchasing patterns.
• In the field of biology, it can be used to derive plant and animal
taxonomies, categorize genes with similar functionalities and gain
insight into structures inherent to populations.
• Clustering also helps in identification of areas of similar land use in an
earth observation database. It also helps in the identification of groups
of houses in a city according to house type, value, and geographic
location.
• Clustering also helps in classifying documents on the web for
information discovery.

Requirements of Clustering in Data Mining
The following points throw light on why clustering is required in data mining −
Scalability − We need highly scalable clustering algorithms to deal with large
databases.
Ability to deal with different kinds of attributes − Algorithms should be capable
to be applied on any kind of data such as interval-based (numerical) data,
categorical, and binary data.
Discovery of clusters with attribute shape − The clustering algorithm should be
capable of detecting clusters of arbitrary shape. They should not be bounded to only
distance measures that tend to find spherical cluster of small sizes.
High dimensionality − The clustering algorithm should not only be able to handle
low-dimensional data but also the high dimensional space.
Ability to deal with noisy data − Databases contain noisy, missing or erroneous
data. Some algorithms are sensitive to such data and may lead to poor quality
clusters.
Interpretability − The clustering results should be interpretable, comprehensible,
and usable.

Clustering Methods
Clustering methods can be classified into the
following categories −
• Partitioning Method
• Hierarchical Method
• Density-based Method
• Grid-Based Method
• Model-Based Method
• Constraint-based Method

UNIT 4

Multidimensional Analysis and Descriptive
Mining of Complex Data Objects
Many advanced, data-intensive applications, such as scientiﬁc
research and engineering design, need to store, access, and
analyze complex but relatively structured data objects. These
objects cannot be represented as simple and uniformly
structured records (i.e., tuples) in data relations. Such
application requirements have motivated the design and
developmentofobject-relationalandobject-
orienteddatabasesystems.Bothkindsofsystems deal with the
efﬁcient storage and access of vast amounts of disk-based
complex structured data objects.

Spatial Data Mining
Spatial data mining is the application of data mining
to spatial models. In spatial data mining, analysts
use geographical or spatial information to produce
business intelligence or other results. This requires
specific techniques and resources to get the
geographical data into relevant and useful formats.

Challenges in Web Mining
The web poses great challenges for resource and knowledge discovery based on
the following observations −
The web is too huge − The size of the web is very huge and rapidly increasing.
This seems that the web is too huge for data warehousing and data mining.
Complexity of Web pages − The web pages do not have unifying structure. They
are very complex as compared to traditional text document. There are huge amount
of documents in digital library of web. These libraries are not arranged according to
any particular sorted order.
Web is dynamic information source − The information on the web is rapidly
updated. The data such as news, stock markets, weather, sports, shopping, etc.,
are regularly updated.
Diversity of user communities − The user community on the web is rapidly
expanding. These users have different backgrounds, interests, and usage purposes.
There are more than 100 million workstations that are connected to the Internet and
still rapidly increasing.
Relevancy of Information − It is considered that a particular person is generally
interested in only small portion of the web, while the rest of the portion of the web
contains the information that is not relevant to the user and may swamp desired
results.

Mining Web page layout structure
The basic structure of the web page is based on the
Document Object Model (DOM). The DOM structure refers to
a tree like structure where the HTML tag in the page
corresponds to a node in the DOM tree. We can segment the
web page by using predefined tags in HTML. The HTML
syntax is flexible therefore, the web pages does not follow
the W3C specifications. Not following the specifications of
W3C may cause error in DOM tree structure.
The DOM structure was initially introduced for presentation in
the browser and not for description of semantic structure of
the web page. The DOM structure cannot correctly identify
the semantic relationship between the different parts of a
web page.

Vision-based page segmentation (VIPS)
The purpose of VIPS is to extract the semantic structure of a web page
based on its visual presentation.
Such a semantic structure corresponds to a tree structure. In this tree
each node corresponds to a block.
A value is assigned to each node. This value is called the Degree of
Coherence. This value is assigned to indicate the coherent content in
the block based on visual perception.
The VIPS algorithm first extracts all the suitable blocks from the HTML
DOM tree. After that it finds the separators between these blocks.
The separators refer to the horizontal or vertical lines in a web page that
visually cross with no blocks.
The semantics of the web page is constructed on the basis of these
blocks.

The following figure shows the procedure of VIPS
algorithm −

Data Mining, System Products and Research Prototypes
Although data mining is a young field with many issues that still need to be
researched in depth, there are already great many off-the-shelf data mining system
products and domain-specific data mining application software available. As a
young discipline, data mining has a relatively short history and are constantly
evolving-new data mining systems appear on the market every year; new
functions, features, and visualization tools are added to existing systems on a
constant basis; and efforts toward the standardization of data mining language
have only just begun. Therefore, it is not our intention in this book to provide a
detailed description of commercial data mining systems. Instead, we describe the
features to consider when selecting a data mining product and offer a quick
introduction to a few typical data mining systems How to Choose a Data Mining
System With many data mining system products available on the market, you may
ask, "What kind of system should J choose?" Some people may be under the
impression that data mining systems, like many commercial relational database
systems, share the same well-defined operations and a standard query language,
and behave similarly on common functionalities. If such were the case, the choice
would depend more on the systems' hardware platform, compatibility, robustness,
scalability, price, and service. Unfortunately, this is far from reality.

Social impact of data mining
As data mining is one of the most rapidly changing disciplines with new
technologies and concepts continually under development, academicians,
researchers, and professionals of the discipline need access to the most
current information about the concepts, issues, trends, and technologies in
this emerging field.
Social Implications of Data Mining and Information Privacy:
Interdisciplinary Frameworks and Solutions serves as a critical source
of information related to emerging issues and solutions in data mining and
the influence of political and socioeconomic factors. An immense
breakthrough, this essential reference provides concise coverage of
emerging issues and technological solutions in data mining, and covers
problems with applicable laws governing such issues.

Trends in Data Mining
Data mining concepts are still evolving and here are the latest trends that
we get to see in this field −
• Application Exploration.
• Scalable and interactive data mining methods.
• Integration of data mining with database systems, data warehouse
systems and web database systems.
• Standardization of data mining query language.
• Visual data mining.
• New methods for mining complex types of data.
• Biological data mining.
• Data mining and software engineering.
• Web mining.
• Distributed data mining.
• Real time data mining.
• Multi database data mining.
• Privacy protection and information security in data mining.

THANK YOU

Data Ware Housing And Data Mining

More Related Content

What's hot (20)

Similar to Data Ware Housing And Data Mining (20)

More from CHANDERPRABHU JAIN COLLEGE OF HIGHER STUDIES & SCHOOL OF LAW (20)

Recently uploaded (20)

Data Ware Housing And Data Mining