SlideShare a Scribd company logo
Overview of BA Discussion
 Business Analytics (BA)
 Overview
 History
 Types of Business Analytics
 Real world examples
 Challenges
 Relations to Data Mining
Business Analytics (BA) : an
overview
 BA can be considered a subset of Business intelligence
 A set of skills, technologies, applications and practices
 exploration and investigation of past business performance
to gain insight and drive business planning.
 Like Business Intelligence, BA can focus either on the
business as a whole or only on segments of it
 Focuses on developing new insights and understanding
of performance based on data and statistical methods
BA : Short History
 Analytics in business dates far before computing
 Frederick Taylor, father of scientific management, 19th
century
 time management exercises used in industrial settings
 Henry Ford : assembly line pacing used to improve output
and business profitability
 BA becomes widespread when computers were used in
DSS systems in the 60’s
 Evolved into ERP, data warehouses, etc.
Types of Business Analytics
 Reporting or Descriptive Analytics
 Affinity grouping
 Clustering
 Modeling or Predictive analytics
BA: Reporting
 Based on the need to locate and distribute business
insights and experiences
 Often involves ETL procedures used alongside a data
warehousing scheme
 The data is then collected, quantified, and organized
using reporting tools
 Reporting, allows for information describing different
views of an enterprise to come together one place
 A user could query a production and marketing database to
determine if production of a product could be moved closer
to where a product is sold
BA: Affinity grouping
 A tool used by businesses and
organizations to take ideas
and data and organize them.
 Often takes the form of an affinity diagram
 Enables data and ideas stemming from
brainstorming to be sorted into groups
 Sorting is based on their natural relationships
BA: Clustering
 Placing a set of objects into groups (called clusters) so
that the objects in the same cluster are more similar (in
some sense or another) to each other than to those in
other clusters – wikipedia
 Is a main task of explorative data mining and statistical
data analysis
 Clustering is a general task that does not have one set
solution
 Clustering can be hard or fuzzy
 Can be done by people or machines
 The latter is preferred
BA: how do we model clusters?
 Connectivity models – how data can be connected to
other points
 Density models – defining a cluster by determining where
sets of data points are densest
 Distribution model – clusters are modeled using statistical
distributions
 Expectation maximization
BA: Predictive Analysis
 Stems from the desire to predict future events through
analyzing data an enterprise has collected
 Pattern exploitation results in the identification of
opportunities and also risks
 Allow relationships in disparate data to be identified
 Helps guide in decision making in a business
 Is often implemented in the form of data mining
BA : Examples
 Credit company– uses business analytics to track credit risk of
customers as well as matching customers to offerings
 Sales and offers – companies can track customer interaction,
and use that information to determine appropriate product
offerings.
 Sales groups can use BA to optimize inventory and analyze
past sales
 Could measure peak purchasing times for products
 Could decide whether or not to stock poorly selling items
 Give examples of business cases where data mining might be
useful, and describe how data mining would be used
 Preventing credit card fraud through detecting spending patterns
 Inventory management by tracking sales
BA : Challenges
 Acquiring sufficient volumes of high quality data
 Most data acquired in the field is unsorted and appears in
many different formats
 When dealing with high volume data, deciding what is
important and what is noise
 Rapidly reacting storage structures
 BA can influence customer interactions, and as such that
information must be available fast
 Ex: a customized sales pitch
Business Analytics & Data Mining
 Data Mining is an important sub task of Business
Analytics
 Both Predictive analysis and clustering tasks
utilize information retrieved from data mining
 Data mining helps handle some of the specific
problems faced when conducting Business
Analytics
 Dealing with and sorting through large data sets
Data Mining : An Overview
 What is Data Mining ?
 History
 Applications of Data Mining
 Detecting data discrepancies or outliers
 Relationship identification
 Data-Function mapping for modeling/prediction
 Categorizing and Summarizing Data
 Standards
 Challenges
Data Mining : What is it?
 Applying statistical analysis techniques to data
 the goal often being to determine unnoticed patterns or to
collect categorized information
 turns collected data into understandable structures
 Data Mining is often used as a buzz word to describe
processing large amounts of data
 In essence, its correct use relates to discovery of new
things through observation
 Synonymous with knowledge discovery
Data Mining : History
 Though HNC trademarked the term in 1990, hands on
pattern extraction is centuries old
 As long as statistic analysis has existed
 Discoveries in computer science have increasingly
shifted the field from hands on to machine dependent,
this allows for :
 The use of data indexing and DB systems to handle data
efficiently
 The application of statistical algorithms on a large scale,
possibly in a distributed manner, with less error
Data Mining : Use : Application
 Data Mining is often broken into several different
categories of tasks
 Detecting data discrepancies or outliers
 Relationship identification
 Data-Function mapping for modeling/prediction
 Categorizing and Summarizing Data
Data Mining : Finding outliers
 The process of analyzing large, mostly
homogeneous, sets of data and determining
which sets or points
 “go with the flow” and conform with patterns the rest
of the data seem to follow
 do not follow expected results when viewed against
the entire set of data
 An outlier can be a point or set of points, but can
also be defined through other means
 A period of time could yield unexpected results
 Ex. Network Intrusion
Data Mining : Techniques in finding outliers
 Rule Based – deciding a set of rules that
determine an outlier (or what isn’t one)
 Can be fuzzy or hard rules
 Cluster Analysis – As mentioned earlier
 Distance or Standard Deviation – Determining an
average over a data set and marking points that
aren’t within a Deviation or Distance
Applications of Outlier Detection
 Network Intrusion Detection
 Unusual bursts of network activity
 Identity Theft Detection
 Unusual spending or customer activity
 Detecting Software bugs
 Software does not deliver expected outputs
 Sensor event detection
 Monitoring patient health fluctuations in a medical setting
 Preprocessing
 Removing data skews based on extenuating
circumstances
Relationship Discovery: Basics
 Understanding how data is related is a key factor
in trend and knowledge discovery
 This is the definition of data mining
 Ex: Which products are often bought before a major
forecasted storm
 {hamburger buns} => {???}
 With small sets of data, or with correlations that
aren’t subtle (as the one above), identifying
relationships is not as difficult
 With large data sets or subtle relations a
combination of rule generation and data analysis
can be used to expedite the process
Relationship Discovery: How its done
 Since the number of relationships between points
of data could be boundless, two important
concepts are often introduced in relationship
discovery:
 The amount of data within which a relationship
might exist, called the support of a rule.
 The probability that data in the support will verify a
selected rule, called the confidence of a rule.
Relationship Discovery: How its done
 Generally we apply minimum bounds to both the support of
a rule and its confidence to determine relationships
 First : determine possible relationships
 Set a minimum support
 Orders with hamburgers, Orders with hamburger buns
 Other, user specific rules can be used here
 Second : take the remaining sets, look for patterns in the
items sets such that occurrence rate is above the minimum
confidence
 How many people bought hamburgers and buns together
 Ex: we find that if the customer is a male, and they buy
diapers, they will also buy beer
 {male, diapers} => {beer}
Matching data to functions
 Often, it is desirable to match data sets and the
factors that determine them to functions
 Allows for the possibility of predicting future results
 Involves learning how dependent and
independent variables in our data interact
 Dependent : the result, or where a point exists
 Independent : an cause or circumstance that
determines the dependent variable
 If we know how dependent and independent
variables interact, we can create a function and
run simulations to see results
Uses of Function-Data Mapping
 Weather Forecasting
 Determining what conditions lead to what kinds of
weather
 Stock market analysis
 When to buy and when to sell
 Crime Prevention
 What conditions cause or prevent crime
Categorizing
 Categorizing – Often we want to separate data
based off of a set of predefined attributes
 Very helpful in pattern recognition
 Ex: a persons political preference
 The process :
 we synthetically generate or measure a set of
observations (data points) with known categories
 we extract properties from said observations which
we believe contribute to the category
 These are called explanatory variables
 Finally we examine new data for these properties
Summarizing
 Summarizing – we almost never want to look at all of
the data individually
 Having too much data can actually hider the decision
making process
 Known as information overload
 Summarizing takes the results from data mining and
transforms it into formats that can be easily read
without omitting important information
 Summarizing might :
 Extract and display only important data
 correlate and abstract data to display trends
 Formats Include : Reports, Graphs, Dashboards, etc.
Standards : CRISP-DM
 Cross Industry Standard Process for Data Mining
 describes common practice for conducting data mining in an
enterprise setting
 KD nuggets – a community resource in DM and analytics
took polls and found CRISP-DM was the top methodology
in 02’, 04’, & 07’
 Six step methodology
 Business Understanding
 Data Understanding
 Data Preparation
 Modeling
 Evaluation
 Deployment
CRISP-DM : Explained
 Business Understanding
 Determining the business purpose
 Define success conditions – how do we know we succeeded
 Ex : improved prediction accuracy
 Map purpose/success conditions to data mining results
 Ex: fraud prevention => detect deviations
 Data Understanding
 Collecting and exploring data – defining its attributes
 Data quality verification
CRISP-DM : Explained
 Data Preparation
 Data Cleaning
 Normalization – fitting data within ranges
 Outlier removal – removing cases that could skew the model
 Handle missing attributes – the data was not obtained
 Formatting – changing data so that it fits with our tools
 Modeling – fitting the data to a model following the
methods previously described and then interpreting that
model
 Assess the accuracy of the collected data
 General purpose divided into prediction or description
CRISP-DM : Explained
 Evaluation – look at results and measure them with respect
to the success cases defined earlier
 Determine if one has succeeded
 Determine next steps, how do we apply the results
 Deployment – The execution of a strategy for using the
results of our data mining
 Includes preparing ways to monitor and maintain the
application of data mining results in the day to day
 Includes some sort of final summary
SEMMA
 Sample, Explore, Modify, Model and Assess
 Proposed by SAS Institute : A producer of BI and BA
software suites.
 Though this model is often considered general SAS
prefers to apply it directly to their products
 Focuses mainly on data mining and not on applying results
to business (unlike CRISP-DM)
Sampl
e
selecting the data set
Explor
e
Understand data through discovering relationships, both expected and
otherwise
Modify Transform and clean the data in order to prepare it for the modeling
process
Model Apply models to the data in order to discover trends and make predictions
Assess Evaluate the results of the modeling process to determine the reliability of
the mined data
Challenges in data mining
 Not enough or too much data
 Oftentimes it is difficult to access sufficient quantities of data
for small enterprises
 If the enterprise is large however, sometimes there is too
much and deciding what to keep is difficult
 Acquiring clean data
 Multiple formats or no format at all
 Privacy and ethical concerns
 Data aggregation : data compiled from multiple sources can
lead to revelations that violate privacy concerns
 Ex: anonymous data is collected and aggregated, leading to
identification

More Related Content

PPTX
Introduction to Business Data Analytics
PPTX
Data analytics
PPTX
Business analytics and data mining
PPT
PDF
Big Data Analytics
PPT
Data mining & data warehousing
ODP
Data mining
Introduction to Business Data Analytics
Data analytics
Business analytics and data mining
Big Data Analytics
Data mining & data warehousing
Data mining

What's hot (13)

PPTX
Data analytics
PDF
data analysis-mining
PPTX
Data analytics vs. Data analysis
PPTX
Classification of data
PDF
SAS/MIT/Sloan Data Analytics
PDF
Predictive modeling
PPTX
Data mining: Classification and prediction
PPTX
Introduction to data analytics
PDF
Introduction to data analytics
PPTX
Data mining
PPTX
Foundation of information system
PDF
Data mining
Data analytics
data analysis-mining
Data analytics vs. Data analysis
Classification of data
SAS/MIT/Sloan Data Analytics
Predictive modeling
Data mining: Classification and prediction
Introduction to data analytics
Introduction to data analytics
Data mining
Foundation of information system
Data mining
Ad

Viewers also liked (20)

PPT
Access data connection
PPTX
Crypto passport authentication
PPT
List in webpage
PDF
Bumper Mounting LED Light Bar ELM-5027W
PDF
Valentines Golden Ratio.pdf
PPTX
Data and assessment
PPT
Reflection
PPT
List and iterator
DOC
Ooad lab manual
PPTX
Constitución 1812
PPTX
Constitución 1812 erf
PPTX
Data structures and algorithms
PPTX
Exception handling
PPT
Text classification
PPTX
Python data structures
PPT
List in webpage
PPTX
Abstraction file
PPT
Poo java
PPTX
Key exchange in crypto
PPTX
Hardware managed cache
Access data connection
Crypto passport authentication
List in webpage
Bumper Mounting LED Light Bar ELM-5027W
Valentines Golden Ratio.pdf
Data and assessment
Reflection
List and iterator
Ooad lab manual
Constitución 1812
Constitución 1812 erf
Data structures and algorithms
Exception handling
Text classification
Python data structures
List in webpage
Abstraction file
Poo java
Key exchange in crypto
Hardware managed cache
Ad

Similar to Business analytics and data mining (20)

PPT
PPTX
Data mining an introduction
PDF
Chapter 1.pdf
PPT
Data mining final year project in jalandhar
PPT
Data mining final year project in ludhiana
PPT
Data mining
PPTX
Data warehousing and mining furc
PPT
6 weeks summer training in data mining,jalandhar
PPT
6 weeks summer training in data mining,ludhiana
PPT
6months industrial training in data mining, jalandhar
PPT
6months industrial training in data mining,ludhiana
PPTX
01 Introduction to Data Mining
PPTX
Data mining
PPT
data mining presentation power point for the study
PPT
lect1.ppt
PPT
Data Mining Course Overview Overview.ppt
PPT
lect1lect1lect1lect1lect1lect1lect1lect1.ppt
PPT
lghjghgggkgjhgjghhjgjhgkhjghjghjghjghect1.ppt
PPT
datamining.ppt
PPT
datamining.ppt
Data mining an introduction
Chapter 1.pdf
Data mining final year project in jalandhar
Data mining final year project in ludhiana
Data mining
Data warehousing and mining furc
6 weeks summer training in data mining,jalandhar
6 weeks summer training in data mining,ludhiana
6months industrial training in data mining, jalandhar
6months industrial training in data mining,ludhiana
01 Introduction to Data Mining
Data mining
data mining presentation power point for the study
lect1.ppt
Data Mining Course Overview Overview.ppt
lect1lect1lect1lect1lect1lect1lect1lect1.ppt
lghjghgggkgjhgjghhjgjhgkhjghjghjghjghect1.ppt
datamining.ppt
datamining.ppt

More from James Wong (20)

PPT
Data race
PPT
Multi threaded rtos
PPT
Recursion
PPTX
Data mining and knowledge discovery
PPTX
Cache recap
PPTX
Big picture of data mining
PPTX
How analysis services caching works
PPTX
Optimizing shared caches in chip multiprocessors
PPTX
Directory based cache coherence
PPT
Abstract data types
PPTX
Abstraction file
PPTX
Hardware managed cache
PPTX
Object model
PPT
Abstract class
PPTX
Object oriented analysis
PPTX
Concurrency with java
PPTX
Data structures and algorithms
PPTX
Cobol, lisp, and python
PPTX
Inheritance
PPTX
Api crash
Data race
Multi threaded rtos
Recursion
Data mining and knowledge discovery
Cache recap
Big picture of data mining
How analysis services caching works
Optimizing shared caches in chip multiprocessors
Directory based cache coherence
Abstract data types
Abstraction file
Hardware managed cache
Object model
Abstract class
Object oriented analysis
Concurrency with java
Data structures and algorithms
Cobol, lisp, and python
Inheritance
Api crash

Recently uploaded (20)

PDF
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
PDF
Assigned Numbers - 2025 - Bluetooth® Document
PPTX
TLE Review Electricity (Electricity).pptx
PPTX
A Presentation on Touch Screen Technology
PDF
From MVP to Full-Scale Product A Startup’s Software Journey.pdf
PDF
MIND Revenue Release Quarter 2 2025 Press Release
PDF
DASA ADMISSION 2024_FirstRound_FirstRank_LastRank.pdf
PDF
Hybrid model detection and classification of lung cancer
PDF
A comparative study of natural language inference in Swahili using monolingua...
PPTX
Tartificialntelligence_presentation.pptx
PDF
Mushroom cultivation and it's methods.pdf
PDF
A comparative analysis of optical character recognition models for extracting...
PDF
Encapsulation theory and applications.pdf
PDF
Hindi spoken digit analysis for native and non-native speakers
PDF
DP Operators-handbook-extract for the Mautical Institute
PDF
Zenith AI: Advanced Artificial Intelligence
PDF
Unlocking AI with Model Context Protocol (MCP)
PPTX
1. Introduction to Computer Programming.pptx
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
Agricultural_Statistics_at_a_Glance_2022_0.pdf
Assigned Numbers - 2025 - Bluetooth® Document
TLE Review Electricity (Electricity).pptx
A Presentation on Touch Screen Technology
From MVP to Full-Scale Product A Startup’s Software Journey.pdf
MIND Revenue Release Quarter 2 2025 Press Release
DASA ADMISSION 2024_FirstRound_FirstRank_LastRank.pdf
Hybrid model detection and classification of lung cancer
A comparative study of natural language inference in Swahili using monolingua...
Tartificialntelligence_presentation.pptx
Mushroom cultivation and it's methods.pdf
A comparative analysis of optical character recognition models for extracting...
Encapsulation theory and applications.pdf
Hindi spoken digit analysis for native and non-native speakers
DP Operators-handbook-extract for the Mautical Institute
Zenith AI: Advanced Artificial Intelligence
Unlocking AI with Model Context Protocol (MCP)
1. Introduction to Computer Programming.pptx

Business analytics and data mining

  • 1. Overview of BA Discussion  Business Analytics (BA)  Overview  History  Types of Business Analytics  Real world examples  Challenges  Relations to Data Mining
  • 2. Business Analytics (BA) : an overview  BA can be considered a subset of Business intelligence  A set of skills, technologies, applications and practices  exploration and investigation of past business performance to gain insight and drive business planning.  Like Business Intelligence, BA can focus either on the business as a whole or only on segments of it  Focuses on developing new insights and understanding of performance based on data and statistical methods
  • 3. BA : Short History  Analytics in business dates far before computing  Frederick Taylor, father of scientific management, 19th century  time management exercises used in industrial settings  Henry Ford : assembly line pacing used to improve output and business profitability  BA becomes widespread when computers were used in DSS systems in the 60’s  Evolved into ERP, data warehouses, etc.
  • 4. Types of Business Analytics  Reporting or Descriptive Analytics  Affinity grouping  Clustering  Modeling or Predictive analytics
  • 5. BA: Reporting  Based on the need to locate and distribute business insights and experiences  Often involves ETL procedures used alongside a data warehousing scheme  The data is then collected, quantified, and organized using reporting tools  Reporting, allows for information describing different views of an enterprise to come together one place  A user could query a production and marketing database to determine if production of a product could be moved closer to where a product is sold
  • 6. BA: Affinity grouping  A tool used by businesses and organizations to take ideas and data and organize them.  Often takes the form of an affinity diagram  Enables data and ideas stemming from brainstorming to be sorted into groups  Sorting is based on their natural relationships
  • 7. BA: Clustering  Placing a set of objects into groups (called clusters) so that the objects in the same cluster are more similar (in some sense or another) to each other than to those in other clusters – wikipedia  Is a main task of explorative data mining and statistical data analysis  Clustering is a general task that does not have one set solution  Clustering can be hard or fuzzy  Can be done by people or machines  The latter is preferred
  • 8. BA: how do we model clusters?  Connectivity models – how data can be connected to other points  Density models – defining a cluster by determining where sets of data points are densest  Distribution model – clusters are modeled using statistical distributions  Expectation maximization
  • 9. BA: Predictive Analysis  Stems from the desire to predict future events through analyzing data an enterprise has collected  Pattern exploitation results in the identification of opportunities and also risks  Allow relationships in disparate data to be identified  Helps guide in decision making in a business  Is often implemented in the form of data mining
  • 10. BA : Examples  Credit company– uses business analytics to track credit risk of customers as well as matching customers to offerings  Sales and offers – companies can track customer interaction, and use that information to determine appropriate product offerings.  Sales groups can use BA to optimize inventory and analyze past sales  Could measure peak purchasing times for products  Could decide whether or not to stock poorly selling items  Give examples of business cases where data mining might be useful, and describe how data mining would be used  Preventing credit card fraud through detecting spending patterns  Inventory management by tracking sales
  • 11. BA : Challenges  Acquiring sufficient volumes of high quality data  Most data acquired in the field is unsorted and appears in many different formats  When dealing with high volume data, deciding what is important and what is noise  Rapidly reacting storage structures  BA can influence customer interactions, and as such that information must be available fast  Ex: a customized sales pitch
  • 12. Business Analytics & Data Mining  Data Mining is an important sub task of Business Analytics  Both Predictive analysis and clustering tasks utilize information retrieved from data mining  Data mining helps handle some of the specific problems faced when conducting Business Analytics  Dealing with and sorting through large data sets
  • 13. Data Mining : An Overview  What is Data Mining ?  History  Applications of Data Mining  Detecting data discrepancies or outliers  Relationship identification  Data-Function mapping for modeling/prediction  Categorizing and Summarizing Data  Standards  Challenges
  • 14. Data Mining : What is it?  Applying statistical analysis techniques to data  the goal often being to determine unnoticed patterns or to collect categorized information  turns collected data into understandable structures  Data Mining is often used as a buzz word to describe processing large amounts of data  In essence, its correct use relates to discovery of new things through observation  Synonymous with knowledge discovery
  • 15. Data Mining : History  Though HNC trademarked the term in 1990, hands on pattern extraction is centuries old  As long as statistic analysis has existed  Discoveries in computer science have increasingly shifted the field from hands on to machine dependent, this allows for :  The use of data indexing and DB systems to handle data efficiently  The application of statistical algorithms on a large scale, possibly in a distributed manner, with less error
  • 16. Data Mining : Use : Application  Data Mining is often broken into several different categories of tasks  Detecting data discrepancies or outliers  Relationship identification  Data-Function mapping for modeling/prediction  Categorizing and Summarizing Data
  • 17. Data Mining : Finding outliers  The process of analyzing large, mostly homogeneous, sets of data and determining which sets or points  “go with the flow” and conform with patterns the rest of the data seem to follow  do not follow expected results when viewed against the entire set of data  An outlier can be a point or set of points, but can also be defined through other means  A period of time could yield unexpected results  Ex. Network Intrusion
  • 18. Data Mining : Techniques in finding outliers  Rule Based – deciding a set of rules that determine an outlier (or what isn’t one)  Can be fuzzy or hard rules  Cluster Analysis – As mentioned earlier  Distance or Standard Deviation – Determining an average over a data set and marking points that aren’t within a Deviation or Distance
  • 19. Applications of Outlier Detection  Network Intrusion Detection  Unusual bursts of network activity  Identity Theft Detection  Unusual spending or customer activity  Detecting Software bugs  Software does not deliver expected outputs  Sensor event detection  Monitoring patient health fluctuations in a medical setting  Preprocessing  Removing data skews based on extenuating circumstances
  • 20. Relationship Discovery: Basics  Understanding how data is related is a key factor in trend and knowledge discovery  This is the definition of data mining  Ex: Which products are often bought before a major forecasted storm  {hamburger buns} => {???}  With small sets of data, or with correlations that aren’t subtle (as the one above), identifying relationships is not as difficult  With large data sets or subtle relations a combination of rule generation and data analysis can be used to expedite the process
  • 21. Relationship Discovery: How its done  Since the number of relationships between points of data could be boundless, two important concepts are often introduced in relationship discovery:  The amount of data within which a relationship might exist, called the support of a rule.  The probability that data in the support will verify a selected rule, called the confidence of a rule.
  • 22. Relationship Discovery: How its done  Generally we apply minimum bounds to both the support of a rule and its confidence to determine relationships  First : determine possible relationships  Set a minimum support  Orders with hamburgers, Orders with hamburger buns  Other, user specific rules can be used here  Second : take the remaining sets, look for patterns in the items sets such that occurrence rate is above the minimum confidence  How many people bought hamburgers and buns together  Ex: we find that if the customer is a male, and they buy diapers, they will also buy beer  {male, diapers} => {beer}
  • 23. Matching data to functions  Often, it is desirable to match data sets and the factors that determine them to functions  Allows for the possibility of predicting future results  Involves learning how dependent and independent variables in our data interact  Dependent : the result, or where a point exists  Independent : an cause or circumstance that determines the dependent variable  If we know how dependent and independent variables interact, we can create a function and run simulations to see results
  • 24. Uses of Function-Data Mapping  Weather Forecasting  Determining what conditions lead to what kinds of weather  Stock market analysis  When to buy and when to sell  Crime Prevention  What conditions cause or prevent crime
  • 25. Categorizing  Categorizing – Often we want to separate data based off of a set of predefined attributes  Very helpful in pattern recognition  Ex: a persons political preference  The process :  we synthetically generate or measure a set of observations (data points) with known categories  we extract properties from said observations which we believe contribute to the category  These are called explanatory variables  Finally we examine new data for these properties
  • 26. Summarizing  Summarizing – we almost never want to look at all of the data individually  Having too much data can actually hider the decision making process  Known as information overload  Summarizing takes the results from data mining and transforms it into formats that can be easily read without omitting important information  Summarizing might :  Extract and display only important data  correlate and abstract data to display trends  Formats Include : Reports, Graphs, Dashboards, etc.
  • 27. Standards : CRISP-DM  Cross Industry Standard Process for Data Mining  describes common practice for conducting data mining in an enterprise setting  KD nuggets – a community resource in DM and analytics took polls and found CRISP-DM was the top methodology in 02’, 04’, & 07’  Six step methodology  Business Understanding  Data Understanding  Data Preparation  Modeling  Evaluation  Deployment
  • 28. CRISP-DM : Explained  Business Understanding  Determining the business purpose  Define success conditions – how do we know we succeeded  Ex : improved prediction accuracy  Map purpose/success conditions to data mining results  Ex: fraud prevention => detect deviations  Data Understanding  Collecting and exploring data – defining its attributes  Data quality verification
  • 29. CRISP-DM : Explained  Data Preparation  Data Cleaning  Normalization – fitting data within ranges  Outlier removal – removing cases that could skew the model  Handle missing attributes – the data was not obtained  Formatting – changing data so that it fits with our tools  Modeling – fitting the data to a model following the methods previously described and then interpreting that model  Assess the accuracy of the collected data  General purpose divided into prediction or description
  • 30. CRISP-DM : Explained  Evaluation – look at results and measure them with respect to the success cases defined earlier  Determine if one has succeeded  Determine next steps, how do we apply the results  Deployment – The execution of a strategy for using the results of our data mining  Includes preparing ways to monitor and maintain the application of data mining results in the day to day  Includes some sort of final summary
  • 31. SEMMA  Sample, Explore, Modify, Model and Assess  Proposed by SAS Institute : A producer of BI and BA software suites.  Though this model is often considered general SAS prefers to apply it directly to their products  Focuses mainly on data mining and not on applying results to business (unlike CRISP-DM) Sampl e selecting the data set Explor e Understand data through discovering relationships, both expected and otherwise Modify Transform and clean the data in order to prepare it for the modeling process Model Apply models to the data in order to discover trends and make predictions Assess Evaluate the results of the modeling process to determine the reliability of the mined data
  • 32. Challenges in data mining  Not enough or too much data  Oftentimes it is difficult to access sufficient quantities of data for small enterprises  If the enterprise is large however, sometimes there is too much and deciding what to keep is difficult  Acquiring clean data  Multiple formats or no format at all  Privacy and ethical concerns  Data aggregation : data compiled from multiple sources can lead to revelations that violate privacy concerns  Ex: anonymous data is collected and aggregated, leading to identification

Editor's Notes

  • #4: Taylor : mechanical engineer who focused on improving industrial efficiency DSS – Decision Support Systems, ERP – Enterprise Resource Planning
  • #5: 4:40
  • #8: Fuzzy clustering – each object has a likeliness of belonging to a cluster
  • #9: Expected max - multivariate normal distributions - One can simply pick arbitrary values for one of the two sets of unknowns, use them to estimate the second set, then use these new values to find a better estimate of the first set, and then keep alternating between the two until the resulting values both converge to fixed points
  • #13: 17:20
  • #21: Agrawal, R.; Imieliński, T.; Swami, A. (1993). "Mining association rules between sets of items in large databases". Proceedings of the 1993 ACM SIGMOD international conference on Management of data - SIGMOD '93. pp. 207. doi:10.1145/170035.170072.ISBN 0897915925.  http://en.wikipedia.org/wiki/Association_rule_learning#Useful_Concepts
  • #23: Agrawal - Agrawal, R.; Imieliński, T.; Swami, A. (1993). "Mining association rules between sets of items in large databases". Proceedings of the 1993 ACM SIGMOD international conference on Management of data - SIGMOD '93. pp. 207 30 min
  • #24: http://en.wikipedia.org/wiki/Regression_analysis
  • #26: http://en.wikipedia.org/wiki/Statistical_classification
  • #27: http://en.wikipedia.org/wiki/Information_overload
  • #28: http://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining http://www.kdnuggets.com/polls/2002/methodology.htm http://www.kdnuggets.com/polls/2004/data_mining_methodology.htm http://www.kdnuggets.com/polls/2007/data_mining_methodology.htm
  • #29: http://dms.irb.hr/tutorial/tut_prob_understand.php http://dms.irb.hr/tutorial/tut_data_understand.php
  • #30: http://dms.irb.hr/tutorial/tut_data_prepare.php http://dms.irb.hr/tutorial/tut_modelling.php
  • #31: http://dms.irb.hr/tutorial/tut_evaluation.php http://dms.irb.hr/tutorial/tut_deployment.php
  • #32: http://www.sas.com/offices/europe/uk/technologies/analytics/datamining/miner/semma.html
  • #33: Data mining - wikipedia