Adventures in Data Profiling

Jim Harris
Blogger‐in‐Chief
www.ocdqblog.com
Jim Harris
Digitally signed by Jim Harris
DN: cn=Jim Harris, o=Obsessive-Compulsive Data Quality (OCDQ), ou, email=jim.harris@ocdqblog.
com, c=US
Date: 2010.03.04 10:55:20 -06'00'

Jim Harris
Blogger‐in‐Chief
www.ocdqblog.com

E‐mail
jim.harris@ocdqblog.com

Twitter
twitter.com/ocdqblog

LinkedIn
linkedin.com/in/jimharris

Adventures in Data Profiling Copyright © 2010, Jim Harris. All rights reserved. 2

Let the Adventures Begin . . .
This will be a vendor‐neutral presentation:

Focusing on general methodology of data profiling
and common functionality of data profiling tools

Discussing how a data profiling tool helps automate
some of the work needed for preliminary data analysis

Reviewing fictional data and results produced by a
fictional data profiling tool to illustrate basic concepts


Understanding Your Data
Understanding your data is essential to using it
effectively and improving its quality

You need a reality check for the perceptions and
assumptions you have about the quality of your data

You need to prepare meaningful questions to ask your
business analysts and subject matter experts

There is simply no substitute for data analysis


Profiling Your Data
Data profiling includes many types of analysis such as:

Verify data matches the metadata that describes it

Identify representations for the absence of data

Identify potential default and invalid values

Check data formats for inconsistencies

Assess domain, structural, and relational integrity


Getting Your Data Freq On
Data profiling tools can help you by automating some
of the grunt work needed to begin your data analysis

One of their basic features is the ability to generate
statistical summaries and frequency distributions for
the unique values and formats found within your fields

Therefore, I like to refer to using a data profiling tool as:
“Getting Your Data Freq On”


Let Me Count The Ways

NULL – record count of NULL values Cardinality – count of the number of
distinct actual values
Missing – record count of Missing values
(i.e., non‐NULL absence of data) Uniqueness – percentage calculated as
Cardinality divided by total record count
Actual – record count of Actual values
(i.e., non‐NULL and non‐Missing) Distinctness – percentage calculated as
Cardinality divided by Actual
Completeness – percentage calculated as
Actual divided by total record count

You Uniquely Complete Me
Completeness and
Uniqueness are useful in
evaluating potential key
fields and especially a
single primary key,
which should be both:
100% Complete
100% Unique


It’s a Distinct Possibility

Distinctness can be useful
in evaluating the potential
for duplicate records
< 100% Distinct means some
distinct actual values occur on
more than one record

Gimme the lo down, Drill‐down


Freq’ing Distribution of Values

Frequency distribution of
values is useful for fields
with a low cardinality
Extremely low cardinality
might be an indication of
default values


Reviewing the Top N List

Reviewing the Top N most
frequently occurring values

Freq’ing Distribution of Formats

Frequency distribution of formats is useful for fields having
both a high cardinality and free‐form values

Unlocking the Combination

Combination of values
and formats can help with
unlocking the mystery of
more complex fields


. . . the Adventures Conclude
What can just your analysis of data tell you about it?

Understand your data better by first looking at it from a
starting point of blissful ignorance and curiosity

A tool can help automate some of the grunt work, but
the actual data analysis can not be automated

Your analytical goal is not to try to find answers, but to
discover the right questions


Adventures in Data Profiling

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Adventures in Data Profiling (20)

Recently uploaded (20)

Adventures in Data Profiling