Data Preparation Fundamentals

Data
Preparation
Fundamentals
© Copyright 2022 by Peter Aiken Slide # 1
peter.aiken@anythingawesome.com +1.804.382.5957 Peter Aiken, PhD
0%
50%
100%
Data Analysis Data Preparation
0%
0% ?
Peter Aiken, Ph.D.
• I've been doing this a long time
• My work is recognized as useful
• Associate Professor of IS (vcu.edu)
• Institute for Defense Analyses (ida.org)
• DAMA International (dama.org)
• MIT CDO Society (iscdo.org)
• Anything Awesome (anythingawesome.com)
• Experienced w/ 500+ data
management practices worldwide
• Multi-year immersions
– US DoD (DISA/Army/Marines/DLA)
– Nokia
– Deutsche Bank
– Wells Fargo
– Walmart
– HUD …
• 12 books and
dozens of articles
https://anythingawesome.com
+
• DAMA International President 2009-2013/2018/2020
• DAMA International Achievement Award 2001
(with Dr. E. F. "Ted" Codd
• DAMA International Community Award 2005

Your
Sponsor
Today
Quest
Software
Solutions Consultant
Information Management Specialist
Quest Software
Gary Jerep
Where Next Meets Now.

Foglight
SharePlex
Kace
Toad
QoreStor
NetVault
erwin
Binary Tree
Change Auditor
Migration Manager
Metalogix
Quadrotech
SCALE & STREAMLINE
IT OPERATIONS
Migrate faster, strengthen
cyber security resilience
and stay in control to keep your
business running
Identity Manager
Safeguard
Active Roles
Quest: Helping Customers Achieve True IT Resilience NOW
IDENTITY
DATA EMPOWERMENT
& GOVERNANCE
Empower your business with the
visibility and context to better
manage and develop data pipelines
that deliver faster insights, while
safeguarding the data
and infrastructure
Information
& System
Management
One
Identity
Microsoft
Platform
Management
HARDENED CYBER
SECURITY
Using an identity-focused,
cloud-first, customer-centric approach
from the Cloud to the edge
making Zero Trust a reality now

Monitoring
Sensitive Data
Identification
SQL Tuning &
Optimization
Development
Administration
Backup
DevOps
Upgrades
Migration
Load Balancing
Diagnostics
Data Modeling
Systems Monitoring &
Diagnostics
Inventory &
Asset Management
Policy &
Access Management
Secondary Storage &
Cost Optimization
Backup & Recovery
Software Compliance
Helpdesk
Cloud Cost Optimization
erwin Data
Modeler
erwin Data
Catalog
erwin
Evolve
erwin Data
Literacy
erwin Data
Intelligence
Active Metadata
Management
Business Process
Modeling
Data
Stewardship
Regulatory
Compliance
Data Catalog
Data Lineage
Enterprise
Architecture
Data Architecture
Data Profiling
Impact Analysis
RapidRecovery®
Data
Protection
Data
Operations
Data
Governance Application
Modernization
Cloud Migration
Empower NoSQL
Sensitive Data Protection
Model-Driven
DevOps
SLA Performance
Cyber
Resilience
The Quest Data Empowerment Platform

Where Next Meets Now.
®
A Few Resources!
• www.Quest.com/Solutions/Data-
empowerment
• eBook: The Four Roadblocks to Data
Preparation
• Toad Data Point Case Study:
Philadelphia Youth Network
• eBook: Enabling Agile Database
Development with Toad
• Tech Brief: Keep Using Toad for Oracle
with Databases in the Cloud
• Tech Brief: Accelerate and Secure your
SQL Server DevOps CI/CD Pipelines

Current approaches are not and have not been working
Driving Innovation with Data
Competing on data and analytics
Managing data as a business asset
Created a data-driven organization
Forged a data culture
25% 50% 75% 100%
24%
24%
39%
41%
56%
Yes No
Source: Big Data and AI Executive Survey 2022 by Randy Bean & Thomas Davenport @ www.newvantage.com
2020
0%
25%
50%
75%
100%
technology people/process
90%
10%
80% of data challenges are people/process based!
0%
25%
50%
75%
100%
20%
80%
Everyone wants to do better data analysis …
• Some data preparation is inevitable
• What would a 'good' ratio be?
• "Everyone knows"

1. 80% of your data is redundant, trivial, or obsolete
2. 80% of your data is of unknown quality
3. 80% of your data is 'standards free'
4. Your highly paid data analytics capabilities spend
80% of their time working under these conditions
Pareto Data Realities
• IT thinks data is a business problem
– "If they can connect to the server, then my job is done!"
• The business thinks IT is managing data adequately
– "Who else would be taking care of it?"
Confusion as to data responsibility

You must address data debt proactively
https://www.merkleinc.com/blog/are-you-buried-alive-data-debt
https://johnladley.com/a-bit-more-on-data-debt/
https://uk.nttdataservices.com/en/blog/2020/february/how-to-get-rid-of-your-data-debt
Data debt:
• Slows progress
• Decreases quality
• Increases costs
• Presents greater risks
• Data debt
– The time and effort it will take to return your
shared data to a governed state from its
(likely) current state of ungoverned
• Getting back to zero
– Involves undoing existing stuff
– Likely new skills are required
Bad Data Decisions Spiral
Bad data decisions
Poor organizational outcomes
Technical decision
makers are not data
knowledgable
Business decision
makers are not data
knowledgable
Poor treatment
of organizational
data assets
Poor
quality
data

© Copyright 2022 by Peter Aiken Slide #
Program
Program
9
• Motivation
• Data Preparation Considerations
– No standard data curricula
– No standard audience
– Technology is a one-legged stool
• Data Problems are Different
– Dependence on high speed automation
– Hidden data factories sap resources
– Require a unified approach
• Reverse Engineering (Introducing Yourself to a Dataset)
– No measures (other than size)
– Hype Cycle
– Column Profiling
– Dependency Profiling
– Redundancy Profiling
• Take Aways/References/Q&A
0%
50%
100%
80%
20%
Data is not broadly or widely understood
adapted from: http://www.dailymirror.lk/print/opinion/editorial-we-need-to-become-channels-of-peace/172-27164
It is like a fan!
It is like a snake!
It is like a wall!
It is like a rope!
It is like a tree! Blind Persons and the Elephant
It is like a story!
It is like a dashboard!
It is like pipes!
It is like a warehouse!
It is like statistics!

Unrefined
data management
definition
Sources
Uses
Data Management
More refined
data management
definition
Sources
Uses
Reuse
Data Management
➜
➜

Governance & Ethical Use Framework
Specialized
Data
Skills
Collection
Evaluation
Engineering
Evolution
Access
Storage
Preparation
Data
Science
Delivery
Presentation
Story
Telling
Exploitation
Better still data management definition
Sources
➜
Reuse
➜
Data Preparation 80% Data Exploitation/Analysis 20%
Formal Data Reuse Management
Data
Technologies by themselves, are a 1–Legged Stool

Success Requires a 3–Legged Stool
P
e
o
p
l
e
Process
T
e
c
h
n
o
l
o
g
y
But not just a stool–these are interdependent
People
Process
Technology

Success is identifying a winning combination
People
Process
Technology
Defining Data Technology Architecture
• Data technology is part of the overall technology architecture
• It is also often considered part of the enterprise’s data architecture
• Data technology architecture addresses 3 questions:
– What technologies are
standard/required/preferred/acceptable?
– Which technologies apply to which
purposes and circumstances?
– In a distributed environment, which
technologies exist where, and
how does data move from one node to another?
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International

• Managing data technology should follow
the same principles and standards for
managing any technology
• Leading reference model for technology
management is the Information Technology
Infrastructure Library (ITIL):
– http://www.itil-officialsite.com/home/home.asp
Data Management Technologies
Understanding Data Technology Requirements
• Need to understand:
– How the technology works
– How it provides value in the
context of your organization
– Requirements of a data technology before
determining what technical solution to choose for a particular situation
• Suggested questions:
– What problem does this data technology mean to solve?
– What sets this data technology apart from others?
– Are there specific hardware/software/operating systems/storage/network/
connectivity requirements?
– Does this technology include data security functionality?

Program
Program
21
• Motivation
– Hype Cycle
0%
50%
100%
80%
20%
Data
Sandwich

Data supply
Data literacy
Standard data
Standard data
Leverage point - high performance automation
Data literacy
23
Data supply
Standard data
Data supply
Data literacy
24

This cannot happen without investments in
engineering and architecture!
25
Data supply
Data literacy
Standard data
Quality engineering/
architecture work products
do not happen accidentally!
Quality data engineering/
architecture work products
do not happen accidentally!
This cannot happen without investments in
data engineering and architecture!
26
Data supply
Data literacy
Standard data

Tacoma Narrows Bridge/Gallopin' Gertie
• Slender, elegant and graceful
• World's 3rd longest suspension span
• Opened on July 1st 1940
• Collapsed in a windstorm on November 7,1940
• "The most dramatic failure in
bridge engineering history"
• Changed forever how engineers
design suspension bridges leading
to safer spans today
Hidden Data Factories
https://hbr.org/2016/09/bad-data-costs-the-u-s-3-trillion-per-year
Work products are
delivered to
Customers
Customers
Knowledge Workers
80% looking for stuff
20% doing useful work
Department B
1. Check A's work
2. Make any corrections
3. Complete B's work
4. Deliver to Department C
Department A
https://en.wikipedia.org/wiki/Theory_of_constraints
Department C
1. Check B's work
2. Make any corrections
3. Complete C's work
4. Deliver to Customer
5. Deal with consequences

Hidden Data Factories
Poor data manifests as multifaceted organizational challenges

Poor data manifests as multifaceted organizational challenges
IT
System
Business
Challenge
Business
Process
Business
Challenge
IT
Process
Business
Challenge
Business
System
Business
Challenge
IT
Process
Business
Challenge
IT
System
Business
Challenge
Business
Process
Business
Challenge
Poor results
Root cause
analysis is
part of data
governance
Consistency Encourages Quality Analysis
IT
System
Business
Challenge
Business
Process
Business
Challenge
IT
Process
Business
Challenge
Business
System
Business
Challenge
IT
Process
Business
Challenge
IT
System
Business
Challenge
Business
Process
Business
Challenge
Eliminating data debt
requires a team with
specialized skills
deployed to create a
repeatable process
and develop sustained
organizational
skillsets

Program
Program
33
• Motivation
– Hype Cycle
0%
50%
100%
80%
20%
As Is Requirements
Assets WHAT?
As Is Design Assets
HOW?
As Is Implementation
Assets AS BUILT
Forward Engineering

As Is Requirements
Assets WHAT?
As Is Design Assets
HOW?
Assets AS BUILT
Existing
Reverse Engineering
A structured technique aimed at recovering rigorous
knowledge of the existing system to leverage
enhancement efforts [Chikofsky & Cross 1990]
As Is Requirements
Assets WHAT?
As Is Design Assets
HOW?
Assets AS BUILT
Existing
New
Reengineering
Reverse Engineering
Forward engineering
Reimplement
To Be
Implementation
Assets
To Be
Design
Assets
To Be Requirements
Assets
• First, reverse engineering the existing system
to understand its strengths/weaknesses
• Next, use this information to inform the
design of the new system

Data Preparation Tools & Vendor Hype
• CIOs/CDOs feel pressure
• Vendor/project promise auditing
• No understanding of hype cycle
Who wrote this … ?
• In considering any new subject,
• there is frequently a tendency
first to overrate what we find to
be already interesting or
remarkable, and
• secondly - by a sort of natural
reaction - to undervalue the true
state of the case.
– Lady Augusta Ada King,
(1815 – 1852)
Countess of Lovelace
– (aka) Ada Lovelace,
daughter of Lord Byron
– Publisher of the first
computing program

Technology Trigger: A potential technology breakthrough kicks things off. Early proof-of-concept stories and media interest
trigger significant publicity. Often no usable products exist and commercial viability is unproven.
Trough of Disillusionment: Interest wanes as experiments and implementations fail to deliver. Producers of the
technology shake out or fail. Investments continue only if the surviving providers improve their products to the
satisfaction of early adopters.
Peak of Inflated Expectations: Early publicity produces a number of
success stories—often accompanied by scores of failures. Some
companies take action; many do not.
Slope of Enlightenment: More instances of how the technology can benefit the
enterprise start to crystallize and become more widely understood. Second- and third-
generation products appear from technology providers. More enterprises fund pilots;
conservative companies remain cautious.
Plateau of Productivity: Mainstream adoption starts to
take off. Criteria for assessing provider viability are more
clearly defined. The technology’s broad market
applicability and relevance are clearly paying off.
Gartner Five-phase Hype Cycle
http://www.gartner.com/technology/research/methodologies/hype-cycle.jsp
Gartner 2021 Hype Cycle for Data Management

Metadata
Management
41
Data
Management
Body
of
Knowledge
(DM
BoK
V2)
Practice
Areas
from The DAMA Guide to the Data Management Body of Knowledge 2E © 2017 by DAMA International
Data Management Tools
1. Data Governance
– Intranet Website
– E-Mail
– Metadata Tools
– Issue Mgt. Tools
2. Data Architecture
– Management
– E-Mail
– Meta-data Tools
– Data Modeling Tools
– Model Mgt. Tool
– Metadata Repository
– Office Productivity
Tools
3. Data Development
– Cloud
– DBMS
– Software Dev. Tools
– Testing Tools
– CASE/Model Tools
– Config. Mgt. Tools
– Office Tools
4. Data Ops Mgt.
– DBMS
– Data Dev. Tools
– DBA Tools
– ETL Tools
5. Data Security Mgt.
– DBMS
– BI Tools
– Application
Frameworks
– Identity Mgt. Tech.
– Change Control Sys.
6. Reference and
Master Data
– Management
– Reference DM Apps.
– Master DM Apps
– Process Modeling
– Metadata Repositories
– Data Profiling Tools
– Data Cleansing Tools
– Data Integration Tools
– Rule Engines
– Change Mgt. Tools
7. Analytics
– Business Execs/Mgrs.
– DM Execs/IT Mgt.
– BI Program Manager
– SMEs/Info
Consumers
– Data Stewards
– Project Managers
– Data Architects/
Analysts
– Data Integration
Specs.
– BI Specialists
– DBAs
– Data Security
– Data Quality Analysts
8. Content
Management
– All Employees
– Data Stewards
– DM Professionals
– Records Mgt. Staff
– Other IT
Professionals
– Data Mgt. Executive
– CIO/CKO
9. Metadata Mgt.
– Metadata
Repositories
– DBMS
– BI Tools
– System Mgt. Tools
– Object Modeling Tools
Tools
– Report Generators
– Data Quality Tools
– Data Dev. Tools
– Reference/Master
Data Tools
10. Data Quality Tools
– Statistical Analysis
Tools
– Issue and Event
Management Tools
1. Data Governance
– E-Mail
– Metadata Tools
2. Data Architecture
– Management
– E-Mail
– Meta-data Tools
– Model Mgt. Tool
– Metadata Repository
Tools
3. Data Development
– Cloud
– DBMS
– Software Dev. Tools
– Testing Tools
– CASE/Model Tools
– Config. Mgt. Tools
– Office Tools
4. Data Ops Mgt.
– DBMS
– Data Dev. Tools
– DBA Tools
– ETL Tools
5. Data Security Mgt.
– DBMS
– BI Tools
– Application
Frameworks
– Identity Mgt. Tech.
– Change Control Sys.
6. Reference and
Master Data
– Management
– Reference DM Apps.
– Master DM Apps
– Metadata Repositories
– Rule Engines
– Change Mgt. Tools
7. Analytics
– Business Execs/Mgrs.
– DM Execs/IT Mgt.
– BI Program Manager
– SMEs/Info
Consumers
– Data Stewards
– Project Managers
– Data Architects/
Analysts
– Data Integration
Specs.
– BI Specialists
– DBAs
– Data Security
– Data Quality Analysts
8. Content
Management
– All Employees
– Data Stewards
– DM Professionals
– Records Mgt. Staff
– Other IT
Professionals
– Data Mgt. Executive
– CIO/CKO
9. Metadata Mgt.
– Metadata
Repositories
– DBMS
– BI Tools
– System Mgt. Tools
– Object Modeling Tools
Tools
– Report Generators
– Data Quality Tools
– Data Dev. Tools
– Reference/Master
Data Tools
10. Data Quality Tools
– Statistical Analysis
Tools
– Issue and Event
Management Tools

Data Management Tools
• Cloud
• CASE/Model Tools
• ETL Tools
• Data Quality Tools
• Data Profiling Tools
Gartner Strategic Planning Assumptions
• By 2023
– 75% of all databases will be cloud, reducing the DBMS vendor landscape and
increasing complexity for data governance and integration.
https://www.gartner.com/document/3894971?ref=solrAll&refval=219836558&qid=de595a5685b6f86db0ec6

Transform
Problems with forklifting
1. no basis for decisions made
2. no inclusion of architecture/
engineering concepts
3. no idea that these concepts
are missing from the process
4. 80% of organizational
data is ROT
Cleaner
More shareable
Less …
… data
Successful Cloud Adoption
Gartner Cloud Vendor Offerings
Data combined with easy access, becomes a key differentiator -
obvious combinations include:
• Google:
– Google Search data
– YouTube data
– Google Ads data
– Retailers.
• Azure
– LinkedIn
– Office 365 data
– Sales and customer-relationship-focused analytics
• Amazon
– Anything retail
https://www.gartner.com/document/3894971?ref=solrAll&refval=219836558&qid=de595a5685b6f86db0ec6

• Computer-aided software engineering (CASE) is the
scientific application of a set of tools and methods to a
software system which is meant to result in high-quality,
defect-free, and maintainable software products. It also
refers to methods for the development of information
systems together with automated tools that can be used in
the software development process.
• Scientific application of a set of tools and methods to a
software system which is meant to result in high-quality,
defect free, and maintainable software products
• Refers to methods for the development of information
systems together with automated tools that can be used in
the software development process
• CASE functions include analysis, design, and
programming
Source: http://en.wikipedia.org/wiki/
Computer Aided Software/Systems Engineering (CASE) Tools
CASE-based Support
http://www.visible.com

CASE-based Support
CASE-based Support

This includes:
• Senders
– flows from the CASE effort that
can inform the re-architecting
effort.
• Receivers
– flows from the project that can
inform the CASE effort.
• Senders and receivers
– some elements, such as
restructuring and reengineering,
are both senders and receivers.
CASE Tool: "Taxonomy"
A variety of
CASE-based
methods and
technologies can
access and
update the
metadata
metadata
Integration
Additional metadata uses
accessible via: web; portal;
XML; RDBMS
Everything must "fit" into one
CASE technology
Changing Model of CASE Tool Usage
Limited access
from outside
the CASE
technology
environment
CASE
tool-specific
methods
and
technologies
Limited additional
metadata use

IBM's AD/Cycle Information Model
Implementing Metadata
Repository Functionality
• "The repository" does not have to be an integrated solution
– it must be an easily integrateable solution
• Repository functionality (does not equal a) repository
– metadata must easily evolve to repository solution
• Multiple repositories are not necessarily bad
– as interim solutions, Excel has been working quite well
• Minimal functionality includes
– ability to create, read, update, delete, and evolve metadata items
• Remember the 1st law of data management
– In order to manage metadata, you need metadata repository functionality

Defining The "E" Spaces
• ETL Extract Transform, Load
– delivers aggregated data to a
new database
• EAI Enterprise Application Integration
– connects applications to other applications in a
predictable manner using
pre-established connections
• EII Enterprise Information Integration
– between ETL and EAI - delivers tailored views of
information to users at the time that it is required
Data Quality Engineering Tools
• 4 categories of activities:
– Analysis
– Cleansing
– Enhancement
– Monitoring
• Principal tools:
– Parsing and Standardization
– Data Transformation
– Identity Resolution and Matching
– Enhancement
– Reporting
– Data Profiling

Data Preparation Activities
Data Preparation Parsing & Standardization Tools
• Data parsing tools enable the
definition of patterns that feed into a
rules engine used to distinguish
between valid and invalid data
values
• Actions are triggered upon matching
a specific pattern
• When an invalid pattern is
recognized, the application may
attempt to transform the invalid value
into one that meets expectations
https://www.youtube.com/watch?v=r9UhJxFT5rk

Data Preparation Identity Resolution & MatchingTools
Basic approaches to matching:
• Deterministic
– Relies on defined patterns and
rules for assigning weights and
scores to determine similarity
• Predictable
– Only as good as anticipations of
the rules developers
• Probabilistic
– Uses statistical techniques to
assess probabilities that pairs of
records represent the same
entity
• Not reliant on rules
– Refined based on experience ->
matchers can improve precision
as more data is analyzed
Data Preparation Enhancement Tools
• A method for adding value to information by accumulating
additional information about a base set of entities and then
merging all the sets of information to provide a focused view
• Examples:
– Time/date stamps
– Auditing information
– Contextual information
– Geographic information
– Demographic information
– Psychographic information

Data Preparation Reporting Tools
• Good reporting supports:
– Inspection and monitoring of conformance to data quality expectations
– Monitoring performance of data stewards conforming to data quality SLAs
– Workflow processing for data quality incidents
– Manual oversight of data cleansing and correction
• Associate report results w/:
– Data quality measurement
– Metrics
– Activity
Data Preparation Portals as a Tools

Data Preparation Profiling Tools
• Need to be able to distinguish between good and bad data before
making any improvements
• Data profiling is a set of algorithms for 2 purposes:
– Statistical analysis and assessment of the data quality values within a data set
– Exploring relationships that exist between value collections within and across
data sets
Sample Existing Environment
?
64

VSAM, DB2, Oracle, Client
Server, IDMS, Flat files
65
RDBMS 1
Finance
HR
RDBMS 2
Marketing
R
&
D
#
1
R
&
D
#
2
R&D
#3 Network
Database
BackOffice
Applications
Manufacturing
Systems Flat Files
Logistics
Systems Flat Files
66

Technology
Logical/
Virtual
Databases
Subject
Areas Tables Attributes
Unique
Attributes Records
Programs/
Copybooks
Lines of
Code
Data Management Technology Type 1 44 20 13,067 6,600
Global schema 1,049
T01 5
C01 4
Data Management Technology Type 2
R2 613
R3 108
R5 127
R7 447
R72 5,996
R73 11,224
Data Management Technology Type 3 1,227 9,000 2,514
Application 10,970 15,700,000
Copybooks 5,518 498,966
Totals: 44 20 19,742 22,067 9,114 1,058 16,488 16,198,966
67
Profiling
Discovery
Analysis
Data Discovery Technologies
• Data analysis software technologies deliver up to 10X
productivity over manual approaches
• Based on a powerful computing technology that allows data
engineers to quickly form candidate hypotheses with respect
to the existing data structures
• Hypotheses are then presented to the SMEs (both business
and technical) who confirm, refine, or deny them
• Allows existing data structures to be inferred at rate that is an
order of magnitude more effective than previous manual
approaches
• Semi-automated
68

How has this been done in the past?
• Old
– Manually
– Brute force
– Repository dependent
– Quality indifferent
– Not repeatable
• New
– Semi-automated
– Engineered
– Repository independent
– Integrated quality
– Repeatable
– Currency
– Accuracy
Semi-
automating
Reverse
Engineering:
Column
Profiling with
Attribute
Summary
Report

Semi-automating Reverse Engineering:
Column Profiling, Compare Documented vs. Actual
Screen shots of Migration Architect used by permission of Evoke Software http://www.evokesoft.com
72
Semi-automating Reverse
Engineering:
Column Profiling, Drilling
Down on Column Values

Select an Attribute to get a list of values
Double-click a value to
see rows with that value
Column
Profiling
Demonstration
74

Dependency Profiling, Candidate Dependencies
Dependency Profiling, Promoting Dependencies

Dependency
Profiling
Demonstration
77
Semi-automating
Reverse Engineering:
Redundancy
Profiling, Domain
Comparison Detail

Redundancy
Profiling
Demonstration
Comparing Weekly Progress (80/20)
MONDAY
Morning:
Model
preparation
Afternoon:
Model refinement/
validation session
TUESDAY
Afternoon:
Model refinement/
validation session
WEDNESDAY
Afternoon:
Model refinement/
validation session
THURSDAY
Afternoon:
Model refinement/
validation session
FRIDAY
Afternoon:
Model refinement/
validation session
MONDAY
Morning:
Model
preparation
Afternoon:
Model
preparation
TUESDAY
Morning:
Model
preparation
Afternoon:
Model refinement/
validation session
WEDNESDAY
Morning:
Model
preparation
Afternoon:
Model
preparation
THURSDAY
Morning:
Model
preparation
Afternoon:
Model refinement/
validation session
FRIDAY
Morning:
Model
preparation
Afternoon:
Model
preparation
80
Morning:
Model refinement/
validation session
Morning:
Model
preparation
Morning:
Model refinement/
validation session
Morning:
Model
preparation

Baseline
Relative
Condition
&
Amount
of
Evidence
[ ]
Confounding
characteristics
Data Handling,
Operating
Environment
&
Language
Factor
(Factor => 1)
[ ]
[
Beneficial
characteristics
Key End User
Participation &
Net Automation
Impact
(Impact =<1)
]
Historical
organizational
reverse
engineering
performance data
[ ]
= Project
characteristics
* The purpose of the Preliminary System Survey is to determine how long and how
many resources will be required to reverse engineer the selected system components.
[ ]
Project
characteristics
=
Project
Estimate
Preliminary System Survey (PSS*)
Ancillary Results
• 4.7 billion empty bytes in just three
data warehouse tables
– Reducing the need to upgrade
company infrastructure capacity
– Made a strong case for normalization
– Ovation for the team from the Data
Warehouse Board of Directors
• Preserved multi-million dollar US
Postal Service (and other) sort
discounts
• Accurate measurable views of how
effectively certain processes work
• Hundreds of files have processed
by the domain profiling process.
• Many of the files through initial
stages of normalization

Program
Program
83
• Motivation
– Hype Cycle
0%
50%
100%
80%
20%
Supply/demand for data talent
https://www.logianalytics.com/bi-trends/3-keys-understanding-data/
Growth of Data vs. Growth of Data Analysts
• Stored data accumulating at
28% annual growth rate
• Data analysts in workforce
growing at 5.7% growth rate

Unmatched
Items
Ignorable
Items
Items
Matched
Week # (% Total) (% Total) (% Total)
1 31.47% 1.34% N/A
2 21.22% 6.97% N/A
3 20.66% 7.49% N/A
4 32.48% 11.99% 55.53%
… … … …
14 9.02% 22.62% 68.36%
15 9.06% 22.62% 68.33%
16 9.53% 22.62% 67.85%
17 9.5% 22.62% 67.88%
18 7.46% 22.62% 69.92%
Determining Diminishing Returns
Before
After
85
Quantifying Benefits: Original Plan
Time needed to review all NSNs once over the life of the project:
NSNs 2,000,000
Average time to review & cleanse (in minutes) 5
Total Time (in minutes) 10,000,000
Time available per resource over a one year period of time:
Work weeks in a year 48
Work days in a week 5
Work hours in a day 7.5
Work minutes in a day 450
Total work minutes/year 108,000
Person years required to cleanse each NSN once prior to migration:
Minutes needed 10,000,000
Minutes available person/year 108,000
Total Person-Years 92.6
Resource Cost to cleanse NSN's prior to migration:
Avg salary for SME year (not including overhead) $60,000.00
Projected years required to cleanse/total DLA person years saved 93
Total cost to cleanse/Total DLA savings to cleanse NSN's: $5.5 million

Quantifying Benefits: Revised Plan
NSNs 2,000,000
NSNs 150,000
Total Time (in minutes) 750,000
Minutes needed 750,000
Total Person-Years 7
Total cost to cleanse/Total DLA savings to cleanse NSN's: $420,000
Quantifying Benefits: Social Engineering
NSNs 2,000,000

Take Aways
• Too much investment is spent focused on
tools and technology at the expense of
problem understanding
• It is useful to understand data management
technologies and their use as part of a
people process & technology (3-legged)
stools
• Value that can be gained by profiling data
• Data volume is still increasing faster than we
are able to process it
• Data interchange overhead and other costs
of poor data practices are measurably
sapping organization and individual
resources–and therefore productivity
• Reliance on existing technology-based
approaches and education methods has not
materially addressed this gap between
creation and processing or reduced bottom
line costs
R. Buckminster Fuller
[ Clicking any webinar title will link directly to the registration page ]
Upcoming Events
Conceptual vs. Logical vs. Physical Data
12 July 2022
The Importance of Metadata
9 August 2022
Data Preparation Fundamentals
13 September 2022
Brought to you by:
Time: 19:00 UTC (2:00 PM NYC) | Presented by: Peter Aiken, PhD

Peter.Aiken@AnythingAwesome.com +1.804.382.5957
Thank You!
Book a call with Peter to discuss anything - https://anythingawesome.com/OfficeHours.html
Critical Design Review?
Hiring Assistance?
Reverse Engineering Expertise?
Executive Data
Literacy Training?
Mentoring?
Tool/automation evaluation?
Use your data more strategically?

Data Preparation Fundamentals

More Related Content

What's hot (20)

Similar to Data Preparation Fundamentals (20)

More from DATAVERSITY (20)

Recently uploaded (20)

Data Preparation Fundamentals