Subscribing to Your Critical Data Supply Chain - Getting Value from True Data Lineage

Subscribing to Your Critical Data Supply Chain
Getting Value from True Data Lineage
Stewart Bond
Research Director, Data Integration Software
1

Data is core to digital transformation:
intelligence is critical to data integrity
The data environment and supply
chain today is more complex than it
has ever been.
 Data needs to be available when and
where it is needed.
 Data needs to be secured at all levels of
access and consumption.
 Data needs to be compliant with
increasing regulatory environments.
 Data needs to be trusted.
Data intelligence gives organizations
the ability to answer the 5 W’s of
data: Who, What, Where, When,
Why and How.
© IDC Visit us at IDC.com and follow us on Twitter: @IDC 2

A Lack of Enterprise Data Integrity is
Already Impacting Digital Initiatives
3.07
3.08
3.19
3.22
3.31
3.39
3.41
2.90 3.00 3.10 3.20 3.30 3.40 3.50
Lack of an Enterprise Data Architecture
Poorly defined requirements
Human resource constraints
Technology (network bandwidth, storage, processing capacity)
constraints
Data constraints (Availability, Quality, Metadata, Lineage)
Budget constraints
Security and/or Compliance Policy constraints
(Mean Response)
(Scale 1=No Impact, 5=High Impact)
N=650
Source: IDC, 2015
Q. To what extent did the following project issues impact your ability to deliver solutions?

The 3rd Platform is Re-distributing Data
into Cloud and On-premises Silos
The number of hybrid and cloud-only data environments is
now greater than the number of environments on-premises.
Data Deployment Environments
Q. Thinking of your IT environment, select the platforms that data are being sent from or to in your data integration solutions.
Poll Question:
Where is the data
your organization
is integrating?
(Pick one)
On-Premise Only
Hybrid
Cloud Only

Cloudy Data is Harder to Trust
N=650
Source: IDC, 2015
As data moves into the
cloud, the number of
organizations with positive
measurements decreases.
The number of
organizations with negative
measurements increases.
Q. Thinking of your IT environment, select the platforms that data are being sent from or to in your
data integration solutions.
Q. Has your organization experienced a positive, negative, or neutral change as a result of tracking
each metric (duplicates, conflicts, stale, incomplete)?

Where and How Lineage
Source
One
Vendor
Vendor_ID
Name
Street_Address
City_Address
State_Address
Country_Address
Tax_ID
Invoices
Invoice_Number
Vendor_ID
Invoice_Total
Number_Lines
Target
Source
Two
Periods
2015Q1
2015Q2
Spend Report
Vendor Name
Vendor Tax_ID
Sum(Invoice_Total)
Period
Where
How
Data
Transformation
Process
Schema
Instance
Vendor Tax ID Spend Period
Acme 8524X921 $100000 2015Q1
Acme 8524X921 $150000 2015Q2
Sum
Group
By

Data intelligence requires more than
Where and How lineage

Internal to external relationships
 How do my customers relate to each other?
 How are my products and services related to where my customer is and what my customer is doing?
 How do my customers relate to, and engage with me?
External relationships
 Who is my customer?
 Where is my customer?
 How does my customer relate to people, places
and things?
Internal relationships
 Where is my customer’s data inside my organization?
 What relationships exist between master data about my
customer?
 Are transactions for my customer related to each other?
To other customers?
5 W’s aren’t enough; Relationship now
needs to be part of data intelligence
8

Data lineage delivers a range of impact
and value across use cases
 Data Governance
 Compliance
 Change Management
 Solution Development
 Storage Optimization
 Data Quality
 Problem Resolution
 Business Impact

Measuring the impacts of lineage
on data trust
n = 352
Base = respondents filtered to those that have implemented data integration software
Source: IDC's Data Integration End User Survey, 2015
Q. Has your organization experienced a positive, negative, or neutral change in each metric?

Measuring the impacts of data
lineage on availability and security
Q. Has your organization experienced a positive, negative, or neutral change in each metric?
n = 352
Base = respondents filtered to those that have implemented data integration software
Source: IDC's Data Integration End User Survey, 2015
Poll Question:
Is your organization
tracing lineage and is
if yes, is it automated?
(Pick one)
Yes/Automated
Yes/Manual
No

Data lineage as an input to solution
design: a case study
• Metadata
management
software
implemented and
provided to
development
teams.
• Information
catalog software
with business
glossary.
Solution
• More than 80% of
information about
data elements
available at
beginning of
sprints.
• Removed at least
one, 2-week sprint
per iteration.
• Higher quality
deliverables.
Outcome
• Multiple distributed
agile development
teams didn’t have
a consistent view
of data and data
lineage.
• Agile sprint length
impacted by data
lineage research
and impact
analysis.
Problem

Data lineage in data stewardship and
protection
• Metadata
management
software inclusive
of lineage and a
business glossary.
• A data lineage and
element glossary
dashboard was
developed and
rolled out to
business users.
Solution
• Time data
stewards spend on
data forensics has
become negligible.
• Backward and
forward lineage,
discovered
compliance and
security gaps, then
closed for policy
adherence.
Outcome
• Data stewards
spent 30-50% of
their time in data
forensics in
responding to
business users’
requests.
• Stale system
architecture
documents and
information.
Problem

Data lineage in compliance reporting
and app modernization: a case study
• Metadata
management
software inclusive
of lineage and
glossary.
• Automated zero-
gap data lineage
discovery software
able to interrogate
application code
and data schema.
Solution
• Regulatory audit
cycle time and
compliance
outcomes have
improved.
• Improved change
management and
reduced
operational risk in
modernization
solutions.
Outcome
• TARP and Basel II
regulation audit
requirements
drove a need for
data lineage.
• Application
modernization
projects also
required data
forensics and
impact analysis.
Problem

Zero-Gap Lineage
Data Intelligence
Data Governance
Instance Lineage

Copyright © 2016 ASG.
All rights reserved.
Sue Habas
Vice President of Product Management
Tuesday, June 28, 2016 – Dataversity Webinar
SUBSCRIBING TO YOUR CRITICAL DATA
SUPPLY CHAIN – GETTING VALUE FROM
TRUE DATA LINEAGE

App-File-Field
Transform
Rule
DB-Tab-Col
Calculation
Rule
Universe-Rep-
Field
The business asset foundation
CRITICAL LINEAGE SUPPLY CHAIN
Data Supply Chain
Business Traceability
E2E Data Driven Lineage
VerticalBusinessContextDriven
Customer/Patient/Event
Business
Terms
Policies
Critical Data
Elements

About 2 million hospital stays
were for patients without
insurance.
Patients covered by Medicare
experienced the longest average
length of stay (5.2 days), and
privately insured patients had the
shortest average length of stay
(3.8 days).
FAST ACCURATE ALIGNED
Reference
Data
Data
Inventory
Bus Glossary
Package
Data
Governance
Data
Lineage
The
Complete
Workflow
VALIDATE DATA INSIGHTS (HEALTHCARE)

How are claims processing impacted by a
change to LOS?
Rep
DB
BDL
MDM
Hub
Patient
Claims
EDW
Data Insights
Patient Claim
Reports
Where Lineage
How Lineage
CHANGE DETECTION AND SUBSCRIPTION
Level 1: Application Supply Chain
Are the upstream
Application owners
aware of the EDW
change?

The Critical Supply Chain for Average Length of Stay (ALOS)
SQL Override: In April of 2016 the LOS Supply Chain snapshot showed
Private Claims set to “Y” use “procedure start/end” versus Hospital
“check in/out date time” for “Region 14 (UK) “ for “out patient stays”.
Claims
In_pat_stay
Out_pat_stay
Claim_num
Private Y/N
Patient
Gender_cd:
Region:
State:
Age:
Discharge
Doc_release
Hosp_check_out
Procedure_comp
Claim_num
Patient
Reports
In_pat_stay
Out_pat_stay
Claim_num
BDL
Patient Insights
Medicare vs
Insured
LOS
Claims
LOS
Claims
May
April
MDM
Hub
CHANGE DETECTION AND SUBSCRIPTION
Level 2: Detail
Admission
Patient_arv_dt
Patient_arv_tm
Check_in
Start_proc
Claim_num
Hos_Check_in

INDUSTRY USE CASES FOR SUPPLY CHAIN CHANGE DETECTION
FINANCE
• BCBS –CCAR
Compliance
• FR Y-14
Attestation from
CFO
RETAIL
• Salesforce
• Facebook
• Big Data
• Information
Security
PHARMA
• Clinical
Outcomes
• Treatment
Patterns
• Safety
Poll question:
Do you have
resources
managing BI
lineage supply
chains in your
organizations
today?
(Pick one)
Yes
No
If yes, what
industry are you
in: Finance,
Retail,
Healthcare,
Pharma,
Manufacture,
Entertainment
HEALTHCARE
• Regional
Policies (ALOS)
• Claims
Processing
• Analytics
MANUFACTURING
• Product
Demographics
• Solution
Development
and Design
ENTERTAINMENT
• Client
Demographics
• PII

IMPLEMENTING DATA LINEAGE
We build out a reverse tracing methodology and base line
for comprehensive and accurate end to end data lineage.
Lineage
Change DetectionBusiness TraceabilityBusiness Glossary
ID Critical Data Scan Lineage Automate

EASILY DEPLOY DATA FACTS TO
ANYONE, ANYWHERE
ZERO GAP
DATA LINEAGE
LINEAGE
ANYWHERE
DATA
GOVERNANCE
THE LOB
LINEAGE
WORKFLOW
+
LINEAGE WITH THE DATA INTELLIGENCE SOLUTION

COBOL
IMS/DBD
COBOL BMS Map
Copybook
Business Objects
Universe
Informatica ETL
EDW to BI
Dynamic views of cross platform tracing
JAVA
Oracle EDW
{PLSQL}
ZERO GAP LINEAGE - FEATURES
• Level
• Trace
• Unpack
SNAPSHOTS STITCH
LINEAGE
LIFECYCLE
COMPLIANCE
REPORTS
ISSUE
MANAGEMENT
220+ Scanners and Parsers, Tool Agnostic
MAINFRAME

LINEAGE APPLIANCE
Includes code based ETL
MF/Dist/Hadoop
• Scan
• Analyze & Parse
• Link
• Store (version history/snapshots)
• Port to External System
Lineage
Anywhere
Portable
E2E Lineage Package

THANK YOU!
Stewart Bond
Research Director, Data Integration Software
IDC
@StewartLBond
ca.linkedin.com/in/stewartlbond
Sue Habas
Vice President, Product Management
ASG
sue.habas@asg.com

Subscribing to Your Critical Data Supply Chain - Getting Value from True Data Lineage

More Related Content

What's hot (20)

Similar to Subscribing to Your Critical Data Supply Chain - Getting Value from True Data Lineage (20)

More from DATAVERSITY (20)

Recently uploaded (20)

Subscribing to Your Critical Data Supply Chain - Getting Value from True Data Lineage