SlideShare a Scribd company logo
Data Modeling & Metadata Management
Donna Burbank
Global Data Strategy Ltd.
Lessons in Data Modeling DATAVERSITY Series
May 25th, 2017
Global Data Strategy, Ltd. 2017
Donna is a recognized industry expert in
information management with over 20
years of experience in data strategy,
information management, data modeling,
metadata management, and enterprise
architecture.
She is currently the Managing Director at
Global Data Strategy, Ltd., an
international information management
consulting company that specialises in the
alignment of business drivers with data-
centric technology. In past roles, she has
served in a number of roles related to
data modeling & metadata:
• Metadata consultant (US, Europe, Asia,
Africa)
• Product Manager PLATINUM Metadata
Repository
• Director of Product Management,
ER/Studio
• VP of Product Marketing, Erwin
• Data modeling & data strategy
implementation & consulting
• Author of 2 books of data modeling &
contributor to 1 book on metadata
management, plus numerous articles
• OMG committee member of the
Information Management Metamodel
(IMM)
As an active contributor to the data
management community, she is a long
time DAMA International member, Past
President and Advisor to the DAMA Rocky
Mountain chapter, and was recently
awarded the Excellence in Data
Management Award from DAMA
International in 2016. She has worked
with dozens of Fortune 500 companies
worldwide in the Americas, Europe, Asia,
and Africa and speaks regularly at
industry conferences. She has co-
authored two books: Data Modeling for
the Business and Data Modeling Made
Simple with ERwin and is a regular
contributor to industry publications such
as DATAVERSITY, EM360, & TDAN. She can
be reached at
donna.burbank@globaldatastrategy.com
Donna is based in Boulder, Colorado, USA.
Donna Burbank
2
Follow on Twitter @donnaburbank
Today’s hashtag: #LessonsDM
Global Data Strategy, Ltd. 2017
Lessons in Data Modeling Series
• January 26th How Data Modeling Fits Into an Overall Enterprise Architecture
• February 23rd Data Modeling and Business Intelligence
• March Conceptual Data Modeling – How to Get the Attention of Business Users
• April The Evolving Role of the Data Architect – What does it mean for your Career?
• May Data Modeling & Metadata Management
• June Self-Service Data Analysis, Data Wrangling, Data Munging, and Data Modeling
• July Data Modeling & Metadata for Graph Databases
• August Data Modeling & Data Integration
• September Data Modeling & MDM
• October Agile & Data Modeling – How Can They Work Together?
• December Data Modeling, Data Quality & Data Governance
3
This Year’s Line Up
Global Data Strategy, Ltd. 2017
Agenda
• How data modeling fits within a larger metadata management landscape
• When can data modeling provide “just enough” metadata management
• Key data modeling artifacts for metadata
• Organization, roles & implementation considerations
• Summary & questions
4
What we’ll cover today
Global Data Strategy, Ltd. 2017
Metadata is Hotter than ever
5
A Growing Trend
In a recent DATAVERSITY survey, over 80% of
respondents stated that:
Metadata is as important, if not more
important, than in the past.
Global Data Strategy, Ltd. 2017
What is Metadata?
Metadata is Data In Context
6
Global Data Strategy, Ltd. 2017
Metadata is the “Who, What, Where, Why, When & How” of Data
7
Who What Where Why When How
Who created this
data?
What is the business
definition of this data
element?
Where is this data
stored?
Why are we storing
this data?
When was this data
created?
How is this data
formatted?
(character, numeric,
etc.)
Who is the Steward of
this data?
What are the business
rules for this data?
Where did this data
come from?
What is its usage &
purpose?
When was this data
last updated?
How many databases
or data sources store
this data?
Who is using this
data?
What is the security
level or privacy level
of this data?
Where is this data
used & shared?
What are the business
drivers for using this
data?
How long should it be
stored?
Who “owns” this
data?
What is the
abbreviation or
acronym for this data
element?
Where is the backup
for this data?
When does it need to
be purged/deleted?
Who is regulating or
auditing this data?
What are the technical
naming standards for
database
implementation?
Are there regional
privacy or security
policies that regulate
this data?
Global Data Strategy, Ltd. 2017
Metadata is Part of a Larger Enterprise Landscape
8
A Successful Data Strategy Requires Many Inter-related Disciplines
“Top-Down” alignment with
business priorities
“Bottom-Up” management &
inventory of data sources
Managing the people, process,
policies & culture around data
Coordinating & integrating
disparate data sources
Leveraging & managing data for
strategic advantage
Global Data Strategy, Ltd. 2017
Metadata Across & Beyond the Organization
• Metadata exists in many sources across & beyond the organization.
9
COBOL
Legacy Systems
JCL
Spreadsheets
Media
Social
Media
IoTOpen Data
Databases
Data Models
Documents
Data
In Motion
Global Data Strategy, Ltd. 2017
Types of Metadata
• The DATAVERSITY Emerging Trends in Metadata survey revealed some interesting findings about
what types of metadata organizations will be managing now and in the future.
10
= Supported by most data modeling tools
Now Future
Global Data Strategy, Ltd. 2017
Data Models are a Good Source of Metadata
11
• Data Models are another good source of both business & technical metadata for relational
databases.
• They store structural metadata as well as business rules & definitions.
Customer
Customer_ID CHAR(18) NOT NULL
First Name
Last Name
City
Date Purchased
CHAR(18)
CHAR(18)
CHAR(18)
CHAR(18)
NOT NULL
NOT NULL
NULL
NULL
Technical Metadata Business Metadata
Global Data Strategy, Ltd. 2017
Data vs. Metadata
12
First Name Last Name Company City
Year
Purchased
Joe Smith Komputers R Us New York 1970
Mary Jones The Lord’s Store London 1999
Proful Bishwal The Lady’s Store Mumbai 1998
Ming Lee My Favorite Store Beijing 2001
Metadata
Data
Customer
Global Data Strategy, Ltd. 2017
Data vs. Metadata
13
STR01 STR02 TXT123 TXT127 DT01
Joe Smith Komputers R Us New York 1970
Mary Jones The Lord’s Store London 1999
Proful Bishwal The Lady’s Store Mumbai 1998
Ming Lee My Favorite Store Beijing 2001
Metadata?
Data
Customer
Global Data Strategy, Ltd. 2017
Metadata adds Context & Definition
14
First Name Last Name Company City
Year
Purchased
Joe Smith Komputers R Us New York 1970
Mary Jones The Lord’s Store London 1999
Proful Bishwal The Lady’s Store Mumbai 1998
Ming Lee My Favorite Store Beijing 2001
Customer Definition
Last Name represents the surname or family name of
an individual.
Business Rules
In the Chinese market, family name is listed first in
salutations.
Format VARCHAR(30)
Abbreviation LNAME
Required YES
Etc.
Numerous technical & business metadata including
security, privacy, nullability, primary key, etc.Is this the city where the customer lives
or where the store is located?
Global Data Strategy, Ltd. 2017
Technical & Business Metadata
• Technical Metadata describes the structure, format, and rules for storing data
• Business Metadata describes the business definitions, rules, and context for data.
• Data represents actual instances (e.g. John Smith)
15
CREATE TABLE EMPLOYEE (
employee_id INTEGER NOT NULL,
department_id INTEGER NOT NULL,
employee_fname VARCHAR(50) NULL,
employee_lname VARCHAR(50) NULL,
employee_ssn CHAR(9) NULL);
CREATE TABLE CUSTOMER (
customer_id INTEGER NOT NULL,
customer_name VARCHAR(50) NULL,
customer_address VARCHAR(150) NULL,
customer_city VARCHAR(50) NULL,
customer_state CHAR(2) NULL,
customer_zip CHAR(9) NULL);
Technical Metadata
John Smith
Business Metadata
Data
Term Definition
Employee
An employee is an individual who currently
works for the organization or who has been
recently employed within the past 6 months.
Customer
A customer is a person or organization who
has purchased from the organization within
the past 2 years and has an active loyalty card
or maintenance contract.
Global Data Strategy, Ltd. 2017
Business vs. Technical Metadata
• The following are examples of types of business & technical metadata.
16
Business Metadata Technical Metadata
• Definitions & Glossary
• Data Steward
• Organization
• Privacy Level
• Security Level
• Acronyms & Abbreviations
• Business Rules
• Etc.
• Column structure of a database table
• Data Type & Length (e.g. VARCHAR(20))
• Domains
• Standard abbreviations (e.g. CUSTOMER ->
CUST)
• Nullability
• Keys (primary, foreign, alternate, etc.)
• Validation Rules
• Data Movement Rules
• Permissions
• Etc.
Global Data Strategy, Ltd. 2017
Metadata is Needed by Business Stakeholders
17
Making business decisions on accurate and well-understood data
80% of users of metadata are from
the business, according to the
recent DATAVERSITY survey.
Business users often
“get” metadata more
than IT does!
Global Data Strategy, Ltd. 2017
Business Definitions
From Data Modeling for the Business by
Hoberman, Burbank, Bradley, Technics
Publications, 2009
Global Data Strategy, Ltd. 2017
Human Metadata
• Much business metadata and the history of the business exists in employee’s heads.
• It is important to capture this metadata in an electronic format for sharing with others.
• Avoid the dreaded “I just know”
19
Avoid the dreaded “I just know”
Part Number is what used to
be called Component
Number before the
acquisition.
Business Glossary
Metadata Repository
Data Models
Etc.
Global Data Strategy, Ltd. 2017
Poor Metadata Management Can be Expensive
20
On average organizations waste
15-18% of their budgets dealing
with data problems.
Source: Experian
56% of UK marketing organizations
say managing data quality is a
‘significant challenge’ .
Source: UK Marketing Today
The US economy loses $3.1 trillion
a year due to poor data quality .
Source: Artemis Ventures In the US, 6.9 billion pieces of mail
are undeliverable annually because
of address issues .
Source: US Postal Service
Correcting poor data quality is a
Data Scientist’s least favorite task,
consuming on average 80% of their
working day
Source: Forbes 2016
Global Data Strategy, Ltd. 2017
A Very Expensive Example - NASA
21
• On September 23, 1999 NASA lost the $125 million Mars Climate Orbiter spacecraft after a
286-day journey to Mars.
• Missing Metadata was the culprit
• Thruster data was sent in English units of pound-seconds (lbf s) instead of Metric units of newton-
seconds (N s)
• This metadata inconsistency caused thrusters to fire incorrectly, sending the craft off course –
60 miles in all (96.56 km).
• In addition to the cost of the orbiter were:
• Brand and Reputational Damage
• Lost Opportunities for research on the Martian atmosphere & climate
Global Data Strategy, Ltd. 2017
Efficiencies & Reuse
22
• Metadata Management can help rationalize data storage throughout the organization,
leading to significant efficiencies, cost reduction, and lessened risk of brand damage due to
incorrect definitions.
Customer information is stored
175 different ways across the
organization.
Customer, CUST, CDB1, Client, Party, etc, etc.
Identify the issue via
Metadata Discovery &
Inventory
1
Customer
Customer
Resolve & Prevent issues via
published metadata standards.
2
Standard Reference Metadata
Global Data Strategy, Ltd. 2017
Metadata Discovery Tools
• Metadata Discovery Tools extract metadata from source systems, and rationalize
them to a common metamodel and storage facility.
23
Metadata Discovery
Tools
Metamodel(s)
Metadata Storage
(Database)
Metadata Storage
(Repository)
Metadata
Population
Global Data Strategy, Ltd. 2017
Data Lineage - Data Warehousing Example
• In the data warehouse example below, metadata for CUSTOMER exists in a
number tools & data stores.
• This lineage can be tracked in most data modeling tools.
24
Sales Report
CUSTOMER
Database Table
CUST
Database Table
CUSTOMER
Database Table
CUSTOMER
Database Table
TBL_C1
Database Table
Business Glossary
ETL Tool ETL Tool
Physical Data Model
Physical Data Model
Logical Data Model
Dimensional
Data Model
BI Tool
Global Data Strategy, Ltd. 2017
Impact Analysis & Where Used
• Impact Analysis shows the relationship between a piece of metadata and other sources that rely
on that metadata to assess the impact of a potential change.
• For example, if I change the length & name of a field, what other systems that are referencing
that field will be affected?
25
What happens if I change the name &
length of the “Brand” field?
Brand CHAR(10)
MyBrand VARCHAR(30)
Sales Application
Sales Database
DB2
Staging Area
ETL
Customer
Database
Oracle
Global Data Strategy, Ltd. 2017
Design Layer Relationships
• In a data model there are several design layers that describe a given data
concept.
26
Metadata Beyond the Relational
Database
Many data modeling tools support these,
too
Global Data Strategy, Ltd. 2017
ERP, CRM and Packaged Application Metadata
28
• Packaged applications such as CRM and ERP systems (e.g. Salesforce, PeopleSoft, etc.) are
typically based on a relational database system.
• Therefore, there is important metadata about both the physical table structures as well as the
business names & definitions.
Technical Metadata Business Metadata
Global Data Strategy, Ltd. 2017
*
NoSQL Metadata – Document Databases
• Document databases are popular ways to store unstructured information in a flexible way (e.g.
multimedia, social media posts, etc. )
• Each Collection can contain numerous Documents which could all contain different fields.
29
• Some data modeling can be done, and some data modeling tools support this (e.g. MongoDB).
* Example from docs.mongodb.com
{type: “Artifact”,
medium: “Ceramic”
country: “China”,
}
{type: “Book”,
title: “Ancient China”
country: “China”,
}
Global Data Strategy, Ltd. 2017
NoSQL – Key Value Databases
• NoSQL Databases are often optimal solutions for flexibility & performance in certain scenarios.
• One common NoSQL database is a key-value pair database (e.g. Redis, Oracle NoSQL, etc.)
• They can support extremely high volumes of records & state changes per second through distributed
processing and distributed storage.
• Use cases include: Managing user sessions in web applications, online gaming, online shopping carts,
etc.
• The structure is often created by the application code, not within a database or metadata
structure.
• Metadata for NoSQL databases is typically minimal or non-existent.
• The structure & metadata is generally determined by the application code
30
Key Value
1839047 John Doe, Prepaid, 40.00
9287320 01/01/2008, 50.00, Green
Global Data Strategy, Ltd. 2017
COBOL Copybook Metadata
31
• What is a COBOL Copybook? – In COBOL, a copybook file is used to define data elements that
can be referenced by many programs
• What is COBOL Copybook Metadata? – structure, definition
Metadata
Describes structure & format of
data
The demand for COBOL & legacy
metadata is growing, according to
the recent DATAVERSITY survey.
Global Data Strategy, Ltd. 2017
XML Metadata
32
• What is XML? – (Extensible Markup Language) is used to store and transport data. It’s often a complement to HTML,
which is used to format the data.
• What is XML Metadata? – Similar to DDL, an XML Schema (XSD) defines the structure & format of data
<?xml version="1.0" encoding="UTF-8" ?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="shiporder">
<xs:complexType>
<xs:sequence>
<xs:element name="orderperson" type="xs:string"/>
<xs:element name="shipto">
<xs:complexType>
<xs:sequence>
<xs:element name="name" type="xs:string"/>
<xs:element name="address" type="xs:string"/>
<xs:element name="city" type="xs:string"/>
<xs:element name="country" type="xs:string"/>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
<xs:attribute name="orderid" type="xs:string" use="required"/>
</xs:complexType>
</xs:element>
</xs:schema>
XSD
Metadata
Ship to:
John Smith
123 Main ST
Boise
USA
………………………………………
………………………………………
Order Shipment
Data
<?xml version="1.0"?>
<shipto>
<name>John Smith</name>
<address>123 Main ST</address>
<city>Boise</city>
<country>USA</country>
</shipto>
XML
Data
Global Data Strategy, Ltd. 2017
JSON Metadata
33
• What is JSON? – (JavaScript Object Notation) is a minimal, readable format for structuring
data. It is used primarily to transmit data between a server and web application, as an
alternative to XML.
• What is JSON Metadata? – structure, definition
{
"id": 127849,
“brand": “Super Cooler",
"price": 12.50,
"tags": [“camping", “sports"]
}
Example Product in the API
Data
• Can the ID contain letters?
• What is a brand?
• Is a price required?
• Etc.
Context Needed
(i.e. Metadata)
For example, assume we have a JSON based product catalog. This catalog has a product which has an id, a brand,
a price, and an optional set of tags.
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Product",
"description": "A retail product from Acme's online catalog",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
“brand": {
"description": “The brand name of the product as shown in the online catalogue",
"type": "string"
},
"price": {
"type": "number",
},
"tags": {
"type": "array",
"items": {
"type": "string"
},
"minItems": 1,
}
},
"required": ["id", “brand", "price"]
}
JSON Schema
Metadata
Organization, Roles &
Implementation Considerations
Ensuring that metadata is used effectively
across the organization
Global Data Strategy, Ltd. 2017
Who Uses Metadata?
• In addition to sharing metadata between tools and via export,
many users across both IT & the business want to view the metadata through
reports, portals, etc.
35
Developer
If I change this field, what
else will be affected?
Business Person
(e.g. Finance)
What’s the definition of
“Regional Sales”
Auditor
How was “Total Sales”
calculated? Show me the
lineage.
Data
Architect
What is the approved data
structure for storing
customer data?
Data
Warehouse
Architect
What are the source-to-
target mappings for the
DW?
Business Person
(e.g. HR)
How can I get new staff up-
to-speed on our company’s
business terminology?
Global Data Strategy, Ltd. 2017
Metadata Publication & Reporting – Business Glossary
• A Business Glossary is a common way to publish business terms & their definitions.
• When sourced from a common repository, these terms are integrated with the wider data
landscape.
• Most data modeling tools can take the definitions from Logical and/or Conceptual data models
and publish them to a Glossary-style format, via web portals or reports.
36
Business Term Abbreviation Definition Data Steward Security Level
BFPO Number BFPO Num
BFPO Number is for British Forces Postal Office. It can be
used in UK and overseas addresses. Accounting Unclassified
Interest Int The growth in capital of a monetary investment Finance Unclassified
PO Box POB
A numbered box in a post office assigned to a person or
organization, where mail for them is kept until collected Accounting Unclassified
A feedback mechanism is important to gather valuable input & updates
from users.
Global Data Strategy, Ltd. 2017
Data Models can provide “Just Enough” Metadata Management
37
Metadata
Storage
Metadata
Lifecycle &
Versioning
Data Lineage
Visualization
Business Glossary Data Modeling
Metadata
Discovery &
Integration w/
Other Tools
Customizable
Metamodel
Data Modeling Tools
(e.g. Erwin, SAP
PowerDesigner, Idera
ER/Studio)
x X x X X x
Metadata Repositories (e.g.
ASG, Adaptive, DAG) X X X X X X
Data Governance Tools (e.g.
Collibra, Diaku) x x X x
Spreadsheets x x
• While data modeling tools are not metadata repositories, nor designed to be, they offer many features shared with these
repository solutions:
• Metadata storage, Data lineage visualization, Business Glossary, Integration with BI tools, ETL tools, etc.
• Metadata repositories have a broader range metadata sources & dedicated metadata management support.
• And Data Modeling tools, of course, have the added benefit of doing data modeling! 
• And the benefit is that much of the needed metadata is in these data models.
Global Data Strategy, Ltd. 2017
Key Components of Metadata Management
38
Metadata Strategy Metadata Capture &
Storage
Metadata Integration &
Publication
Metadata Management &
Governance
Alignment with business goals
& strategy
Identification of all internal &
external metadata sources
Identification of all technical
metadata sources
Metadata roles &
responsibilities defined
Identification of & feedback
from key stakeholders
Population/import mechanism
for all identified sources
Identification of key
stakeholders & audiences
(internal & external)
Metadata standards created
Prioritization of key activities
aligned with business needs &
technical capabilities
Identification of existing
metadata storage
Integration mechanism for key
technologies (direct
integration, export, etc.)
Metadata lifecycle
management defined &
implemented
Prioritization of key data
elements/subject areas
Definition of enterprise
metadata storage strategy
Publication mechanism for
each audience
Metadata quality statistics
defined & monitored
Communication Plan
developed
Feedback mechanism for each
audience
Metadata integrated into
operational activities & related
data management projects
Global Data Strategy, Ltd. 2017
Implementing a Metadata Strategy
• A successful metadata strategy requires input from multiple factors.
39
Business Drivers & Motivation
Metadata Sources & Technology
Metadata Management MaturityStakeholders & Audience Metadata
Strategy
Global Data Strategy, Ltd. 2017
Stakeholder Feedback
• Determine key business issues & drivers through direct feedback.
40
I didn’t know we had any
documented data
standards
Where do I go to get the
definition of “default
banking standard”?
$12m has been spent on
projects to clean up the data
over the past 2-3 years
What are the data structures
used in the application?
We have 15 customer
databases – with many
duplications.
There is limited ownership or
enforcement of common
practices and standards
across the projects
Key subject matter experts
are relied upon to review
detailed data from various
systems to ensure accuracy.
I just joined the company and
don’t understand all of the
acronyms!
There was an error in reporting
products by customer & region
that was noticed by upper
management.
I need a central, accurate
view of all my customers
worldwide.
Global Data Strategy, Ltd. 2017
Mapping Business Drivers to Metadata Management
Capabilities
41
Business Drivers
Digital
Self Service
Increasing Regulatory
Pressures
Online Community &
Social Media
Community Building
External Drivers
Internal Drivers
Targeted Marketing
360 View of Customer
Brand Reputation
Efficient IT
Stakeholder Challenges
Lack of Business Alignment
• Data spend not aligned to Business Plans
• Business users not involved with data
Integrating Data
• Siloed systems
• No common view of key information
3 Data Quality Issues
• Bad customer info causing Brand damage
• Completeness & Accuracy Needed
4
Cost of Data Management
• Manual entry increases costs
• System redundancy
• No reuse or standards
5 No Audit Trails
• No lineage of changes
• Fines had been levied in past for lack of
compliance
6 Big Data Exploitation
• Exploiting Unstructured Data
• Access to External & Social Data
1
Shows “Heat
Map” of Priorities
2
3
4
5
6
Metadata Capability
Metadata Strategy
Metadata Capture &
Storage
Metadata Integration &
Publication
Metadata Management &
Governance
1 2 3 4 5 6
2 3 4 5 6
2 3 4
1 2 3 4 5 6
Global Data Strategy, Ltd. 2017
Inventory & Usage Mapping
• It’s also important to determine which teams are using these technologies to
create a “heat map” of usage & priority.
42
Metadata Sources Leadership Sales Finance Marketing Support R&D HR Legal Compliance
Relational Databases
MySQL X
Oracle X X X X X X X X
SQL Server X X
Sybase X
Etc.
BI Tools
Tableau X X X X X X
Qlik X X X
Etc.
Open Data
Data.gov – agricultural data X X X
Etc.
Global Data Strategy, Ltd. 2017
Metadata Roles & Responsibilities
• It’s important to establish formal roles & responsibilities for your metadata effort.
• Some may be part-time, and some full-time, but they should be clearly defined and
communicated so that staff has understanding of and accountability for their roles.
• Executive Sponsor/Champion: Understands & communicates the importance of metadata
management across the organization.
• Steering Group: As part of a metadata management effort, or part of a larger data governance effort,
the steering group prioritizes & sets direction for key activities.
• Data Stewards: Responsible for business definitions & rules for key data elements.
• Metadata Repository Administrator: Manages the administration, population, and interfaces of a
metadata repository.
• Metadata Publicist: Establishes reports & publication methods to end users.
• Metadata Consumers: Actively use metadata as part of their daily jobs, and are held accountable for
using published standards.
• Data Modelers
• Developers
• Business Users
• Report Developers
• Etc.
43
Global Data Strategy, Ltd. 2017
Monitoring Metadata Quality & Metrics
• Metadata is a key driver of data quality, and to support this, the metadata itself must be of high
quality.
• In order to ensure that quality metadata is maintained, it must be actively managed and
monitored. Dashboards & Reports can be used to monitor key quality indicators.
• Key metadata quality indicators include:
• Completeness: e.g. Do definitions exist for all key data elements?
• Accuracy: e.g. Are current definitions correct? Do data types accurately represent currently
implemented standards?
• Currency/ Timeliness: e.g. Are metadata definitions current or outdated?
• Consistency: e.g. Are metadata standards defined, published & implemented consistently across the
organization?
• Accountability: e.g. Are data stewards or owners defined?
• Integrity: e.g. Are linkages and relationships established between critical metadata items?
• Privacy: e.g. Is any metadata subject to privacy restrictions?
• Usability: e.g. Are people actually using this metadata?
44
Global Data Strategy, Ltd. 2017
Summary
• Metadata is more important than ever
• Data models are a rich source of metadata
• Metadata exists in a number of sources beyond relational databases
• ERP & CRM
• Legacy systems
• NoSQL
• XML & JSON
• Etc.
• While metadata repositories are valuable, data models & associated functionality can often provide “just
enough” metadata management
• Business definitions
• Technical data structures (relational and beyond)
• Data lineage & impact analysis
• Visual models
• Organizational considerations are critical to achieve success
• Understanding business drivers
• Defining roles & responsibilities
• Monitoring metadata quality & metrics
• Have fun! Metadata is for the cool kids.
Global Data Strategy, Ltd. 2017
About Global Data Strategy, Ltd
• Global Data Strategy is an international information management consulting company that specializes
in the alignment of business drivers with data-centric technology.
• Our passion is data, and helping organizations enrich their business opportunities through data and
information.
• Our core values center around providing solutions that are:
• Business-Driven: We put the needs of your business first, before we look at any technology solution.
• Clear & Relevant: We provide clear explanations using real-world examples.
• Customized & Right-Sized: Our implementations are based on the unique needs of your organization’s
size, corporate culture, and geography.
• High Quality & Technically Precise: We pride ourselves in excellence of execution, with years of
technical expertise in the industry.
46
Data-Driven Business Transformation
Business Strategy
Aligned With
Data Strategy
Visit www.globaldatastrategy.com for more information
Global Data Strategy, Ltd. 2017
Contact Info
• Email: donna.burbank@globaldatastrategy.com
• Twitter: @donnaburbank
@GlobalDataStrat
• Website: www.globaldatastrategy.com
• Company Linkedin: https://www.linkedin.com/company/global-data-strategy-ltd
• Personal Linkedin: https://www.linkedin.com/in/donnaburbank
47
Global Data Strategy, Ltd. 2017
White Paper: Emerging Trends in Metadata Management
48
Free Download
• Download from www.dataversity.net
• Also available on www.globaldatastategy.com
Global Data Strategy, Ltd. 2017
DATAVERSITY Training Center
• Learn the basics of Metadata Management and practical tips on how to apply metadata
management in the real world. This online course hosted by DATAVERSITY provides a series of six
courses including:
• What is Metadata
• The Business Value of Metadata
• Sources of Metadata
• Metamodels and Metadata Standards
• Metadata Architecture, Integration, and Storage
• Metadata Strategy and Implementation
• Purchase all six courses for $399 or individually at $79 each.
Register here
• Other courses available on Data Governance & Data Quality
49
Online Training Courses
Metadata Management Course
Visit: http://training.dataversity.net/lms/
Global Data Strategy, Ltd. 2017
Lessons in Data Modeling Series
• January 26th How Data Modeling Fits Into an Overall Enterprise Architecture
• February 23rd Data Modeling and Business Intelligence
• March Conceptual Data Modeling – How to Get the Attention of Business Users
• April The Evolving Role of the Data Architect – What does it mean for your Career?
• May Data Modeling & Metadata Management
• June Self-Service Data Analysis, Data Wrangling, Data Munging, and Data Modeling
• July Data Modeling & Metadata for Graph Databases
• August Data Modeling & Data Integration
• September Data Modeling & MDM
• October Agile & Data Modeling – How Can They Work Together?
• December Data Modeling, Data Quality & Data Governance
50
This Year’s Line Up
Global Data Strategy, Ltd. 2017
Questions?
51
Thoughts? Ideas?

More Related Content

PDF
Modern Metadata Strategies
PDF
The Importance of Metadata
PDF
Best Practices in Metadata Management
PDF
Data Management vs Data Strategy
PDF
The Business Value of Metadata for Data Governance
PDF
Data at the Speed of Business with Data Mastering and Governance
PDF
Essential Reference and Master Data Management
PDF
Data Governance and Metadata Management
Modern Metadata Strategies
The Importance of Metadata
Best Practices in Metadata Management
Data Management vs Data Strategy
The Business Value of Metadata for Data Governance
Data at the Speed of Business with Data Mastering and Governance
Essential Reference and Master Data Management
Data Governance and Metadata Management

What's hot (20)

PDF
Best Practices in Metadata Management
PDF
Data Catalog for Better Data Discovery and Governance
PDF
Data Governance Best Practices
PDF
Master Data Management – Aligning Data, Process, and Governance
PPTX
How to Build & Sustain a Data Governance Operating Model
PDF
Data Modeling, Data Governance, & Data Quality
PPTX
Data Governance
PDF
Reference master data management
PDF
Data modeling for the business
PDF
Introduction to Data Governance
PPT
Data Governance
PDF
CDMP preparation workshop EDW2016
PPTX
DMBOK - Chapter 1 Summary
PPTX
Chapter 1: The Importance of Data Assets
PPT
Gartner: Master Data Management Functionality
PPTX
Data Quality & Data Governance
PDF
Data Management Maturity Assessment
PDF
You Need a Data Catalog. Do You Know Why?
PDF
Data Catalogs Are the Answer – What is the Question?
PPTX
‏‏‏‏‏‏‏‏‏‏‏‏Chapter 13: Professional Development
Best Practices in Metadata Management
Data Catalog for Better Data Discovery and Governance
Data Governance Best Practices
Master Data Management – Aligning Data, Process, and Governance
How to Build & Sustain a Data Governance Operating Model
Data Modeling, Data Governance, & Data Quality
Data Governance
Reference master data management
Data modeling for the business
Introduction to Data Governance
Data Governance
CDMP preparation workshop EDW2016
DMBOK - Chapter 1 Summary
Chapter 1: The Importance of Data Assets
Gartner: Master Data Management Functionality
Data Quality & Data Governance
Data Management Maturity Assessment
You Need a Data Catalog. Do You Know Why?
Data Catalogs Are the Answer – What is the Question?
‏‏‏‏‏‏‏‏‏‏‏‏Chapter 13: Professional Development
Ad

Similar to Data Modeling & Metadata Management (20)

PDF
LDM Webinar: Data Modeling & Metadata Management
PDF
LDM Slides: Conceptual Data Models - How to Get the Attention of Business Use...
PDF
LDM Webinar: Data Modeling & Business Intelligence
PDF
Lessons in Data Modeling: Why a Data Model is an Important Part of Your Data ...
PDF
Data Modeling & Data Integration
PDF
Data Modeling Best Practices - Business & Technical Approaches
PDF
DAS Slides: Building a Future-State Data Architecture Plan - Where to Begin?
PDF
DAS Slides: Master Data Management – Aligning Data, Process, and Governance
PPTX
dataversitydatacatalogslidenotesslidenotesslidenotes
PDF
Data Catalogues - Architecting for Collaboration & Self-Service
PDF
DAS Webinar: Emerging Trends in Data Architecture – What’s the Next Big Thing?
PDF
DAS Slides: Self-Service Reporting and Data Prep – Benefits & Risks
PDF
dataversitydatagovernanceorgchangeapril2019-190429155809.pdf
PDF
DAS Slides: Data Governance - Combining Data Management with Organizational ...
PDF
The Evolving Role of the Data Architect – What Does It Mean for Your Career?
PDF
Lessons in Data Modeling: Data Modeling & MDM
PDF
The Missing Link in Enterprise Data Governance - Automated Metadata Management
PDF
DAS Slides: Enterprise Architecture vs. Data Architecture
PDF
DAS Slides: Best Practices in Metadata Management
PDF
Self-Service Data Analysis, Data Wrangling, Data Munging, and Data Modeling –...
LDM Webinar: Data Modeling & Metadata Management
LDM Slides: Conceptual Data Models - How to Get the Attention of Business Use...
LDM Webinar: Data Modeling & Business Intelligence
Lessons in Data Modeling: Why a Data Model is an Important Part of Your Data ...
Data Modeling & Data Integration
Data Modeling Best Practices - Business & Technical Approaches
DAS Slides: Building a Future-State Data Architecture Plan - Where to Begin?
DAS Slides: Master Data Management – Aligning Data, Process, and Governance
dataversitydatacatalogslidenotesslidenotesslidenotes
Data Catalogues - Architecting for Collaboration & Self-Service
DAS Webinar: Emerging Trends in Data Architecture – What’s the Next Big Thing?
DAS Slides: Self-Service Reporting and Data Prep – Benefits & Risks
dataversitydatagovernanceorgchangeapril2019-190429155809.pdf
DAS Slides: Data Governance - Combining Data Management with Organizational ...
The Evolving Role of the Data Architect – What Does It Mean for Your Career?
Lessons in Data Modeling: Data Modeling & MDM
The Missing Link in Enterprise Data Governance - Automated Metadata Management
DAS Slides: Enterprise Architecture vs. Data Architecture
DAS Slides: Best Practices in Metadata Management
Self-Service Data Analysis, Data Wrangling, Data Munging, and Data Modeling –...
Ad

More from DATAVERSITY (20)

PDF
Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...
PDF
Exploring Levels of Data Literacy
PDF
Building a Data Strategy – Practical Steps for Aligning with Business Goals
PDF
Make Data Work for You
PDF
Data Catalogs Are the Answer – What Is the Question?
PDF
Data Modeling Fundamentals
PDF
Showing ROI for Your Analytic Project
PDF
How a Semantic Layer Makes Data Mesh Work at Scale
PDF
Is Enterprise Data Literacy Possible?
PDF
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
PDF
Emerging Trends in Data Architecture – What’s the Next Big Thing?
PDF
Data Governance Trends - A Look Backwards and Forwards
PDF
Data Governance Trends and Best Practices To Implement Today
PDF
2023 Trends in Enterprise Analytics
PDF
Data Strategy Best Practices
PDF
Who Should Own Data Governance – IT or Business?
PDF
Data Management Best Practices
PDF
MLOps – Applying DevOps to Competitive Advantage
PDF
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...
PDF
Empowering the Data Driven Business with Modern Business Intelligence
Architecture, Products, and Total Cost of Ownership of the Leading Machine Le...
Exploring Levels of Data Literacy
Building a Data Strategy – Practical Steps for Aligning with Business Goals
Make Data Work for You
Data Catalogs Are the Answer – What Is the Question?
Data Modeling Fundamentals
Showing ROI for Your Analytic Project
How a Semantic Layer Makes Data Mesh Work at Scale
Is Enterprise Data Literacy Possible?
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
Emerging Trends in Data Architecture – What’s the Next Big Thing?
Data Governance Trends - A Look Backwards and Forwards
Data Governance Trends and Best Practices To Implement Today
2023 Trends in Enterprise Analytics
Data Strategy Best Practices
Who Should Own Data Governance – IT or Business?
Data Management Best Practices
MLOps – Applying DevOps to Competitive Advantage
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve D...
Empowering the Data Driven Business with Modern Business Intelligence

Recently uploaded (20)

PDF
Electronic commerce courselecture one. Pdf
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
DOCX
The AUB Centre for AI in Media Proposal.docx
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
PDF
Empathic Computing: Creating Shared Understanding
PDF
Approach and Philosophy of On baking technology
PPTX
Understanding_Digital_Forensics_Presentation.pptx
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
PPTX
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
PDF
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
PDF
Network Security Unit 5.pdf for BCA BBA.
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
Electronic commerce courselecture one. Pdf
Reach Out and Touch Someone: Haptics and Empathic Computing
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
The AUB Centre for AI in Media Proposal.docx
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
Empathic Computing: Creating Shared Understanding
Approach and Philosophy of On baking technology
Understanding_Digital_Forensics_Presentation.pptx
Per capita expenditure prediction using model stacking based on satellite ima...
20250228 LYD VKU AI Blended-Learning.pptx
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
Dropbox Q2 2025 Financial Results & Investor Presentation
Network Security Unit 5.pdf for BCA BBA.
Diabetes mellitus diagnosis method based random forest with bat algorithm

Data Modeling & Metadata Management

  • 1. Data Modeling & Metadata Management Donna Burbank Global Data Strategy Ltd. Lessons in Data Modeling DATAVERSITY Series May 25th, 2017
  • 2. Global Data Strategy, Ltd. 2017 Donna is a recognized industry expert in information management with over 20 years of experience in data strategy, information management, data modeling, metadata management, and enterprise architecture. She is currently the Managing Director at Global Data Strategy, Ltd., an international information management consulting company that specialises in the alignment of business drivers with data- centric technology. In past roles, she has served in a number of roles related to data modeling & metadata: • Metadata consultant (US, Europe, Asia, Africa) • Product Manager PLATINUM Metadata Repository • Director of Product Management, ER/Studio • VP of Product Marketing, Erwin • Data modeling & data strategy implementation & consulting • Author of 2 books of data modeling & contributor to 1 book on metadata management, plus numerous articles • OMG committee member of the Information Management Metamodel (IMM) As an active contributor to the data management community, she is a long time DAMA International member, Past President and Advisor to the DAMA Rocky Mountain chapter, and was recently awarded the Excellence in Data Management Award from DAMA International in 2016. She has worked with dozens of Fortune 500 companies worldwide in the Americas, Europe, Asia, and Africa and speaks regularly at industry conferences. She has co- authored two books: Data Modeling for the Business and Data Modeling Made Simple with ERwin and is a regular contributor to industry publications such as DATAVERSITY, EM360, & TDAN. She can be reached at donna.burbank@globaldatastrategy.com Donna is based in Boulder, Colorado, USA. Donna Burbank 2 Follow on Twitter @donnaburbank Today’s hashtag: #LessonsDM
  • 3. Global Data Strategy, Ltd. 2017 Lessons in Data Modeling Series • January 26th How Data Modeling Fits Into an Overall Enterprise Architecture • February 23rd Data Modeling and Business Intelligence • March Conceptual Data Modeling – How to Get the Attention of Business Users • April The Evolving Role of the Data Architect – What does it mean for your Career? • May Data Modeling & Metadata Management • June Self-Service Data Analysis, Data Wrangling, Data Munging, and Data Modeling • July Data Modeling & Metadata for Graph Databases • August Data Modeling & Data Integration • September Data Modeling & MDM • October Agile & Data Modeling – How Can They Work Together? • December Data Modeling, Data Quality & Data Governance 3 This Year’s Line Up
  • 4. Global Data Strategy, Ltd. 2017 Agenda • How data modeling fits within a larger metadata management landscape • When can data modeling provide “just enough” metadata management • Key data modeling artifacts for metadata • Organization, roles & implementation considerations • Summary & questions 4 What we’ll cover today
  • 5. Global Data Strategy, Ltd. 2017 Metadata is Hotter than ever 5 A Growing Trend In a recent DATAVERSITY survey, over 80% of respondents stated that: Metadata is as important, if not more important, than in the past.
  • 6. Global Data Strategy, Ltd. 2017 What is Metadata? Metadata is Data In Context 6
  • 7. Global Data Strategy, Ltd. 2017 Metadata is the “Who, What, Where, Why, When & How” of Data 7 Who What Where Why When How Who created this data? What is the business definition of this data element? Where is this data stored? Why are we storing this data? When was this data created? How is this data formatted? (character, numeric, etc.) Who is the Steward of this data? What are the business rules for this data? Where did this data come from? What is its usage & purpose? When was this data last updated? How many databases or data sources store this data? Who is using this data? What is the security level or privacy level of this data? Where is this data used & shared? What are the business drivers for using this data? How long should it be stored? Who “owns” this data? What is the abbreviation or acronym for this data element? Where is the backup for this data? When does it need to be purged/deleted? Who is regulating or auditing this data? What are the technical naming standards for database implementation? Are there regional privacy or security policies that regulate this data?
  • 8. Global Data Strategy, Ltd. 2017 Metadata is Part of a Larger Enterprise Landscape 8 A Successful Data Strategy Requires Many Inter-related Disciplines “Top-Down” alignment with business priorities “Bottom-Up” management & inventory of data sources Managing the people, process, policies & culture around data Coordinating & integrating disparate data sources Leveraging & managing data for strategic advantage
  • 9. Global Data Strategy, Ltd. 2017 Metadata Across & Beyond the Organization • Metadata exists in many sources across & beyond the organization. 9 COBOL Legacy Systems JCL Spreadsheets Media Social Media IoTOpen Data Databases Data Models Documents Data In Motion
  • 10. Global Data Strategy, Ltd. 2017 Types of Metadata • The DATAVERSITY Emerging Trends in Metadata survey revealed some interesting findings about what types of metadata organizations will be managing now and in the future. 10 = Supported by most data modeling tools Now Future
  • 11. Global Data Strategy, Ltd. 2017 Data Models are a Good Source of Metadata 11 • Data Models are another good source of both business & technical metadata for relational databases. • They store structural metadata as well as business rules & definitions. Customer Customer_ID CHAR(18) NOT NULL First Name Last Name City Date Purchased CHAR(18) CHAR(18) CHAR(18) CHAR(18) NOT NULL NOT NULL NULL NULL Technical Metadata Business Metadata
  • 12. Global Data Strategy, Ltd. 2017 Data vs. Metadata 12 First Name Last Name Company City Year Purchased Joe Smith Komputers R Us New York 1970 Mary Jones The Lord’s Store London 1999 Proful Bishwal The Lady’s Store Mumbai 1998 Ming Lee My Favorite Store Beijing 2001 Metadata Data Customer
  • 13. Global Data Strategy, Ltd. 2017 Data vs. Metadata 13 STR01 STR02 TXT123 TXT127 DT01 Joe Smith Komputers R Us New York 1970 Mary Jones The Lord’s Store London 1999 Proful Bishwal The Lady’s Store Mumbai 1998 Ming Lee My Favorite Store Beijing 2001 Metadata? Data Customer
  • 14. Global Data Strategy, Ltd. 2017 Metadata adds Context & Definition 14 First Name Last Name Company City Year Purchased Joe Smith Komputers R Us New York 1970 Mary Jones The Lord’s Store London 1999 Proful Bishwal The Lady’s Store Mumbai 1998 Ming Lee My Favorite Store Beijing 2001 Customer Definition Last Name represents the surname or family name of an individual. Business Rules In the Chinese market, family name is listed first in salutations. Format VARCHAR(30) Abbreviation LNAME Required YES Etc. Numerous technical & business metadata including security, privacy, nullability, primary key, etc.Is this the city where the customer lives or where the store is located?
  • 15. Global Data Strategy, Ltd. 2017 Technical & Business Metadata • Technical Metadata describes the structure, format, and rules for storing data • Business Metadata describes the business definitions, rules, and context for data. • Data represents actual instances (e.g. John Smith) 15 CREATE TABLE EMPLOYEE ( employee_id INTEGER NOT NULL, department_id INTEGER NOT NULL, employee_fname VARCHAR(50) NULL, employee_lname VARCHAR(50) NULL, employee_ssn CHAR(9) NULL); CREATE TABLE CUSTOMER ( customer_id INTEGER NOT NULL, customer_name VARCHAR(50) NULL, customer_address VARCHAR(150) NULL, customer_city VARCHAR(50) NULL, customer_state CHAR(2) NULL, customer_zip CHAR(9) NULL); Technical Metadata John Smith Business Metadata Data Term Definition Employee An employee is an individual who currently works for the organization or who has been recently employed within the past 6 months. Customer A customer is a person or organization who has purchased from the organization within the past 2 years and has an active loyalty card or maintenance contract.
  • 16. Global Data Strategy, Ltd. 2017 Business vs. Technical Metadata • The following are examples of types of business & technical metadata. 16 Business Metadata Technical Metadata • Definitions & Glossary • Data Steward • Organization • Privacy Level • Security Level • Acronyms & Abbreviations • Business Rules • Etc. • Column structure of a database table • Data Type & Length (e.g. VARCHAR(20)) • Domains • Standard abbreviations (e.g. CUSTOMER -> CUST) • Nullability • Keys (primary, foreign, alternate, etc.) • Validation Rules • Data Movement Rules • Permissions • Etc.
  • 17. Global Data Strategy, Ltd. 2017 Metadata is Needed by Business Stakeholders 17 Making business decisions on accurate and well-understood data 80% of users of metadata are from the business, according to the recent DATAVERSITY survey. Business users often “get” metadata more than IT does!
  • 18. Global Data Strategy, Ltd. 2017 Business Definitions From Data Modeling for the Business by Hoberman, Burbank, Bradley, Technics Publications, 2009
  • 19. Global Data Strategy, Ltd. 2017 Human Metadata • Much business metadata and the history of the business exists in employee’s heads. • It is important to capture this metadata in an electronic format for sharing with others. • Avoid the dreaded “I just know” 19 Avoid the dreaded “I just know” Part Number is what used to be called Component Number before the acquisition. Business Glossary Metadata Repository Data Models Etc.
  • 20. Global Data Strategy, Ltd. 2017 Poor Metadata Management Can be Expensive 20 On average organizations waste 15-18% of their budgets dealing with data problems. Source: Experian 56% of UK marketing organizations say managing data quality is a ‘significant challenge’ . Source: UK Marketing Today The US economy loses $3.1 trillion a year due to poor data quality . Source: Artemis Ventures In the US, 6.9 billion pieces of mail are undeliverable annually because of address issues . Source: US Postal Service Correcting poor data quality is a Data Scientist’s least favorite task, consuming on average 80% of their working day Source: Forbes 2016
  • 21. Global Data Strategy, Ltd. 2017 A Very Expensive Example - NASA 21 • On September 23, 1999 NASA lost the $125 million Mars Climate Orbiter spacecraft after a 286-day journey to Mars. • Missing Metadata was the culprit • Thruster data was sent in English units of pound-seconds (lbf s) instead of Metric units of newton- seconds (N s) • This metadata inconsistency caused thrusters to fire incorrectly, sending the craft off course – 60 miles in all (96.56 km). • In addition to the cost of the orbiter were: • Brand and Reputational Damage • Lost Opportunities for research on the Martian atmosphere & climate
  • 22. Global Data Strategy, Ltd. 2017 Efficiencies & Reuse 22 • Metadata Management can help rationalize data storage throughout the organization, leading to significant efficiencies, cost reduction, and lessened risk of brand damage due to incorrect definitions. Customer information is stored 175 different ways across the organization. Customer, CUST, CDB1, Client, Party, etc, etc. Identify the issue via Metadata Discovery & Inventory 1 Customer Customer Resolve & Prevent issues via published metadata standards. 2 Standard Reference Metadata
  • 23. Global Data Strategy, Ltd. 2017 Metadata Discovery Tools • Metadata Discovery Tools extract metadata from source systems, and rationalize them to a common metamodel and storage facility. 23 Metadata Discovery Tools Metamodel(s) Metadata Storage (Database) Metadata Storage (Repository) Metadata Population
  • 24. Global Data Strategy, Ltd. 2017 Data Lineage - Data Warehousing Example • In the data warehouse example below, metadata for CUSTOMER exists in a number tools & data stores. • This lineage can be tracked in most data modeling tools. 24 Sales Report CUSTOMER Database Table CUST Database Table CUSTOMER Database Table CUSTOMER Database Table TBL_C1 Database Table Business Glossary ETL Tool ETL Tool Physical Data Model Physical Data Model Logical Data Model Dimensional Data Model BI Tool
  • 25. Global Data Strategy, Ltd. 2017 Impact Analysis & Where Used • Impact Analysis shows the relationship between a piece of metadata and other sources that rely on that metadata to assess the impact of a potential change. • For example, if I change the length & name of a field, what other systems that are referencing that field will be affected? 25 What happens if I change the name & length of the “Brand” field? Brand CHAR(10) MyBrand VARCHAR(30) Sales Application Sales Database DB2 Staging Area ETL Customer Database Oracle
  • 26. Global Data Strategy, Ltd. 2017 Design Layer Relationships • In a data model there are several design layers that describe a given data concept. 26
  • 27. Metadata Beyond the Relational Database Many data modeling tools support these, too
  • 28. Global Data Strategy, Ltd. 2017 ERP, CRM and Packaged Application Metadata 28 • Packaged applications such as CRM and ERP systems (e.g. Salesforce, PeopleSoft, etc.) are typically based on a relational database system. • Therefore, there is important metadata about both the physical table structures as well as the business names & definitions. Technical Metadata Business Metadata
  • 29. Global Data Strategy, Ltd. 2017 * NoSQL Metadata – Document Databases • Document databases are popular ways to store unstructured information in a flexible way (e.g. multimedia, social media posts, etc. ) • Each Collection can contain numerous Documents which could all contain different fields. 29 • Some data modeling can be done, and some data modeling tools support this (e.g. MongoDB). * Example from docs.mongodb.com {type: “Artifact”, medium: “Ceramic” country: “China”, } {type: “Book”, title: “Ancient China” country: “China”, }
  • 30. Global Data Strategy, Ltd. 2017 NoSQL – Key Value Databases • NoSQL Databases are often optimal solutions for flexibility & performance in certain scenarios. • One common NoSQL database is a key-value pair database (e.g. Redis, Oracle NoSQL, etc.) • They can support extremely high volumes of records & state changes per second through distributed processing and distributed storage. • Use cases include: Managing user sessions in web applications, online gaming, online shopping carts, etc. • The structure is often created by the application code, not within a database or metadata structure. • Metadata for NoSQL databases is typically minimal or non-existent. • The structure & metadata is generally determined by the application code 30 Key Value 1839047 John Doe, Prepaid, 40.00 9287320 01/01/2008, 50.00, Green
  • 31. Global Data Strategy, Ltd. 2017 COBOL Copybook Metadata 31 • What is a COBOL Copybook? – In COBOL, a copybook file is used to define data elements that can be referenced by many programs • What is COBOL Copybook Metadata? – structure, definition Metadata Describes structure & format of data The demand for COBOL & legacy metadata is growing, according to the recent DATAVERSITY survey.
  • 32. Global Data Strategy, Ltd. 2017 XML Metadata 32 • What is XML? – (Extensible Markup Language) is used to store and transport data. It’s often a complement to HTML, which is used to format the data. • What is XML Metadata? – Similar to DDL, an XML Schema (XSD) defines the structure & format of data <?xml version="1.0" encoding="UTF-8" ?> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> <xs:element name="shiporder"> <xs:complexType> <xs:sequence> <xs:element name="orderperson" type="xs:string"/> <xs:element name="shipto"> <xs:complexType> <xs:sequence> <xs:element name="name" type="xs:string"/> <xs:element name="address" type="xs:string"/> <xs:element name="city" type="xs:string"/> <xs:element name="country" type="xs:string"/> </xs:sequence> </xs:complexType> </xs:element> </xs:sequence> <xs:attribute name="orderid" type="xs:string" use="required"/> </xs:complexType> </xs:element> </xs:schema> XSD Metadata Ship to: John Smith 123 Main ST Boise USA ……………………………………… ……………………………………… Order Shipment Data <?xml version="1.0"?> <shipto> <name>John Smith</name> <address>123 Main ST</address> <city>Boise</city> <country>USA</country> </shipto> XML Data
  • 33. Global Data Strategy, Ltd. 2017 JSON Metadata 33 • What is JSON? – (JavaScript Object Notation) is a minimal, readable format for structuring data. It is used primarily to transmit data between a server and web application, as an alternative to XML. • What is JSON Metadata? – structure, definition { "id": 127849, “brand": “Super Cooler", "price": 12.50, "tags": [“camping", “sports"] } Example Product in the API Data • Can the ID contain letters? • What is a brand? • Is a price required? • Etc. Context Needed (i.e. Metadata) For example, assume we have a JSON based product catalog. This catalog has a product which has an id, a brand, a price, and an optional set of tags. { "$schema": "http://json-schema.org/draft-04/schema#", "title": "Product", "description": "A retail product from Acme's online catalog", "type": "object", "properties": { "id": { "description": "The unique identifier for a product", "type": "integer" }, “brand": { "description": “The brand name of the product as shown in the online catalogue", "type": "string" }, "price": { "type": "number", }, "tags": { "type": "array", "items": { "type": "string" }, "minItems": 1, } }, "required": ["id", “brand", "price"] } JSON Schema Metadata
  • 34. Organization, Roles & Implementation Considerations Ensuring that metadata is used effectively across the organization
  • 35. Global Data Strategy, Ltd. 2017 Who Uses Metadata? • In addition to sharing metadata between tools and via export, many users across both IT & the business want to view the metadata through reports, portals, etc. 35 Developer If I change this field, what else will be affected? Business Person (e.g. Finance) What’s the definition of “Regional Sales” Auditor How was “Total Sales” calculated? Show me the lineage. Data Architect What is the approved data structure for storing customer data? Data Warehouse Architect What are the source-to- target mappings for the DW? Business Person (e.g. HR) How can I get new staff up- to-speed on our company’s business terminology?
  • 36. Global Data Strategy, Ltd. 2017 Metadata Publication & Reporting – Business Glossary • A Business Glossary is a common way to publish business terms & their definitions. • When sourced from a common repository, these terms are integrated with the wider data landscape. • Most data modeling tools can take the definitions from Logical and/or Conceptual data models and publish them to a Glossary-style format, via web portals or reports. 36 Business Term Abbreviation Definition Data Steward Security Level BFPO Number BFPO Num BFPO Number is for British Forces Postal Office. It can be used in UK and overseas addresses. Accounting Unclassified Interest Int The growth in capital of a monetary investment Finance Unclassified PO Box POB A numbered box in a post office assigned to a person or organization, where mail for them is kept until collected Accounting Unclassified A feedback mechanism is important to gather valuable input & updates from users.
  • 37. Global Data Strategy, Ltd. 2017 Data Models can provide “Just Enough” Metadata Management 37 Metadata Storage Metadata Lifecycle & Versioning Data Lineage Visualization Business Glossary Data Modeling Metadata Discovery & Integration w/ Other Tools Customizable Metamodel Data Modeling Tools (e.g. Erwin, SAP PowerDesigner, Idera ER/Studio) x X x X X x Metadata Repositories (e.g. ASG, Adaptive, DAG) X X X X X X Data Governance Tools (e.g. Collibra, Diaku) x x X x Spreadsheets x x • While data modeling tools are not metadata repositories, nor designed to be, they offer many features shared with these repository solutions: • Metadata storage, Data lineage visualization, Business Glossary, Integration with BI tools, ETL tools, etc. • Metadata repositories have a broader range metadata sources & dedicated metadata management support. • And Data Modeling tools, of course, have the added benefit of doing data modeling!  • And the benefit is that much of the needed metadata is in these data models.
  • 38. Global Data Strategy, Ltd. 2017 Key Components of Metadata Management 38 Metadata Strategy Metadata Capture & Storage Metadata Integration & Publication Metadata Management & Governance Alignment with business goals & strategy Identification of all internal & external metadata sources Identification of all technical metadata sources Metadata roles & responsibilities defined Identification of & feedback from key stakeholders Population/import mechanism for all identified sources Identification of key stakeholders & audiences (internal & external) Metadata standards created Prioritization of key activities aligned with business needs & technical capabilities Identification of existing metadata storage Integration mechanism for key technologies (direct integration, export, etc.) Metadata lifecycle management defined & implemented Prioritization of key data elements/subject areas Definition of enterprise metadata storage strategy Publication mechanism for each audience Metadata quality statistics defined & monitored Communication Plan developed Feedback mechanism for each audience Metadata integrated into operational activities & related data management projects
  • 39. Global Data Strategy, Ltd. 2017 Implementing a Metadata Strategy • A successful metadata strategy requires input from multiple factors. 39 Business Drivers & Motivation Metadata Sources & Technology Metadata Management MaturityStakeholders & Audience Metadata Strategy
  • 40. Global Data Strategy, Ltd. 2017 Stakeholder Feedback • Determine key business issues & drivers through direct feedback. 40 I didn’t know we had any documented data standards Where do I go to get the definition of “default banking standard”? $12m has been spent on projects to clean up the data over the past 2-3 years What are the data structures used in the application? We have 15 customer databases – with many duplications. There is limited ownership or enforcement of common practices and standards across the projects Key subject matter experts are relied upon to review detailed data from various systems to ensure accuracy. I just joined the company and don’t understand all of the acronyms! There was an error in reporting products by customer & region that was noticed by upper management. I need a central, accurate view of all my customers worldwide.
  • 41. Global Data Strategy, Ltd. 2017 Mapping Business Drivers to Metadata Management Capabilities 41 Business Drivers Digital Self Service Increasing Regulatory Pressures Online Community & Social Media Community Building External Drivers Internal Drivers Targeted Marketing 360 View of Customer Brand Reputation Efficient IT Stakeholder Challenges Lack of Business Alignment • Data spend not aligned to Business Plans • Business users not involved with data Integrating Data • Siloed systems • No common view of key information 3 Data Quality Issues • Bad customer info causing Brand damage • Completeness & Accuracy Needed 4 Cost of Data Management • Manual entry increases costs • System redundancy • No reuse or standards 5 No Audit Trails • No lineage of changes • Fines had been levied in past for lack of compliance 6 Big Data Exploitation • Exploiting Unstructured Data • Access to External & Social Data 1 Shows “Heat Map” of Priorities 2 3 4 5 6 Metadata Capability Metadata Strategy Metadata Capture & Storage Metadata Integration & Publication Metadata Management & Governance 1 2 3 4 5 6 2 3 4 5 6 2 3 4 1 2 3 4 5 6
  • 42. Global Data Strategy, Ltd. 2017 Inventory & Usage Mapping • It’s also important to determine which teams are using these technologies to create a “heat map” of usage & priority. 42 Metadata Sources Leadership Sales Finance Marketing Support R&D HR Legal Compliance Relational Databases MySQL X Oracle X X X X X X X X SQL Server X X Sybase X Etc. BI Tools Tableau X X X X X X Qlik X X X Etc. Open Data Data.gov – agricultural data X X X Etc.
  • 43. Global Data Strategy, Ltd. 2017 Metadata Roles & Responsibilities • It’s important to establish formal roles & responsibilities for your metadata effort. • Some may be part-time, and some full-time, but they should be clearly defined and communicated so that staff has understanding of and accountability for their roles. • Executive Sponsor/Champion: Understands & communicates the importance of metadata management across the organization. • Steering Group: As part of a metadata management effort, or part of a larger data governance effort, the steering group prioritizes & sets direction for key activities. • Data Stewards: Responsible for business definitions & rules for key data elements. • Metadata Repository Administrator: Manages the administration, population, and interfaces of a metadata repository. • Metadata Publicist: Establishes reports & publication methods to end users. • Metadata Consumers: Actively use metadata as part of their daily jobs, and are held accountable for using published standards. • Data Modelers • Developers • Business Users • Report Developers • Etc. 43
  • 44. Global Data Strategy, Ltd. 2017 Monitoring Metadata Quality & Metrics • Metadata is a key driver of data quality, and to support this, the metadata itself must be of high quality. • In order to ensure that quality metadata is maintained, it must be actively managed and monitored. Dashboards & Reports can be used to monitor key quality indicators. • Key metadata quality indicators include: • Completeness: e.g. Do definitions exist for all key data elements? • Accuracy: e.g. Are current definitions correct? Do data types accurately represent currently implemented standards? • Currency/ Timeliness: e.g. Are metadata definitions current or outdated? • Consistency: e.g. Are metadata standards defined, published & implemented consistently across the organization? • Accountability: e.g. Are data stewards or owners defined? • Integrity: e.g. Are linkages and relationships established between critical metadata items? • Privacy: e.g. Is any metadata subject to privacy restrictions? • Usability: e.g. Are people actually using this metadata? 44
  • 45. Global Data Strategy, Ltd. 2017 Summary • Metadata is more important than ever • Data models are a rich source of metadata • Metadata exists in a number of sources beyond relational databases • ERP & CRM • Legacy systems • NoSQL • XML & JSON • Etc. • While metadata repositories are valuable, data models & associated functionality can often provide “just enough” metadata management • Business definitions • Technical data structures (relational and beyond) • Data lineage & impact analysis • Visual models • Organizational considerations are critical to achieve success • Understanding business drivers • Defining roles & responsibilities • Monitoring metadata quality & metrics • Have fun! Metadata is for the cool kids.
  • 46. Global Data Strategy, Ltd. 2017 About Global Data Strategy, Ltd • Global Data Strategy is an international information management consulting company that specializes in the alignment of business drivers with data-centric technology. • Our passion is data, and helping organizations enrich their business opportunities through data and information. • Our core values center around providing solutions that are: • Business-Driven: We put the needs of your business first, before we look at any technology solution. • Clear & Relevant: We provide clear explanations using real-world examples. • Customized & Right-Sized: Our implementations are based on the unique needs of your organization’s size, corporate culture, and geography. • High Quality & Technically Precise: We pride ourselves in excellence of execution, with years of technical expertise in the industry. 46 Data-Driven Business Transformation Business Strategy Aligned With Data Strategy Visit www.globaldatastrategy.com for more information
  • 47. Global Data Strategy, Ltd. 2017 Contact Info • Email: donna.burbank@globaldatastrategy.com • Twitter: @donnaburbank @GlobalDataStrat • Website: www.globaldatastrategy.com • Company Linkedin: https://www.linkedin.com/company/global-data-strategy-ltd • Personal Linkedin: https://www.linkedin.com/in/donnaburbank 47
  • 48. Global Data Strategy, Ltd. 2017 White Paper: Emerging Trends in Metadata Management 48 Free Download • Download from www.dataversity.net • Also available on www.globaldatastategy.com
  • 49. Global Data Strategy, Ltd. 2017 DATAVERSITY Training Center • Learn the basics of Metadata Management and practical tips on how to apply metadata management in the real world. This online course hosted by DATAVERSITY provides a series of six courses including: • What is Metadata • The Business Value of Metadata • Sources of Metadata • Metamodels and Metadata Standards • Metadata Architecture, Integration, and Storage • Metadata Strategy and Implementation • Purchase all six courses for $399 or individually at $79 each. Register here • Other courses available on Data Governance & Data Quality 49 Online Training Courses Metadata Management Course Visit: http://training.dataversity.net/lms/
  • 50. Global Data Strategy, Ltd. 2017 Lessons in Data Modeling Series • January 26th How Data Modeling Fits Into an Overall Enterprise Architecture • February 23rd Data Modeling and Business Intelligence • March Conceptual Data Modeling – How to Get the Attention of Business Users • April The Evolving Role of the Data Architect – What does it mean for your Career? • May Data Modeling & Metadata Management • June Self-Service Data Analysis, Data Wrangling, Data Munging, and Data Modeling • July Data Modeling & Metadata for Graph Databases • August Data Modeling & Data Integration • September Data Modeling & MDM • October Agile & Data Modeling – How Can They Work Together? • December Data Modeling, Data Quality & Data Governance 50 This Year’s Line Up
  • 51. Global Data Strategy, Ltd. 2017 Questions? 51 Thoughts? Ideas?