13733827.ppt

Reference &
Master Data
Management
Slides is adapted from :
How to identify the correct Master Data subject areas &
toolingforyourMDMinitiative
CHRISTOPHER BRADLEY

Reference &
Master Data
Management:
Context (DMBoK)
DATA
ARCHITECTURE
MANAGEMENT
DATA
DEVELOPMENT
DATABASE
OPERATIONS
MANAGEMENT
DATA QUALITY
MANAGEMENT
METADATA
MANAGEMENT
DATA
WAREHOUSE
& BUSINESS
INTELLIGENCE
MANAGEMENT
› Enterprise Data Modelling
› Value Chain Analysis
› Related Data Architecture
REFERENCE &
MASTER DATA
MANAGEMENT
› External Codes
› Internal Codes
› Customer Data
› Product Data
› Dimension Management
› Acquisition
› Recovery
› Tuning
› Retention
› Purging
DATA SECURITY
MANAGEMENT
› Standards
› Classifications
› Administration
› Authentication
› Auditing
› Analysis
› Data modelling
› Database Design
› Implementation
DATA
GOVERNANCE
› Strategy
› Organisation & Roles
› Policies & Standards
› Issues
› Valuation
› Architecture
› Implementation
› Training & Support
› Monitoring & Tuning
DOCUMENT & CONTENT
MANAGEMENT
› Acquisition & Storage
› Backup & Recovery
› Content Management
› Retrieval
› Retention
› Architecture
› Integration
› Control
› Delivery
› Specification
› Analysis
› Measurement
› Improvement

Master Data Management, WHY?
› Problems
» Data may need to be rekeyed in each system
» Systems may not be in synch (new records, updated records)
» Duplicate data: are “ABC Ltd” and “ABC Limited” the same thing?
» No single version of the truth
» Reporting / Analysis: difficult to combine data from multiple
systems
› No organisation has just one system (unless the aretiny)
› Details about the same noun are found in multiplesystems,
eg. Customer
The same customers may be
defined in:
• Finance systems
• Marketing systems
• Line of business systems

Reference & Master Data Management

What is Event / Transaction Data?
WHO WHAT WHERE WHEN HOW QUANTITY AMOUNT
Bob Bobson Mars bar Morrisons, Bath 16:00 Monday 3rd January 2011 Cash 1 £0.60
Event data example:
› “Bob bought a Mars bar from Morrisons on Monday 3rd Jan at 4pm and paid usingcash.”
CUSTOMER
CODE
PRODUCT
CODE
VENDOR
CODE
DATE PAYMENT
METHOD
QUANTITY AMOUNT
BB005 CONF101 WMBATH 2011-01-0316:00 CASH 1 £0.60
Terminology
Field (or attribute) = column in a database table
Record = row in a database table

About Event Data
› AKA Transaction data
› Describes an action (a verb):
» E.g. “buy”
› May include measurements
about the action:
» Quantity bought
» Amount paid
Includes information identifying
the nouns that were involved in
the event (the Who / What /
Where / When / How and
maybe even the Why):
» Bob Bobson
» Mars bar
» Morrisons, Bath
» 16:00 Monday 3rd Jan 2011
» Cash
Does not include information
describing the nouns:
» Bob Bobson is male, aged
25 and works for British
Airways
» Monday 3rd Jan 2011 is a
bank holiday
» The address of Morrisons
Bath is: York Place, London
Road, Bath, BA1 6AE.

What is Reference Data…
REFERENCE DATA?
» data that defines the set of permissible
values to be used by other data fields.
» Reference data often is defined
by standards organizations
(such as country codes as
defined in ISO 3166-1).
» Example: country code, province code,
etc.
REFERENCE DATA MANAGEMENT ?
» control over defined domain values (also
known as vocabularies), including control
over standardized terms, code values and
other unique identifiers, business
definitions for each value, business
relationships within and across domain
value lists, and the consistent, shared use
of accurate, timely and relevant reference
data values to classify and categorize data..
[DAMA, the Data Management Association]

» Reference data is data used to classify or categorize other data
» Reference data values should conform to a set of allowable data values called value
domain
» Value domain:
 Internal standard:
 Order Status: New, In Progress, Closed, Cancelled, and so on
 External standard (government or industry standard)
 Two-letter United States Postal Service standard postal code abbreviations for U.S.
states, such as CA for California
» More than one set of reference data value domains may refer to the same conceptual
domain.
 An official name (―California‖).
 A legal name (―State of California‖).
 A standard postal code abbreviation (―CA‖).
 An International Standards Organization (ISO) standard code (―US-CA‖).
 A United States Federal Information Processing Standards (FIPS) code (―06‖).

Reference data sets
with cross-reference
multiple code values
representing the same
things
Reference data sets with
business definitions for each
value

Reference data sets define a
taxonomy of data values, specifying
the hierarchical relationships
between data values using the
Universal Standard Products and
Services Classification (UNSPSC)

What is Master Data…
MASTER DATA?
» Defines and describes the nouns (things)
of the business, e.g. Field, Well, Rig,
Product, Store, Theraputic Area, Adverse
Event, etc.
» Data about the “things” that will
participate in events.
» Provides contextual information about
events / transactions.
» Stored in many systems
» Packaged Systems
» Line of Business Systems
» Spreadsheets
» SharePoint Lists
MASTER DATA MANAGEMENT (MDM)?
» The ongoing reconciliation and maintenance
of master data.
» Control over master data values to enable
consistent, shared, contextual use across
systems, of the most accurate, timely, and
relevant version of truth about essential
business entities.
[DAMA, the Data Management Association]
MASTER DATA MANAGEMENT (MDM)?
» Comprises a set of processes and tools that
consistently defines and manages the non-
transactional data entities of an organisation.
[Wikipedia]

» Common organizational master data includes data about:
(1) Parties include individuals, organizations, and their roles, such as
customers, citizens, patients, vendors, suppliers, business partners,
competitors, employees, students, and so on.
 MDM for customer data is also called Customer Data Integration (CDI)
 Managing business party master data poses unique challenges due to:
 The complexity of roles and relationships played by individuals and organizations.
 Difficulties in unique identification.
 The high number of data sources.
 The business importance and potential impact of the data.
(2) Products, both internal and external.
 It may include information about bill-of-materials component assemblies, part
/ ingredient usage, versions, patch fixes, pricing, discount terms, auxiliary
products, manuals, design documents and images (CAD drawings), recipes
(manufacturing instructions), and standard operating procedures
 Product Lifecycle Management (PLM) focuses on managing the lifecycle of a
product or service from its conception (such as research), through its
development, manufacturing, sale / delivery, service, and disposal

» Common organizational master data includes data about:
(3) Financial structures, such as general ledger accounts, cost centers,
profit centers, and so on.
 Typically, an Enterprise Resource Planning (ERP) system serves as the central
hub for financial master data (chart of accounts), with project details and
transactions created and maintained in one or more spoke applications
(4)Locations, such as addresses.
 Location reference data typically includes geopolitical data, such as countries,
states / provinces, counties, cities / towns, postal codes, geographic regions,
sales territories, and so on.
 Location master data includes business party addresses and business party
location, and geographic positioning coordinates, such as latitude, longitude,
and altitude.

The challenges of MDM
1) to determine the most accurate, golden data values from among
potentially conflicting data values
2) to use the golden values instead of other less accurate data
Master data management systems attempt to determine the golden
data values and then make that data available wherever needed

Primary MDM focus areas
» Identification of duplicate records within and across data sources to
build and maintain global IDs and associated cross-references to
enable information integration.
» Reconciliation across data sources and providing the “golden
record” or the best version of the truth. These consolidated records
provide a merged view of the information across systems and seek
to address name and address inconsistencies.
» Provision of access to the golden data across applications, either
through direct reads, or by replication feeds to OLTP and DW / BI
databases.

Master Data
Management
Activities

(1) Understand Reference and Master Data
Integration Needs
» Analyzing the root causes of a data quality problem
Data architecture?
Human habit?
etc
» Challenges for understanding single application vs across
applications
 It is harder to understand the needs of entire enterprise

(2) Identify Reference and Master Data Sources
and Contributors
» Identify the original and interim source databases, files, applications,
organizations, and even the individual roles that create and maintain
the data.
» Understand both the up-stream sources and the down-stream
needs to capture quality data at its source.

(3) Define and Maintain the Data integration
Architecture
» Data integration architecture controls the shared access, replication,
and flow of data to ensure data quality and consistency, particularly
for reference and master data.

RDM “Hub and Spoke” Architecture
» The database of record serves as a reference data “hub” supplying reference data to other
“spoke” applications and databases
» Data updates approaches:
Push data through a subscribe-and-publish approach in near-real time (asynchronous
updates) for data updates
Pull data for data updates

General MDM “Hub and Spoke” Architecture
» The database of record serves as a master data “hub” supplying master data to other
“spoke” applications and databases
» Data updates approaches:
Push data through a subscribe-and-publish approach in near-real time (asynchronous
updates) for data updates
Pull data for data updates

CONSUMING SYSTEMS
SYSTEMS OF ENTRY
SYSTEM
SYSTEM
SYSTEM
SYSTEM
SYSTEM
SYSTEM
MASTER DATA ENVIRONMENT
DATA
SOURCING
SYSTEM OF
RECORD
DATA
DISTRIBUTION
META-DATA
CATALOGUE
USER INTERFACE
DATA STEWARDS
DATA OWNERS
DATA QUALITY
Standardise, De-duplicate,
Merge, Enrich, etc
“Best version”
To highlight issues and
enable them to be
fixed
To store data models and
meta data
Master Data Environment

Master Data Environment: methods of moving Data
CONSUMING SYSTEMS
SYSTEMS OF ENTRY
SYSTEM
SYSTEM
SYSTEM
SYSTEM
SYSTEM
SYSTEM
SYSTEM OF
RECORD
META-DATA
CATALOGUE
USER INTERFACE
DATA STEWARDS
DATA OWNERS
DATA QUALITY
ACTIVE PUSH
Webservices
BATCH SCHEDULE
ETL job
EII/DV cache refresh
PULL ON DEMAND
Webservices
SQL queries
ACTIVE PUSH
Webservices
BATCH SCHEDULE
E
T
LETL job
EII/DV
PULL ON DEMAND
Webservices
SQL queries

Three standard “Hub” architectures
1.Repository
2.Registry
3.Hybrid
*A key difference is the
number of fields that are
stored centrally

Example: Customer
Customer
code
First
name
Last
name
Date of birth Preferred
deliveryaddress
line1
Preferred
deliveryaddress
postcode
Credit
rating
Occupation Car
BB005 Bob Bobson 1985-12-25 Royal Crescent BA17LA A Information
Architect
Audi R8
IDENTIFIERS
CORE FIELDS
ALL FIELDS
ALL FIELDS
REPOSITORY
CORE
FIELDS
HYBRID
IDENTIFIERS
REGISTRY

Master Data Environment:
Repository Architecture
CONSUMING SYSTEMS
SYSTEMS OF ENTRY
SYSTEM
SYSTEM
SYSTEM
SYSTEM
SYSTEM
SYSTEM
DATA
DISTRIBUTION
META-DATA
CATALOGUE
USER INTERFACE
DATA STEWARDS
DATA OWNERS
DATA QUALITY
DATA
ACTIVE PUSH
Webservices
(CUD)
PULL ON DEMAND
Webservices
(R)
S
R
Y
ES
PT
O
E
SM
ITO
O
RF
Y
RECORD
REPOSITORY
› Repository serves as the single source of the master data.
› Repository contains the only version of the master data.
› All applications use the data in the repository via services.
› No latency orsynchronisation.
DATA

Registry Architecture
CONSUMING SYSTEMS
SYSTEMS OF ENTRY
SYSTEM
SYSTEM
SYSTEM
META-DATA
CATALOGUE
USER INTERFACE
DATA STEWARDS
DATA OWNERS
DATA QUALITY
BATCH SCHEDULE
ETL job
EII/DV
BATCH SCHEDULE
ETL job
PULL ON DEMAND
Webservices
SQL queries
REGISTRY
› Registry only contains identifying attributes.
› Persistent values are still stored in the source systems.
› Registry serves as an index to source data locations.
› Some latency due to caching.
VIRTUAL REPOSITORY
› A virtual database built using Enterprise
Information Integration (EII) technology.
› Survivorship rules are applied to return
the master record.
REGISTRY
SYSTEM
R
VIRTUAL
REPOSITORY
SYSTEM
?
SYSTEM
?
SYSTEM
?

CONSUMING SYSTEMS
SYSTEMS OF ENTRY
SYSTEM
SYSTEM
SYSTEM
META-DATA
CATALOGUE
USER INTERFACE
DATA STEWARDS
DATA OWNERS
DATA QUALITY
ACTIVE PUSH
Webservices
BATCH SCHEDULE
ETL job
EII/DV
PULL ON DEMAND
Webservices
SQL queries
ACTIVE PUSH
Webservices
BATCH SCHEDULE
ETL job
DISTRIBUTION
PULL ON DEMAND
Webservices
SQL queries
Hybrid Architecture
HYBRID
› Repository contains identifying information and core information.
› Application-specific data is retained only in the application database.
› Applications still manage the full set of data.
› Core information is published back to source systems.
› 2-way synchronisation – normally some latency.
S
R
Y
ES
PT
O
E
SM
ITO
O
RF
Y
RECORD
SYSTEM
?
SYSTEM
?
SYSTEM
?

Select your MDM
Architecture and
Toolset carefully
A hub is not the only way...

MDM… A Hub is not the only way
ERP CRM LEGACY
MIDDLEWARE SYNCHRONISATION LAYER
ERP CRM LEGACY
EXTRACT-TRANSFORM-LOAD
SYNCHRONISATION LAYER
OPERATIONAL
DATA-STORE
E.G.CUSTOMER
MASTER
Hub Based Master
› Operational Hub structure that overlaps operational &
analytical environments
› Supports concept of an Enterprise Data Warehouse
› Multiple systems acting as data providers
› Appropriate for low data latency & velocity operations
› Requires careful data quality management
MASTER DATACONTRIBUTORS
Synchronized Master
› Multiple operational systems acting as master data
contributors
› Real-time information availability
› Well suited to enterprises where data is stored across
multiple source systems
› Well suited to low data velocity operations

ERP CRM LEGACY
ERP CRM LEGACY
INDUSTRY
SPECIFIC
MASTER
SINGLE DATA
SOURCE
Application Specific Master
› One operational system as master data provider
› Well suited to enterprises where data is primarily stored in
a single source system
› Support from many enterprise vendors
Master Overlay
› Stand-alone, application-neutral master data
› Industry-specific data model
› Well suited to vertical industries in aligning front-office
& back-office systems in real time

Non
SAP
CRM
LEGACY
SAP 2
SAP 3
SAP 1
DW
DSL
MESSAGING
DV (FEDERATED) MASTER DATA
MASTER
DATA
ORACLE
SAP
SIEBEL
…..
….. …..
….. …..
…..
PACKAGES DATABASES FILES XML
Data Virtualization (Federated) Master Data
› Virtual MDM hub created via DV layer
› DV layer informed from Logical Data Model
› Powerful data integration to access multiple disparate systems
› Rapid time to solution
› Outstanding prototyping, proof of value approach
› Compliments other Data Integration approaches
REPORTS
Real Time Data Movement
› Data is moved between the various systems using a
messaging integration hub or ESB
› Data movement is done in real time
› Movement of data can be one or two way as required

Data Integration Service Architecture
Data integration service includes:
» Change request processing, including
review and approval.
» Data quality checks on externally acquired
reference and master data.
» Consistent application of data quality rules
and matching rules.
» Consistent patterns of processing.
» Consistent meta-data about mappings,
transformations, programs and jobs.
» Consistent audit, error resolution and
performance monitoring data.
» Consistent approaches to replicating data
(including ―subscribe and publish‖).

(4) Implement Reference and Master Data
Management Solutions
»The implementation needs several related projects and
phases, guided by their architecture, business priorities,
and an implementation program roadmap
»Vendor solution
customer data integration (CDI)
product data integration (PDI)
some other master data subject area, such as other parties,
locations, and financial structures

(5) Define and Maintain Match Rules
» Data from different sources may have different format and standard
 Need rules for matching, merging, and linking of data from multiple systems about the same
person
 Matching is particularly challenging for data about people
» Purpose:
 to remove redundancy;
 to improve data quality; and
 to provide information that is more comprehensive
» Support by data cleansing tools and MDM applications for inference engines to
match data
» How much the confidence level of your matching rule?
 If two records share the same last name, first name, birth date, and social security number,
but the street address differs, is it safe to assume they are about the same person who has
changed their mailing address?
 If two records share the same social security number, street address, and first name, but the
last name differs, is it safe to assume they are about the same person who has changed their
last name? Would the likelihood be increased or decreased based on gender and age?
 How do these examples change if the social security number is unknown for one record?
What other identifiers are useful to determine the likelihood of a match? How much
confidence is required for the organization to assert a match?

(5) Define and Maintain Match Rules
» Three primary scenarios to match rules:
 Duplicate identification match rules focus on a specific set of fields that uniquely
identify an entity and identify merge opportunities without taking automatic action.
Business data stewards can review these occurrences and decide to take action on a
case-by-case basis.
 Match-merge rules match records and merge the data from these records into a
single, unified, reconciled, and comprehensive record. If the rules apply across data
sources, create a single unique and comprehensive record in each database.
Minimally, use trusted data from one database to supplement data in other
databases, replacing missing values or values thought to be inaccurate.
 Match-link rules identify and cross-reference records that appear to relate to a
master record without updating the content of the cross-referenced record. Match-
link rules are easier to implement and much easier to
» Match-merge rules are complex due to the need to identify so
many possible circumstances, with different levels of confidence
and trust placed on data values in different fields from different
sources

(6) Establish Golden Records
» Vocabulary Management and Reference Data
 A vocabulary is a collection of terms / concepts and their relationships
 Vocabulary management is defining, sourcing, importing, and maintaining a vocabulary and
its associated reference data
 ANSI / NISO Z39.19-2005 Guidelines for the Construction, Format, and Management of
Monolingual Controlled Vocabularies
 Vocabulary management requires data governance, enabling data stewards to assess
stakeholder needs, and the impacts of proposed changes, before making collaborative and
formally approved decisions
» Defining Golden Master Data Values
 Golden data values are the data values thought to be the most accurate, current, and
relevant for shared, consistent use across applications.
 Organizations determine golden values by analyzing data quality, applying data quality rules
and matching rules, and incorporating data quality controls into the applications that
acquire, create, and update data
 Applications can enforce data quality rules, including:
 Incorporating simple edit checks against referenced data and key business rules.
 Ensuring new records, such as addresses, that are being entered do not already exist in the system
through applying data standardization and search-before-create automation.
 Creating prompts for the user if data does not meet accuracy (this address does not exist)
expectations, while providing a way to submit exceptions that can be audited in the future.

(6) Establish Golden Records
» Defining Golden Master Data Values (cont’)
 Assess data quality through a combination of data profiling activities and verification against
adherence to business rules
» Define and Maintain Hierarchies and Affiliations
 Vocabularies usually include hierarchical relationships between the terms.
 general-to-specific classifications (―is a kind of‖ relationships)
 whole-part assemblies (―is a part of‖ relationships)
 There may also be non-hierarchical relationships between terms in vocabularies
 Affiliation management is the establishment and maintenance of relationships between
master data records
 Examples: ownership affiliations (such as Company X is a subsidiary of Company Y, a
parent-child relationship)

(7) Define and Maintain Hierarchies and Affiliations
» Vocabularies hierarchy (relationships between the terms in the
vocabularies)
general-to-specific classifications (―is a kind of‖ relationships)
whole-part assemblies (―is a part of‖ relationships)
» There may also be non-hierarchical relationships between terms in
vocabularies
» Affiliation management is the establishment and maintenance of
relationships between master data records
Examples:
ownership affiliations (such as Company X is a subsidiary of
Company Y, a parent-child relationship)
associations (such as Person XYZ works at Company X)

(8) Plan and Implement Integration of New Data
Sources
» Integrating new reference data sources involves (among other tasks):
 Receiving and responding to new data acquisition requests from different groups.
 Performing data quality assessment services using data cleansing and data profiling tools.
 Assessing data integration complexity and cost.
 Piloting the acquisition of data and its impact on match rules.
 Determining who will be responsible for data quality.
 Finalizing data quality metrics.

(9) Replicate and Distribute Reference and
Master Data
» Reference and master data may be read directly from a database of
record, or may be replicated from the database of record to other
application databases for transaction processing, and data
warehouses for business intelligence.
» Replication mechanism ensures referential integrity
only valid reference data codes and master data identifiers are used
as foreign key values in other tables
» Data integration procedures must ensure timely replication and
distribution of reference and master data to these application
databases

(10) Manage Changes to Reference and Master Data
» Assign specific individuals as business data steward
 to create, update, and retire reference data values, and to a lesser extent, in some
circumstances, master data values
» Business data stewards work with data professionals to ensure the
highest quality reference and master data
» Common change request process:
1. Create and receive a change request.
2. Identify the related stakeholders and understand their interest
3. Identify and evaluate the impacts of the proposed change.
4. Decide to accept or reject the change, or recommend a decision to management
or governance.
5. Review and approve or deny the recommendation, if needed.
6. Communicate the decision to stakeholders prior to making the change.
7. Update the data.
8. Inform stakeholders the change has been made.
» Any changes to reference data that was replicated elsewhere must also be
applied to the replicated data
» ll

(10) Manage Changes to Reference and Master
Data
» How about if terms and codes are retired?
Code tables require effective date and expiration date columns,
Application logic must refer to the currently valid codes when
establishing new foreign key relationships.
» Carefully assess the impact of reference data changes.

Guiding Principles
» Shared reference and master data belongs to the organization, not to a
particular application or department.
» Reference and master data management is an on-going data quality
improvement program; its goals cannot be achieved by one project
alone.
» Business data stewards are the authorities accountable for controlling
reference data values. Business data stewards work with data
professionals to improve the quality of reference and master data.
» Golden data values represent the organization‘s best efforts at
determining the most accurate, current, and relevant data values for
contextual use. New data may prove earlier assumptions to be false.
Therefore, apply matching rules with caution, and ensure that any
changes that are made are reversible.
» Replicate master data values only from the database of record.
» Request, communicate, and, in some cases, approve of changes to
reference data values before implementation.

13733827.ppt

More Related Content

Similar to 13733827.ppt (20)

More from MichelleSaver (6)

Recently uploaded (20)

13733827.ppt