SlideShare a Scribd company logo
Document:
                Oracle Data Integrator 11g (11.1.1.3)
                 Working With Change Data Capture
Description:
(This document describes the basic concepts of the ODI CDC.
Changed Data Capture (CDC) allows Oracle Data Integrator to track
changes in source data caused by other applications.)


History:
      Version             Description Change           Author              Publish Date
         0.1                   Initial Draft        Gourav Atalkar          7-Apr-2011

         0.1                      Review             Amit Sharma            7-Apr-2011




.




    Business Intelligence Solution Providers   http://learnhyperion.wordpress.com        1
Changed Data Capture
Purpose of CDC is to enable applications to process changed data only. CDC enables ODI to
track changes in source data caused by other applications. When running integration
interfaces, ODI can avoid processing unchanged data in the flow. Loads will process only
changes since the last load. The volume of data to be processed is dramatically reduced.

Reducing the source data flow to only changed data is useful in many contexts, such as data
synchronization and replication. It is essential when setting up an event-oriented
architecture for integration. In such architecture, applications make changes in the data
("Customer Deletion", "New Purchase Order") during a business process. These changes are
captured by Oracle Data Integrator and transformed into events that are propagated
throughout the information system.

CDC Techniques
 Trigger based: ODI will create and maintain triggers to keep track of the changes.

 Logs based: ODI retrieves changes from the database logs (Oracle, AS/400).

 Time stamp based: Processes written with ODI can filter the data by comparing the
   time stamp value with the last load time (cannot process deletes)

 Sequence number: If the records are numbered in sequence, ODI can filter the data
   based on the last value loaded (cannot process updates and deletes).

Changed Data Capture is performed by journalizing models. Journalizing a model consists
of setting up the infrastructure to capture the changes (inserts, updates and deletes) made
to the records of this model's datastores.

Oracle Data Integrator supports two journalizing modes:

   •     Simple Journalizing tracks changes in individual datastores in a model.
   •     Consistent Set Journalizing tracks changes to a group of the model's datastores,
         taking into account the referential integrity between these datastores. The group of
         datastores journalized in this mode is called a Consistent Set.

The Journalizing Components
The journalizing components are:




       Business Intelligence Solution Providers   http://learnhyperion.wordpress.com      2
•     Journals: Where changes are recorded. Journals only contain references to the
         changed records along with the type of changes (insert/update, delete).
   •     Capture processes: Journalizing captures the changes in the source datastores
         either by creating triggers on the data tables, or by using database-specific programs
         to retrieve log data from data server log files.
   •     Subscribers: CDC uses a publish/subscribe model. Subscribers are entities
         (applications, integration processes, etc) that use the changes tracked on a datastore
         or on a consistent set. They subscribe to a model's CDC to have the changes tracked
         for them. Changes are captured only if there is at least one subscriber to the changes.
         When all subscribers have consumed the captured changes, these changes are
         discarded from the journals.
   •     Journalizing views: Provide access to the changes and the changed data captured.
         They are used by the user to view the changes captured, and by integration
         processes to retrieve the changed data.

These components are implemented in the journalizing infrastructure

Setting up Journalizing
This is the basic process for setting up CDC on an Oracle Data Integrator data model. Each
of these steps is described in more detail below.

   1.    Set the CDC parameters
   2.    Add the datastores to the CDC
   3.    For consistent set journalizing, arrange the datastores in order
   4.    Add subscribers
   5.    Start the journals

Journalizing Tools

Oracle Data Integrator provides a set of tools that can be used in journalizing to refresh
information on the captured changes or trigger other processes:

   •     SnpsWaitForData waits for a number of rows in a table or a set of tables.
   •     SnpsWaitForLogData waits for a certain number of modifications to occur on a
         journalized table or a list of journalized tables. This tool calls
         SnpsRefreshJournalCount to perform the count of new changes captured.
   •     SnpsWaitForTable waits for a table to be created and populated with a pre-
         determined number of rows.
   •     SnpsRetrieveJournalData retrieves the journalized events for a given table list or
         CDC set for a specified journalizing subscriber. Calling this tool is required if using
         Database-Specific Processes to load journalizing tables. This tool needs to be used
         with specific knowledge modules. See the knowledge module description for more
         information.
   •     SnpsRefreshJournalCount refreshes the number of rows to consume for a given
         table list or CDC set for a specified journalizing subscriber.
       Business Intelligence Solution Providers   http://learnhyperion.wordpress.com         3
Implementing Changed Data Capture
Step:1) Import the appropriate JKM in the project.

Click the Projects tab. Expand the Procedure-Demo > Knowledge Modules node, right-click
Journalization (JKM), and select Import Knowledge Modules.




    Business Intelligence Solution Providers   http://learnhyperion.wordpress.com    4
Step:2) In the Models tab, create a new model named Oracle_relational_01. For Technology,
enter: Oracle. Select the logical schema Sales_Order. Click the Reverse Engineer tab and set
Context to development. Verify the setting, as shown in the following screen. Click the
Journalizing tab.




Step: 3) In the Knowledge Module menu, select JKM Oracle Simple. Procedure-Demo, as
shown in the following screen. Click the Save to save your model and then close the tab.




    Business Intelligence Solution Providers   http://learnhyperion.wordpress.com          5
Step: 4) Reverse-engineer the model Oracle_Relational_01. Expand this model and verify its
structure as follows.




Step: 5) Set up the CDC Infrastructure. You will start the CDC on the EMPLOYEE table in the
Oracle_Relational_01 model.

To add the table to CDC, expand the Oracle_Relational_01 model, right-click the EMPLOYEE
table, and select Change Data Capture > Add to CDC. Click Yes to confirm.




    Business Intelligence Solution Providers   http://learnhyperion.wordpress.com       6
Step: 6) Click the Refresh icon. The small yellow clock icon is added to the table.




Step: 7) Right-click the EMPLOYEE table again and select Changed Data Capture > Start
Journal.




     Business Intelligence Solution Providers   http://learnhyperion.wordpress.com    7
Step: 8) you use the default subscriber SUNOPSIS. For that reason, you do not have to add
another subscriber. Click OK to confirm that your subscriber is SUNOPSIS. In the
Information window, click OK again. Wait several seconds, then click Refresh and verify
that the tiny clock icon at the EMPLOYEE table is green now. This means that your journal
has started properly




Step: 9) Click the ODI Operator icon to open the Operator. Click Refresh. Select All
Executions and verify that the EMPLOYEE session executed successfully.




Step: 10) View the data and the changed data.

In the Designer window, open the Models tab. Right-click the EMPLOYEE datastore and
select Data.




    Business Intelligence Solution Providers   http://learnhyperion.wordpress.com     8
Step: 11) Select the row with Employee_Key = 10. Change the value of the NAME2 column
to “Symond”. Similarly, select the row with Employee_Key = 15, and then change the value
to “jacob”. Save your changes and close the tab.




Step: 12) Right-click the table again and select View Data. Scroll down, and verify that the
rows are modified. Close the tab.




    Business Intelligence Solution Providers   http://learnhyperion.wordpress.com        9
To verify that your changed data is captured, right-click EMPLOYEE, and select Change Data
Capture > Journal Data. Find the captured changed records in the journal data. Close the
tab.




    Business Intelligence Solution Providers   http://learnhyperion.wordpress.com      10

More Related Content

PDF
Oracle database monitors and tools
PDF
Archive data in sap
DOCX
DWMH Projects
PDF
Java Application Portfolio(Production control system)
PPTX
Sql server lesson13
PPT
Sap business intelligence 4.0 report basic
PPTX
Sql server lesson12
Oracle database monitors and tools
Archive data in sap
DWMH Projects
Java Application Portfolio(Production control system)
Sql server lesson13
Sap business intelligence 4.0 report basic
Sql server lesson12

What's hot (20)

DOCX
OBIEE publisher with Report creation - Tutorial
PDF
Designing Database Solutions for Microsoft SQL Server 2012 2012 Microsoft 70-...
DOCX
12 1-man-operation center-ug(2)
PPT
CDC
PPT
Less18 Patching oracle dba 11g
PDF
Catalogic DPX: Dashboard Reporting with Microsoft Power BI
PPTX
PDF
Day 8.1 system_admin_tasks
PPT
Oracle RI ETL process overview.
DOCX
Sql server 2008 r2 performance and scale
DOC
Data Warehouse (ETL) testing process
DOCX
pega training whatsup@8142976573
DOC
SAP data archiving
PDF
ALMA ANALYTICS internal training document IRAM - University of Western Australia
PPTX
Capture Change and Apply It!
PDF
Optimized XT Reader Functionality in Rational System Architect
PDF
Analysing data analytics use cases to understand big data platform
PPTX
Bilir's Business Intelligence Portfolio SSIS Project
PDF
Data warehousing features in oracle
DOC
To Study E T L ( Extract, Transform, Load) Tools Specially S Q L Server I...
OBIEE publisher with Report creation - Tutorial
Designing Database Solutions for Microsoft SQL Server 2012 2012 Microsoft 70-...
12 1-man-operation center-ug(2)
CDC
Less18 Patching oracle dba 11g
Catalogic DPX: Dashboard Reporting with Microsoft Power BI
Day 8.1 system_admin_tasks
Oracle RI ETL process overview.
Sql server 2008 r2 performance and scale
Data Warehouse (ETL) testing process
pega training whatsup@8142976573
SAP data archiving
ALMA ANALYTICS internal training document IRAM - University of Western Australia
Capture Change and Apply It!
Optimized XT Reader Functionality in Rational System Architect
Analysing data analytics use cases to understand big data platform
Bilir's Business Intelligence Portfolio SSIS Project
Data warehousing features in oracle
To Study E T L ( Extract, Transform, Load) Tools Specially S Q L Server I...
Ad

Similar to Oracle data capture c dc (20)

PPTX
Lecture 5: Extraction Transformation and loading.pptx
PPTX
ETL Process
PPTX
Process management seminar
PDF
Change data capture the journey to real time bi
PPTX
Insight on MongoDB Change Stream - Abhishek.D, Mydbops Team
PPT
Technical Presentation - TimeWIzard
PDF
Managing Changing Data: Part 2
PPTX
Data warehouse testing
PDF
Business Analytics System
PDF
Data Warehouses & Deployment By Ankita dubey
PPTX
Sql query analyzer & maintenance
PDF
Sql server 2016 new features
PDF
Sql server 2016 new features
DOCX
Oracle advanced supply chain planning implementation and user
PDF
MetaSuite productfolder- ETL-Tool für große Datenmengen
PPTX
Tableau Basic Questions
PDF
best-practices-for-realtime-data-wa-132882.pdf
PPTX
Warehouse Planning and Implementation
PPT
introduction to datawarehouse
PDF
58750024 datastage-student-guide
Lecture 5: Extraction Transformation and loading.pptx
ETL Process
Process management seminar
Change data capture the journey to real time bi
Insight on MongoDB Change Stream - Abhishek.D, Mydbops Team
Technical Presentation - TimeWIzard
Managing Changing Data: Part 2
Data warehouse testing
Business Analytics System
Data Warehouses & Deployment By Ankita dubey
Sql query analyzer & maintenance
Sql server 2016 new features
Sql server 2016 new features
Oracle advanced supply chain planning implementation and user
MetaSuite productfolder- ETL-Tool für große Datenmengen
Tableau Basic Questions
best-practices-for-realtime-data-wa-132882.pdf
Warehouse Planning and Implementation
introduction to datawarehouse
58750024 datastage-student-guide
Ad

More from Amit Sharma (20)

PDF
Oracle enteprise pbcs drivers and assumptions
PDF
Oracle EPBCS Driver
PDF
Oracle Sales Quotation Planning
PDF
Oracle strategic workforce planning cloud hcmswp converted
PDF
Basics of fdmee
PDF
FDMEE script examples
PDF
Oracle PBCS creating standard application
PPT
Hfm rule custom consolidation
PPT
Hfm calculating RoA
PPT
Adding metadata using smartview
PPT
Hyperion planning weekly distribution
PPT
Hyperion planning scheduling data import
PPT
Hyperion planning new features
TXT
Microsoft dynamics crm videos
PDF
Oracle apex-hands-on-guide lab#1
PDF
Oracle apex hands on lab#2
PDF
Security and-data-access-document
PDF
Sales force managing-data
PDF
Salesforce interview-preparation-toolkit-formula-and-validation-rules-in-sale...
PDF
Sales force certification-lab-ii
Oracle enteprise pbcs drivers and assumptions
Oracle EPBCS Driver
Oracle Sales Quotation Planning
Oracle strategic workforce planning cloud hcmswp converted
Basics of fdmee
FDMEE script examples
Oracle PBCS creating standard application
Hfm rule custom consolidation
Hfm calculating RoA
Adding metadata using smartview
Hyperion planning weekly distribution
Hyperion planning scheduling data import
Hyperion planning new features
Microsoft dynamics crm videos
Oracle apex-hands-on-guide lab#1
Oracle apex hands on lab#2
Security and-data-access-document
Sales force managing-data
Salesforce interview-preparation-toolkit-formula-and-validation-rules-in-sale...
Sales force certification-lab-ii

Oracle data capture c dc

  • 1. Document: Oracle Data Integrator 11g (11.1.1.3) Working With Change Data Capture Description: (This document describes the basic concepts of the ODI CDC. Changed Data Capture (CDC) allows Oracle Data Integrator to track changes in source data caused by other applications.) History: Version Description Change Author Publish Date 0.1 Initial Draft Gourav Atalkar 7-Apr-2011 0.1 Review Amit Sharma 7-Apr-2011 . Business Intelligence Solution Providers http://learnhyperion.wordpress.com 1
  • 2. Changed Data Capture Purpose of CDC is to enable applications to process changed data only. CDC enables ODI to track changes in source data caused by other applications. When running integration interfaces, ODI can avoid processing unchanged data in the flow. Loads will process only changes since the last load. The volume of data to be processed is dramatically reduced. Reducing the source data flow to only changed data is useful in many contexts, such as data synchronization and replication. It is essential when setting up an event-oriented architecture for integration. In such architecture, applications make changes in the data ("Customer Deletion", "New Purchase Order") during a business process. These changes are captured by Oracle Data Integrator and transformed into events that are propagated throughout the information system. CDC Techniques  Trigger based: ODI will create and maintain triggers to keep track of the changes.  Logs based: ODI retrieves changes from the database logs (Oracle, AS/400).  Time stamp based: Processes written with ODI can filter the data by comparing the time stamp value with the last load time (cannot process deletes)  Sequence number: If the records are numbered in sequence, ODI can filter the data based on the last value loaded (cannot process updates and deletes). Changed Data Capture is performed by journalizing models. Journalizing a model consists of setting up the infrastructure to capture the changes (inserts, updates and deletes) made to the records of this model's datastores. Oracle Data Integrator supports two journalizing modes: • Simple Journalizing tracks changes in individual datastores in a model. • Consistent Set Journalizing tracks changes to a group of the model's datastores, taking into account the referential integrity between these datastores. The group of datastores journalized in this mode is called a Consistent Set. The Journalizing Components The journalizing components are: Business Intelligence Solution Providers http://learnhyperion.wordpress.com 2
  • 3. Journals: Where changes are recorded. Journals only contain references to the changed records along with the type of changes (insert/update, delete). • Capture processes: Journalizing captures the changes in the source datastores either by creating triggers on the data tables, or by using database-specific programs to retrieve log data from data server log files. • Subscribers: CDC uses a publish/subscribe model. Subscribers are entities (applications, integration processes, etc) that use the changes tracked on a datastore or on a consistent set. They subscribe to a model's CDC to have the changes tracked for them. Changes are captured only if there is at least one subscriber to the changes. When all subscribers have consumed the captured changes, these changes are discarded from the journals. • Journalizing views: Provide access to the changes and the changed data captured. They are used by the user to view the changes captured, and by integration processes to retrieve the changed data. These components are implemented in the journalizing infrastructure Setting up Journalizing This is the basic process for setting up CDC on an Oracle Data Integrator data model. Each of these steps is described in more detail below. 1. Set the CDC parameters 2. Add the datastores to the CDC 3. For consistent set journalizing, arrange the datastores in order 4. Add subscribers 5. Start the journals Journalizing Tools Oracle Data Integrator provides a set of tools that can be used in journalizing to refresh information on the captured changes or trigger other processes: • SnpsWaitForData waits for a number of rows in a table or a set of tables. • SnpsWaitForLogData waits for a certain number of modifications to occur on a journalized table or a list of journalized tables. This tool calls SnpsRefreshJournalCount to perform the count of new changes captured. • SnpsWaitForTable waits for a table to be created and populated with a pre- determined number of rows. • SnpsRetrieveJournalData retrieves the journalized events for a given table list or CDC set for a specified journalizing subscriber. Calling this tool is required if using Database-Specific Processes to load journalizing tables. This tool needs to be used with specific knowledge modules. See the knowledge module description for more information. • SnpsRefreshJournalCount refreshes the number of rows to consume for a given table list or CDC set for a specified journalizing subscriber. Business Intelligence Solution Providers http://learnhyperion.wordpress.com 3
  • 4. Implementing Changed Data Capture Step:1) Import the appropriate JKM in the project. Click the Projects tab. Expand the Procedure-Demo > Knowledge Modules node, right-click Journalization (JKM), and select Import Knowledge Modules. Business Intelligence Solution Providers http://learnhyperion.wordpress.com 4
  • 5. Step:2) In the Models tab, create a new model named Oracle_relational_01. For Technology, enter: Oracle. Select the logical schema Sales_Order. Click the Reverse Engineer tab and set Context to development. Verify the setting, as shown in the following screen. Click the Journalizing tab. Step: 3) In the Knowledge Module menu, select JKM Oracle Simple. Procedure-Demo, as shown in the following screen. Click the Save to save your model and then close the tab. Business Intelligence Solution Providers http://learnhyperion.wordpress.com 5
  • 6. Step: 4) Reverse-engineer the model Oracle_Relational_01. Expand this model and verify its structure as follows. Step: 5) Set up the CDC Infrastructure. You will start the CDC on the EMPLOYEE table in the Oracle_Relational_01 model. To add the table to CDC, expand the Oracle_Relational_01 model, right-click the EMPLOYEE table, and select Change Data Capture > Add to CDC. Click Yes to confirm. Business Intelligence Solution Providers http://learnhyperion.wordpress.com 6
  • 7. Step: 6) Click the Refresh icon. The small yellow clock icon is added to the table. Step: 7) Right-click the EMPLOYEE table again and select Changed Data Capture > Start Journal. Business Intelligence Solution Providers http://learnhyperion.wordpress.com 7
  • 8. Step: 8) you use the default subscriber SUNOPSIS. For that reason, you do not have to add another subscriber. Click OK to confirm that your subscriber is SUNOPSIS. In the Information window, click OK again. Wait several seconds, then click Refresh and verify that the tiny clock icon at the EMPLOYEE table is green now. This means that your journal has started properly Step: 9) Click the ODI Operator icon to open the Operator. Click Refresh. Select All Executions and verify that the EMPLOYEE session executed successfully. Step: 10) View the data and the changed data. In the Designer window, open the Models tab. Right-click the EMPLOYEE datastore and select Data. Business Intelligence Solution Providers http://learnhyperion.wordpress.com 8
  • 9. Step: 11) Select the row with Employee_Key = 10. Change the value of the NAME2 column to “Symond”. Similarly, select the row with Employee_Key = 15, and then change the value to “jacob”. Save your changes and close the tab. Step: 12) Right-click the table again and select View Data. Scroll down, and verify that the rows are modified. Close the tab. Business Intelligence Solution Providers http://learnhyperion.wordpress.com 9
  • 10. To verify that your changed data is captured, right-click EMPLOYEE, and select Change Data Capture > Journal Data. Find the captured changed records in the journal data. Close the tab. Business Intelligence Solution Providers http://learnhyperion.wordpress.com 10