Talk IT_ Oracle_김태완_110831
0 likes545 views
This document discusses maximizing returns from a data warehouse. It covers the need for real-time data integration to power business intelligence and enable timely, trusted decisions. It outlines challenges with traditional batch-based approaches and how Oracle's data integration solutions address these through products that enable real-time data capture and delivery, bulk data movement, and data quality profiling to build an enterprise data warehouse.
Talk IT_ Oracle_김태완_110831
- 1. <Insert Picture Here> Maximizing Return for your Data Warehouse
- 2. Agenda • Top Business Imperatives and Data Requirements to Succeed <Insert Picture Here> • Data Warehouse Basics and Challenges • The Purpose of Data Warehouse • Why Real-Time Data Warehouse for BI? • Real-Time Data Integration Considerations • Traditional Vs Real-time Data Warehouse • Data Quality & Profiling • Oracle Data Integration Solution • Customer Case Studies 2
- 3. Top Business Imperatives and Data Requirements to Succeed Access to Timely, Trusted, and Consistent data Operational & Analytical Business Applications Risk CRM / Direct Improve Management Marketing Mergers & Decisions & Acquisitions Regulatory Compliance Real-Time Data Quality Data Integration Data DataMart / MDM / Migration & BI DWH / BI Consolidation BI IT Projects 3
- 4. Why Real-Time DataWarehouse for BI? 1. Business Driver: Real-Time= Relevant and actionable To compete more effectively information by using better insights in day to day operations and Improved insights for strategic decisions operational decision making 2. Technical Driver Better customer service To move away from batch and cost savings via ETL jobs and eliminate operational efficiencies the impact it has on the IT infrastructure Increased profitability, customer retention, and competitive advantage 4
- 5. Integration Challenges Fragmented Approach Analytics Business Packaged Intelligence Custom Reporting Applications Accessibility Enterprise Performance Data Replication Data Migration Data Warehousing Up-To-Date Data Silos Data Marts Data Hubs Trusted Data Federation Data Access information Batch Scripts Custom SQL InformationJava OLTP & ODS Oracle, PeopleSoft, Files, Excel OLAP Systems Data Warehouse, Data Warehouse, Data Warehouse, Siebel, SAP, XML Data Mart Data Mart Data Mart Custom Apps High Cost of Lack of clean, Multiple standards, Custom Coding consistent data disciplines 5
- 6. Data Warehouse and It’s Process A centralized repository containing comprehensive detailed and summary data that provides a complete view of customers, suppliers, business processes, and transactions, from a historical perspective with little volatility. Typically loaded on a nightly basis with batch extracts from source transaction processing systems such as CRM, ERP, etc to support reporting and analysis. Sources Target DATA WAREHOUSE ERP/ CRM Integrate DM1 DM2 Data Cleansing Load DM3 DM4 Extract Data RDBMS Enrichment Transform Analytical Reporting Ad-hoc Reporting Dashboards Scorecards 6
- 7. Business Drivers For a Modern Data Integration Platform & Real-Time DWH Demand for Continuous, Real-Time, Trusted Information Real-Time Enterprise Continuous Availability Real-Time Data for 24/7 Global Operations for Intelligence & Operations • Continuous uptime in event of disaster • Up-to-the second data for operations • No downtime during planned outage • Access to timely information for analysis • Low-impact data capture for integration • Data distribution across regions Trusted Information • Consistent with other systems •High data integrity 7
- 8. Traditional Vs Real-Time Data Warehouse Traditional Data Warehouse Real-Time Data Warehouse Transformation & Quality (ETL / ELT & Data Cleansing) ETL + CDC Real-Time Legacy, Continuous Feeds Packaged Enterprise Apps, Data Legacy, OLTP Warehouse Packaged Enterprise Data Databases Apps, OLTP Warehouse/MDM • Day+ old data • Timely, relevant data are continuous feed from operational systems • Batch data extracts within specified • No batch windows on OLTP “off business hours” • Sub-second latency • A middle-tier server for • No impact on source systems transformations • Read-consistent changed data with referential integrity • Process interruptions impact data • Transformations at capture, delivery or recoverability within the database 8
- 9. Oracle Complete Data Integration Solution
- 10. Oracle Data Integration: 3 Key Products Addressing operational and strategic analysis Oracle Data Quality Oracle Data Integrator Oracle GoldenGate & Profiling • E-LT • Real-Time data • Discover data • Bulk data movement capture and delivery problems • Complex • Low impact and non- • Global data cleansing transformation invasive • Data standardization • Easy to use GUI • Transactional integrity • Fuzzy matching design • Guaranteed delivery • Heterogeneous • Data lineage & impact • Delivers continuous analysis availability • Heterogeneous • Heterogeneous Accessibility Up-to-Date Information Trusted Information 10
- 11. Oracle Data Integrator Enterprise Edition Optimized E-LT for Improved Performance Legacy Sources E-LT Transformation Any Data vs. E-T-L Warehouse Application Declarative Set-based Sources design Pluggable Knowledge Any Modules Planning System OLTP DB Hot-pluggable Architecture Sources Change Data Capture for Dynamic Updates 11
- 12. Data Warehouse Bulk Loading w/E-LT Fastest ELT Solution for your Data Warehouse Solution • ODI for bulk loading Data Warehouse Extract • Run ODI Agent within Data Warehouse JVM • Fastest possible data Load transformations • Heterogeneous and loads any 3rd party data warehouse. Benefits Transform • No extra ETL servers • RDBMS Specific Knowledge Modules • Exploit DW RDBMS Optimizer • Easier to deploy than conventional ETL tools • Faster time to market • Enforce DW Best Practices 12
- 13. Optimized for Exadata Fastest E-L-T Processing • Massively parallel high volume hardware to quickly process vast amounts of data OLAP • Exadata runs data intensive processing directly in storage • Most complete analytic capabilities • OLAP, Statistics, Spatial, Data Mining, Real-time transactional ETL, Efficient point queries ELT • Powerful warehouse specific optimizations • Flexible Partitioning, Bitmap Indexing, Join indexing, Materialized Views, Result Cache Data Mining • E-LT runs 20X faster only with Oracle New 13
- 14. ODI is Faster Up to 7TB per hour of real world data loading and complex transformations ODI ELT (on Exadata) ODI scales with Exadata ODI runs on Exadata – no ETL hardware required Common administration, monitoring and management All the benefits of rapid tools-based ETL development Conventional ETL As data sets grow additional hardware ($$) needed ETL parallel optimization and design ($$$) is heavily dependent on resources available to the ETL system Poor performance – transformations take place outside of database, require staging tables Lack of light-weight architecture for rapid data loading ETL engine hardware resources only used for ETL Hardware not co located, multiple vendors Different management, monitoring and administration from database and BI infrastructure ($$) 14
- 15. Differentiator: E-LT Architecture High Performance Conventional: Separate ETL Server Conventional ETL Architecture • Proprietary ETL Engine • Poor Performance Extract Transform Load • High Costs for Separate Standalone Server Oracle: No New Servers • Lower Cost: Leverage Compute Resources & Partition Workload efficiently • Efficient: Exploits Database Optimizer • Fast: Exploits Native Bulk Load & Other Database Interfaces Next Generation Architecture • Scalable: Scales as you add Processors to Source or Target “E-LT” Benefits Transform Transform Optimal Performance & Scalability Extract Load Better Hardware Leverage Easier to Manage & Lower Cost 15
- 16. Oracle Data Profiling and Quality Integrated Data Profiling and Quality for Customer Data Oracle Data Profiling, Oracle Data Quality Metadata Profiling of Source & Target Any Source Duplicate Detection, Matching & Merging System Global Address Cleansing Data Control and Visibility Any Data Warehouse Optimized for Customer/Party Data & MDM Oracle Data Integrator Enterprise Edition Any Planning System 16
- 17. Oracle Data Quality Ensure Data Quality as Part of the Integration Process Best-in-class data quality and profiling for integration processes ODI EE Better Data Visibility Tighter Data Control Greater Data Accuracy Visual Data Quality Oracle Data Profiling Oracle Data Quality Tools Metadata Profiling Duplicate Detection, of Source & Target Matching & Merging 17
- 18. Sample Data Quality Issues COMPLETENESS Completeness What data is missing or CONFORMITY unusable? CONSISTENCY Consistency What data values give conflicting DUPLICATION information? Accuracy Conformity ACCURACY What data is stored What data is incorrect or out of in a non-standard date? format? Duplication What data records or attributes are repeated? 18
- 19. Oracle GoldenGate Enterprise-wide Solution for Real Time Data Needs Zero Downtime New DB/ Migration and OS/HW/App Upgrades Active-Active High Fully Active • Reduce Costs Availability Distributed Database Log Based, Real- • Lower Risks Time Change Data Capture Query Offloading Reporting • Achieve Oracle Database GoldenGate Operational ETL Excellence ODS EDW ETL Heterogeneous Real-time BI EDW Source Systems Data Distribution Global Data Centers SOA/EDA 19
- 20. Data Movement plus Real-Time Oracle GoldenGate provides low-impact capture, routing, transformation, and delivery of transactional data across heterogeneous environments in real time Key Differentiators: Performance Non-intrusive, low-impact, sub-second latency Flexible and Extensible Open, modular architecture - Supports heterogeneous sources and targets Reliable Maintains transactional integrity - Resilient against interruptions and failures 20
- 21. Oracle GoldenGate Architecture Designed for Speed, Flexibility and Reliability Trail Trail Capture Delivery LAN/WAN Internet TCP/IP Source Target Oracle & Non-Oracle Bi-directional Oracle & Non-Oracle Database(s) Database(s) • Real-time change data capture, routing and delivery across heterogeneous systems • Non-invasive , log-based change data capture for minimal impact on infrastructure • Transactional integrity and guaranteed data delivery across regions • Bi-directional replication and support for different replication topologies 21
- 22. Oracle Complete Data Integration Solution Usage Complete, Flexible, Integrated Source OLTP BI Database DR, Query, Reporting Tool Reporting SCM Apps Real-time Historic Analytics Analytics Real-Time Feed Operations ODI/ Oracle ODS ODP/ODQ GoldenGate CRM Apps Transform & Cleanse Oracle EDW GoldenGate Financials Staging Batch or Oracle Exadata Real-Time Feed or Teradata ERP Apps or Netezza, etc 22
- 24. Customer Example Leading online retailer offering a wide variety of high-quality, brand- name merchandise at discount prices and manufacturers, distributors and other retailers an alternative sales channel. Challenges & Objectives Solution & Benefits Need to enable sales, finance, marketing and Oracle GoldenGate captures real-time change data merchandising teams with near real-time data from ecommerce and auction systems. Oracle Data Integrator is used for highly complex transformations and data loading to user tables Required to have business insight on company Resulted in updated, current Teradata data warehouse performance meeting target metrics providing critical business intelligence for decision making Need to be able to handle high-volume data Complete, accurate data to give LOBs a trusted view of loading and transformation requirements like business progress, etc. 1.2M+ SKUs, 5M+ daily transactions, 300+ users 24
- 25. Overstock.com Supply Demand Manufacturers Consumers 350,000 sq. ft. User Fulfillment Friendly Warehouse Web Front End Distributors Small • Information-sensitive business Businesses • Demanding Business Community • Pricing, Forecasting • Rapid Growth 25
- 26. Overstock.com Innovations in Data Warehousing Traditional Framework Emerging Framework • Batch extracts/feeds from • Near real-time feeds from operational systems operational systems • Transformations in ETL engine on • Thin middle tier with E-LT the middle tier • Transformations on the database • Bulk load to the data warehouse platform • Large nightly batch, user online day • Small mini-batches throughout the day 26
- 27. Overstock.com Enterprise Solution for Enhanced Intelligence 27
- 28. Overstock.com: Innovations in Data Integration Speed translates to Improved Opportunities • Batch windows nearly eliminated • Low-latency data provides new opportunities • Trigger/event campaigns • Personalization, scoring, ranking • Marketing and merchandising improvements • Improved Customer Service • Now ranked #2 in the nation • Operational improvements in scoring and logistics led to amazing results • Upgraded Oracle 9i Database without business interruption 28
- 29. Lifetouch Portrait Studios Inc • Serve the portraiture needs of 2M Guests annually • 730 Photographic studios across the country We preserve memories and help our guest tell their unique story… FLASH! Digital Portraits The Studio at Target JCPenney Portraits 29
- 30. Better Customer Insight with Oracle Data Quality Business Challenges Implement a single, high-quality and consistent view of each customer to be available throughout the organization Streamline the sharing of customer data across all the customer facing applications Improve the accuracy of customer data to provide a better visibility and to elevate the customer relationships. Oracle Solution Return on Investment Implemented Oracle Data Quality for Improved customer data quality by a Data Integrator to cleanse customer minimum of 25% data. Studio workflow performance Combined Oracle solution with improved by 10% resulting in higher Oracle Database, MySQL and Oracle customer satisfaction and retention Data Integrator for every data Accurate and complete customer data movement and the overall into data warehouse enabled better orchestration of the process customer segmentation and targeted marketing campaign Reduced the time and budget required to implement data quality processes. 30
- 31. Benefits • Improved customer service with real time guest and appointment data enabling improved Studio efficiencies • Studio workflow performance improvement of 10% • Higher guest satisfaction and retention • Accurate and complete guest data in the Data Warehouse enabling • Improved guest segmentation • Improved targeted marketing campaigns • Cleansed and unique guest profiles shared across primary applications resulting in a minimum of 25% improved data quality. • Expected reduction of technical resource hours in implementing custom code, and performing manual data quality implementation and audits. • Time and budget saved with implementing ODI/ODQ solution vs. building functionality internally. 31
- 32. Benefits of Real-Time DW with Oracle Cut Costs, Reduce Risk, and Revolutionize Business Insight Cut Costs and Improve Efficiencies. • Move only the changed data from redo logs and reduce source and network overhead • Shorten implementation times from months to weeks using pre- packaged integrations to well-known applications, sources and targets. Reduce Risk, Ensure Continuity • Eliminate performance impact on source systems • Reduce the risk of missed orders, poor customer interactions, missed opportunities through improved recoverability, data quality Improve Business Insight • Enable near real-time decision making with real-time data flows • Combine real-time data with historical context for better insights 32
- 33. Questions