Ibm Optim Techical Overview 01282009
- 1. IBM Optim Technical Overview Charles Lucas IBM Data Management Specialist
- 2. Agenda Challenges Optim Concepts Optim Architecture Optim for Archiving Optim Test Data Management Optim Data Privacy Summary
- 3. Challenges
- 4. Challenges Facing Customers Today Mitigate Risk Effectively and securely manage archived data Protect data privacy Accurate, prompt responses to auditing requests Maintain Performance in face of Data Growth Improve application performance by moving historical transaction records to a safe, secure archive Achieve Service Level Agreements (SLAs) consistently Control Costs Reduce infrastructure costs; utilize cost effective tiered storage Minimize cost and time for compliance Improve productivity of development team
- 5. Challenges: Reduce Risk Insiders and hackers are targeting data for profit Data in and of itself has monetary value: Credit Card Number With PIN - $500 Drivers License - $150 Birth Certificate - $150 Source: USA TODAY research 10/06 Average cost of a data breach in 2007 was $197 USD per customer record leaked Source: Ponemon Institute This has been a driving factor for creating data protection and privacy regulations How to protect Personal Identifiable Information (PII)?
- 6. Challenges: Dealing with Data Growth Data is growing at a very rapid rate Annual growth rates for databases exceed 125% Mergers & acquisitions Data Warehousing The Data Multiplier effect OLAP cubes, data marts, and so on Copies of data for test, development, quality assurance, disaster recovery, etc. Retention of data for compliance purposes Info 2.0 applications are verbose How to manage data growth and aging effectively?
- 7. Challenges: Control Costs Growing storage costs due to rapid data growth Cost of storing and managing the many copies of your production data Cost of implementing data privacy measures for compliance across different databases and applications Cost of archive retrievals for compliance requests and e-discovery Growing development time costs How to control storage and data management costs?
- 9. Optim Concepts Complete Business Object Federated Data Enterprise Architecture Terminology
- 10. Complete Business Object Referentially-intact subset of data across related tables and applications; includes metadata Provides “historical reference snapshot” of business activity Federated object support across enterprise data stores Payments
- 11. Federated Data Support Retek / Oracle Other apps / any DBMS Custom Inventory Management / DB2 Capture related business objects and processes from across the enterprise
- 12. Enterprise Architecture Platform independent architecture acts as central point to extract, store, restore and transform application data.
- 13. Optim Terminology Referentially-Complete Optim extracts data based on primary/foreign key relationships (“parent/child”) between tables. Handling data this way reduces errors and allows data to be moved without breaking application software Subsetting Using Optim to create a reduced size but referentially complete copy of a database for development or test. Masking Changing sensitive data before testing by replacing it with false but equally valid data.
- 15. Optim Architecture Optim Workstation Optim installed on a Windows PC capable of performing all Optim functions directly against a data source or by connecting to an Optim Server. Optim Server Optim installed on a Unix or Windows server that handles requests from Optim Workstations or the command line. Open Data Manager (ODM) Allows access to archive files as an ODBC data source. Allows access to archived tables through Oracle Heterogeneous Services as a Database Link
- 16. Optim Architecture Architecture Optim Workstation Optim Server Enterprise Reporting Tools ODM Optim Universal Database Access Layer ERP CRM Custom App Optim Directory Dev QA Archive Archive Archive Archive
- 17. More Important Terminology Optim Directory A DBMS-based repository for Optim metadata DB Alias How Optim refers to a database Access Definition A description of what to extract and how to extract it Relationship User defined connection between the data in two tables based on matching one to many columns in each table Primary Key User defined set of columns unique within a table
- 18. The Optim Directory An “instance” of Optim Holds all Optim metadata in database tables Security possible using windows domain/user Maintains a directory of archive Files Created by Optim configuration when “Configure the First Workstation” is performed Additional users can get access by using the “Configure Additional Workstation” dialog
- 19. Database Alias Defines database in an Optim directory. Created by Optim Configuration Allows any database object to be addressed and accessed by a 3-part address: DB Alias Owner Object Name Allows consistent naming across the enterprise More than just a connect string!
- 20. Created in Optim Design GUI Defines a set of tables and relationships which can be traversed relationally Archive Actions, which are stored procedures or DML statements, may be fired at key points of control Every extract, delete or restore requires an Access Definition These may be named and shared or local and tied to a single job. Access Definition
- 21. Relationships are automatically found when primary keys and foreign keys are defined in DB. User-defined primary keys and relationships can be created in Optim in GUI designer or imported. Relationship can be cross-database (between two databases, named by DB Alias) Relationships Primary Keys Relationships Optim Directory Database Alias
- 23. Archive, Retention and e-Discovery Production Extract Restore Archive E-Discovery Universal Access to Data Optim safely moves inactive or historical data to an archive Archive can be accessed in many ways
- 24. Optim Data Growth Solution: Archiving Production Selective Restore Archives Complete Business Object is historical snapshot of activity Storage device independence enables ILM Immutable file format enables data retention compliance Current Historical Restored Reporting Data Historical Data Reference Data Archive Universal Access to Application Data Application Application XML ODBC / JDBC
- 25. Optim Test Data Management
- 26. Why Optim Test Data Management? Improve development productivity Faster turnaround time Supports test automation Easier to create/verify test results Multiple sandboxes Better quality data (more frequent refreshes) Control costs Reduce storage per test instance Ease DBA workload
- 27. Test Data Management Easily maintain test environments Create targeted, “right-sized” subsets faster and more efficiently than cloning Production Compare Dev QA Test Load Insert / Update Compare Extract Files Extract
- 28. Optim Test Data Management Data Fixes Compare Results TEST Go Live Production Application Refine Data Copy Production Data for Testing Refresh Test Data Optim Extract Optim Edit Optim Compare Optim Extract Optim Edit
- 29. FIND CUSTOMER NOTE INFO EXIT TABLE FIND ORDERS NOTE INFO EXIT TABLE FIND DETAILS NOTE INFO EXIT TABLE Single Table Editors The Relational Editor Traditional vs. Relational Tools One table or view at a time No edit of related data from multiple Simultaneous browse/edit of related data from multiple tables and databases Insert, delete, update Audit trail, access controls CUSTOMERS ORDERS DETAILS ........................ ........................ ........................ ........................ ........................
- 30. Find unexpected changes (or validate expected changes) For application testing, QA, and to verify database contents Single-table or multi-table compare Creates compare file and/or compare Report of results Optim Compare Master Copy Latest Test Files Reports Compare
- 32. Why IBM Data Privacy? Protecting sensitive data Regulatory & Compliance PCI HIPPA EU Safe Harbour Many more… Off shoring testing Sub subcontracting test & dev. Good business practice Training environments
- 33. Optim Data Privacy Solution Substitute confidential information with fictionalized data Deploy multiple masking algorithms Provide consistency across environments and iterations Enable off-shore testing Protect private data in non-production environments Production Contextual, Application- Aware, Persistent Data Masking EBS / Oracle Custom / Sybase Siebel / DB2 Test EBS / Oracle Custom / Sybase Siebel / DB2
- 34. Data Privacy A comprehensive set of data masking techniques to de-identify data R eplaces (masks) confidential data with contextually accurate but fictionalized data Production Transform and Mask Masked Test Data
- 35. De-Identify test data Can Be Performed During Extract Process from DB During Insert/Load Process to DB Or as a Standalone Convert Process Transform or mask sensitive data using : Standard rules: Literals, Special Registers, Expressions, Default Values, Look-up tables Intelligent transformation rules: PCI, Addresses etc. Custom mapping rules: user exits Converted extract file is safe to share – sanitized data
- 36. Masked fields are consistent Data is masked DB2 Client Billing Application Consistent mapping Across the enterprise 132009824 157342266 SSN#s 132009824 157342266 SS#s 323457245 134235489 SSN#s 323457245 134235489 SSN#s
- 37. Map unlike column names Transform/mask sensitive data Datatype conversions Column-level semantic date aging Literals Registers Calculations Default values Exits Social Security (US) Credit Card Email Hash Lookup Lookup Random Lookup NAME tables (US) ADDRESS table (US) Shuffle String manipulation Currency conversion Masking Functions
- 38. Example: Bank Account Numbers First Financial Bank’s account numbers are formatted “123-4567” with the first three digits representing the type of account (checking, savings, or money market) and last four digits representing customer ID To mask account numbers for testing, use the actual first three digits , plus a sequential four-digit number The result is a fictionalized account number with a valid format: “ 001-9898” becomes “001-1000” “ 001-4570” becomes “001-1001” Complexity 1
- 39. Example: First and Last Name Direct Response Marketing, Inc. is testing its order fulfillment system Fictionalize customer names to pull first and last names randomly from the Customer Information table: “ Gerard Depardieu” becomes “Ronald Smith” “ Lucille Ball” becomes “Elena Wu” Optim ships with over 5,000 male/female names and over 80,000 last names Complexity 2
- 40. Example: Addresses Direct Response Marketing, Inc. is testing its order fulfillment system Fictionalize customer addresses to pull an entire address from the Customer Information table: “ 111 Campus Drive Princeton, NJ 08540 ” becomes… “ 1223 E. 12 th Street NY, NY 10079” Optim ships with over 100,000 valid CASS addresses Complexity 3
- 41. Example: Intelligence Generating valid social security numbers (as defined by the US Social Security Administration) Generate valid credit card numbers (as defined by credit card issuers) Generate desensitized e-mail addresses Generate Email address based on format: name@domain Complexity 3
- 42. Using Custom Masking Exits Apply complex data transformation algorithms and populate the resulting value to the destination column Selectively include or exclude rows and apply logic to the masking process Valuable where the desired transformation is beyond the scope of supplied Column Map functions Example: Generate a value for CUST_ID based on customer location, average account balance, and volume of transaction activity Complexity 4
- 43. Summary
- 44. Summary IBM Optim helps solve 3 major challenges for enterprises today: Migrate Risks Maintain performance in the face of major data growth Reduce Costs IBM Optim enables effective ILM The IBM Data Growth solution keeps high performance of applications in the face of data growth by archiving inactive data Once archived, it supports prompt, accurate responses to audit and discovery requests
- 45. Summary Test data management can speed delivery of developed applications IBM Optim’s data masking capabilities protect privacy by de-identifying sensitive data Pre-built modules for many popular applications are supported by IBM Optim Optim is a recognized market leader and used successfully by customers in almost all industries
Editor's Notes
- #2: This presentation provides a technical overview of IBM Optim and its benefits.