IBM InfoSphere Stewardship Center for iis dqec
- 1. © 2015 IBM Corporation IBM InfoSphere Information Server Stewardship Center and Data Quality Exception Console
- 2. 2 © 2015 IBM Corporation Agenda IBM Data StewardShip Center Data Quality Exception Console Questions & Réponses
- 3. © 2015 IBM Corporation Stewardship Center
- 4. 5 © 2015 IBM Corporation Governance Officers Data Stewards (IT and LOB) Data Analysts/SMEs Web Mobile Social Collaboration Dashboard Workflow & Rules Provide a single collaborative environment for business users to define and monitor compliance with data quality policies and to manage data quality issues to resolution Addressing the needs of various roles across the governance organization Key capabilities delivered within customizable and collaborative interfaces IBM InfoSphere Stewardship Center
- 5. 6 © 2015 IBM Corporation InfoSphere Stewardship Center: Event Notification with Workflow and Remediation Read File Errors Business Term Updates Notifications/ Email Exception Fixes Quality Exception Notification Duplicate Resolution Policy Changes Suspect Duplicates Catalog Updates
- 6. 7 © 2015 IBM Corporation Data Steward Stewie Silvers Governance Council Member Gary Grayson Data Steward Manager Martin McGregor Meets the Needs of Critical Business Roles Traditional Data Steward and LOB users collaborate to remediate record violations, improving data quality. Data Steward Manager responsible for overseeing a team of subject matters experts. Data Quality Champion and LOB leader, collaborate defining data policies and rules for the business in addition to monitoring policy compliance
- 7. 8 © 2015 IBM Corporation Stewardship Center Performance Dashboard The Stewardship Center provides out of the box Dashboards: – Team Performance and Process Performance
- 8. 9 © 2015 IBM Corporation Stewardship Center takes advantage of IBM BPM Enterprise Processes Services Information Server Services Customer Patient Vendor Product Account IBM BPM & Decisions BPM Standard Onboarding Registration Party updates Sales Marketing Delivery Information Governance and Data Quality Stewardship Information Server and IBM Business Process Management come together to help you manage your Governance and Data Quality processes: Information Server delivers real time data exceptions and events to be governed using a repeatable business process BPM provides orchestration of the steps required to improve data quality and increase common understanding Business Process Management is a discipline that leverages software and services to provide total visibility into your organization. Discover, document, automate, and continuously improve business processes to increase efficiency and reduce costs.
- 9. 10 © 2015 IBM Corporation Go Mobile, Get Social, and Govern Mobile Access Social Collaboration Governance Workflow Reporting
- 10. 11 © 2015 IBM Corporation What is the Stewardship Center? The integration of IBM BPM Standard Edition and Information Server – ‘Stewardship Center’ is a customized home screen for IBM BPM Process Portal IBM provides a set of sample BPM applications to demonstrate the basic capabilities – Exception Management – Information Governance Catalog Governance Rule workflow – Information Governance Catalog Business Term emails This section will focus on the Exception Management sample application – Information Governance Catalog applications have already been discussed
- 11. 12 © 2015 IBM Corporation InfoSphere Stewardship Center Single Inbox and Remediation Portal MDM tasks DataStage/ QualityStage tasks Information Analyzer tasks Information Governance Catalog tasks
- 12. 13 © 2015 IBM Corporation Start Exception list submit fixes reject approve Approval request list Apply fix request list escalate apply fix Data Steward Data Steward Manager Data Source Owner End Out Of the Box Exception Remediation Sample End resolve User=silvers Name=Stewie Silvers User=mcgregor Name=Martin McGregor User=williams Name=Chad Williams Exception
- 13. 14 © 2015 IBM Corporation A View in IBM Process Designer
- 14. 15 © 2015 IBM Corporation Workflow – Data Steward Sign in as Data Steward ‘silvers’ See ‘Open Tasks’ new exceptions Sample application as defined the due date as date run + 24 hours. PoC/Demo Tip: Switch between ‘Open Tasks’ and ‘Completed Tasks’ to refresh the screen PoC/Demo Tip: Carefully read to see the exception source
- 15. 16 © 2015 IBM Corporation Workflow – Data Steward (continued) Claim the task
- 16. 17 © 2015 IBM Corporation Workflow – Data Steward (continued) Work with the task PoC/Demo Tip: ‘Submit’ after changes ‘Escalate’ if you don’t know what changes to make
- 17. 18 © 2015 IBM Corporation Workflow – Data Steward (continued) Work with the task (continued) View the BPM Process Diagram
- 18. 19 © 2015 IBM Corporation Workflow – Data Steward (continued) Work with the task (continued) – See the exception description You can drill directly to the Information Governance Catalog to see the Data Rule The link is AUTOMATIC and illustrates integration of our suite!
- 19. 20 © 2015 IBM Corporation Workflow – Data Steward (continued) Work with the task (continued) – See the exception description definition in IGC The Data Steward can find what they need with a few clicks! …and continue to drill into the Data Rule Definition
- 20. 21 © 2015 IBM Corporation Workflow – Data Steward (continued) Work with the task (continued) – Edit and change individual records The sample application allows any data to be changed. It also shows how it is possible to have a ‘read only’ flag to prevent data change. The sample application only allows editing of one exception at a time. Stewardship Center supports EDIT!
- 21. 22 © 2015 IBM Corporation Workflow – Data Steward (continued) Work with the task (continued) – Edit and change individual records (continued)
- 22. 23 © 2015 IBM Corporation Workflow – Data Steward (continued) Work with the task (continued) – View changed records Changed records are highlighted in light blue. Changed data is highlighted in dark blue. Data Steward can VIEW changes!
- 23. 24 © 2015 IBM Corporation Workflow – Data Steward (continued) Work with the task (continued) – Changing multiple records with Export • Also can be used to share data with others
- 24. 25 © 2015 IBM Corporation Workflow – Data Steward (continued) Work with the task (continued) – Changing multiple records with Export (continued) A csv file is downloaded to the client Every column output by the rule will have a ‘_new’ column
- 25. 26 © 2015 IBM Corporation Workflow – Data Steward (continued) Work with the task (continued) – Changing multiple records with Export (continued) You can use Microsoft™ Excel® to edit the file PoC Tip: Be very careful when opening .csv files in Excel. Any data with leading zeroes will have those leading zeroes remove. For example, the valid US Postal Code 02201 will become the invalid US Postal Code of 2201. Excel will also corrupt timestamps: 2015-01-25 09:40:13.0 becomes 40:13.0.
- 26. 27 © 2015 IBM Corporation Workflow – Data Steward (continued) Work with the task (continued) – Changing multiple records with Export (continued) Then ‘Import’ Your Firefox ‘import’ screen will be blank PoC/Demo Tip: Import does not currently work with Internet Explorer. Use Firefox instead.
- 27. 28 © 2015 IBM Corporation Workflow – Data Steward (continued) Work with the task (continued) – Changing multiple records with Export (continued) You will need to change the number of ‘Rows per page’ to force a refresh
- 28. 29 © 2015 IBM Corporation But Where Is The Fixed Data Stored? The fixed data is stored in the providers repository DataStage and QualityStage Exceptions Stage = Exceptions Stage Database
- 29. 30 © 2015 IBM Corporation But Where Is The Fixed Data Stored? (continued) DataStage and QualityStage Exceptions Stage – In the same output table as the exceptions • Reminder: Get the table name from the job run:
- 30. 31 © 2015 IBM Corporation But Where Is The Fixed Data Stored? (continued) DataStage and QualityStage Exceptions Stage (continued) – In the same output table as the exceptions (continued) – The data is defined as a BLOB The data is JSON stored in binary format, but CAN be parsed by the Hierarchical Stage… This is the view from IBM Data Studio
- 31. © 2015 IBM Corporation Back to the Workflow
- 32. 33 © 2015 IBM Corporation Workflow – Data Steward (continued) Data Steward can do one of two things: – Submit Fixes • When changes are done – Escalate Task • Ask the Data Steward Manager to review unknown exceptions Updates shown in the DQEC
- 33. 34 © 2015 IBM Corporation Workflow – Data Steward Manager When the Data Steward selects ‘Escalate’ – Resolve – the task is complete – DQEC is updated
- 34. 35 © 2015 IBM Corporation Workflow – Data Steward Manager (continued) When the Data Steward selects ‘Submit Fixes’ – Approve – goes on to Data Source Owner – Reject – goes back to Data Steward • Any fixes are also discarded – Either one updates the Data Quality Exception Console
- 35. 36 © 2015 IBM Corporation Workflow – Data Source Owner When the Data Source Owner select ‘Approve’ – Apply Fixes • Once changes done (however they are done) – Data Quality Exception Console is updated
- 36. 37 © 2015 IBM Corporation Start Exception list submit fixes reject approve Approval request list Apply fix request list escalate apply fix Data Steward Data Steward Manager Data Source Owner End Out Of the Box Exception Remediation Sample - Done End resolve Exception
- 37. 38 © 2015 IBM Corporation InfoSphere Stewardship Center CollaborationMonitoring Inbox Data Quality Exception ConsoleCommon Event Framework Information Analyzer MDM Server DataStage/QualityStage Exception Stage Information Governance Catalog Information Governance Dashboard API (Information Analyzer, DataStage/QualityStage Exceptions Stage) Source Systems WorkflowsWorkflows 1a 1b 2a 2b 1c 1d 3 1a-d 2a-b 3 Information Analyzer and DataStage/QualityStage Exception Stage exception events and notifications Information Governance Catalog events and notifications MDM integration InfoSphere Stewardship Center Architecture Event Notifier Event Registry Event Service
- 38. 39 © 2015 IBM Corporation Subscriptions Subscription Manager is on the Information Server Launchpad
- 39. 40 © 2015 IBM Corporation Subscriptions (continued) PoC/Demo Tip: Name your subscriptions: Asset Type + Application + Other Information For example: Business Term Basic Email Notification Create Email is available for all Event Types. Application will only be listed if there is a registered Callback. All subscriptions allow filtering. Filters must be a single item. ACTION=CREATE or ACTION=MODIFY would require two subscriptions.
- 40. 41 © 2015 IBM Corporation Subscriptions (continued) Notification Type ‘Email’ available to all event types – Make sure to set up email provider (the smtp server) as per this link – The email will go to the email address entered for the user – You can send a copy to any other user PoC/Demo Tip: Subscription Manager Email only supports ‘no security’ (port 25), or ‘SSL’ security (port 465). TLS (port 587) is not supported. Hotmail, Live, and Outlook all use TLS.
- 41. 42 © 2015 IBM Corporation Subscriptions (continued) Notification Type ‘Email’ available to all event types (continued) – At GA, the email will look better for Remediation events than for IGC events • Remediation DataStage/QualityStage emails: • Information Analyzer emails: • IGC emails (from creation of a new Category): Work is underway to make the emails customizable.
- 42. 43 © 2015 IBM Corporation Subscriptions (continued) Note the Event Types are not sorted
- 43. 44 © 2015 IBM Corporation Notifications Notifications are published every time a subscribed action is performed – Notifications are ONLY generated for subscribed actions You can list and delete notifications based on time ranges – -beginDate “YYYY-MM-DD hh:mm:ss” – -endDate “YYYY-MM-DD hh:mm:ss” – You can specific just: • beginDate (everything from a specific time forward) • endDate (everything up to a specific time)
- 44. © 2015 IBM Corporation Thank You End of Section 3
- 45. 46 © 2015 IBM Corporation Agenda Stewardship Center Overview Brief BPM Overview High Level Architecture Information Governance Catalog – Basic Event Notification – Sample IBM BPM Applications Labs Part 1 Data Quality Remediation – Data Quality Exception Console – Stewardship Center Labs Part 2 Common Event Framework – Event Management Architecture – Event Providers – Event Types – Callbacks – Subscriptions – Notifications – Events – Task Scheduling Labs Part 3 Appendices – Delete from Data Quality Exception Console – Delete from Stewardship Center Objective: Clearly Explain Stewardship Center Architecture, Capabilities and Competitive Advantages
- 46. © 2015 IBM Corporation Data Quality Execption Console
- 47. 48 © 2015 IBM Corporation Data Quality Exception Console Replaces the Data Quality Console – Data Quality Console still available today – Data Quality Console still fully functional – Will be removed completely in a future release Is the first step to managing exceptions in the Stewardship Center – Exceptions ‘sets’ must be selected to be ‘managed’ for the the out-of-the-box application The Data Quality Exception Console allows only critical exceptions to be managed in the Stewardship Center.
- 48. 49 © 2015 IBM Corporation Data Quality Exception Console (DQEC) Start from the Launchpad
- 49. 50 © 2015 IBM Corporation Data Quality Exception Console (continued) Filters to limit what is displayed Exceptions Browser based Link to Information Governance Dashboard Sort on any column
- 50. 51 © 2015 IBM Corporation Filtering Many Filter options Combining filters is an ‘and’ – Example: Contain string ‘val’ and produced by Information Analyzer Filters are predefined Filters can not be saved (yet)
- 51. 52 © 2015 IBM Corporation What Does The Data Quality Exception Console Show? Information Analyzer – Output from Data Rule run – Output from Rule Set run This option does not matter. Whatever you output will go to the Data Quality Exception Console. PoC Tip: Choose wisely! These are the columns seen in the Stewardship Center PoC Tip: DO NOT use spaces in your Data Rules Names!
- 52. 53 © 2015 IBM Corporation What Does The Data Quality Exception Console Show? (continued) DataStage and QualityStage – Output from Exceptions Stage only • Not from the One-Source Match Exceptions nor the Two-Source Match Exceptions Tip: Always include the ‘ruleID’ from the Data Rules Stage so you know which Data Rule Definition caused the exception
- 53. 54 © 2015 IBM Corporation What Does The Data Quality Exception Console Show? (continued) Information Analyzer Data Rule Definition and Rule Set Definition ‘Test’ runs will also show System assigned Name Poc/Demo Tip: Delete these test runs when testing is completed.
- 54. 55 © 2015 IBM Corporation Displaying Exception Details Status will be updated by the Stewardship Center!!!!!!! Role based actions For example: ‘Administrators’ can only ‘Delete’ The Data Quality Exception Console is integrated!
- 55. 56 © 2015 IBM Corporation Viewing Records More details to come… Demo Tip: Do not use the browser ‘back’ button. Click here instead.
- 56. 57 © 2015 IBM Corporation ‘Edit’ and ‘Delete’ Edit to change the exception priority – Not to edit the data (that can be done in the Stewardship Center!) – Not reflected in Stewardship Center – Is for this exception only, not all instances of this exception
- 57. 58 © 2015 IBM Corporation ‘Edit’ and ‘Delete’ (continue) Delete to remove the exception – Also removes the output table • Does not matter what type – all are deleted
- 58. 59 © 2015 IBM Corporation But What About ‘Manage’ and ‘Monitor’? All exceptions will be visible in the DQEC Data Steward Manager decides how to handle the exceptions with: – Manage = Which exceptions are to be managed in the Stewardship Center – Monitor = Which exceptions are to be monitored in the DQEC
- 59. 60 © 2015 IBM Corporation But What About ‘Manage’ and ‘Monitor’? (continued) Selecting ‘Manage’ will make this exception visible in the Stewardship Center – AND all future exceptions with the same name
- 60. 61 © 2015 IBM Corporation Questions & Réponses