Designing An Enterprise Data Fabric
36 likes11,860 views
An enterprise data fabric is a comprehensive infrastructure composed of hardware and software designed to support data operations across an organization, facilitating data acquisition, transformation, storage, and management. It provides a flexible framework that enables seamless access and use of data assets while ensuring compliance with regulations and efficient data governance. The document outlines the components, design principles, and the need for a conceptual data fabric model to respond to evolving data challenges and trends.
Designing An Enterprise Data Fabric
- 1. Designing An Enterprise Data Fabric Alan McSweeney http://ie.linkedin.com/in/alanmcsweeney
- 2. What Is An Enterprise Data Fabric? • Set of hardware and software infrastructure, tools and facilities to implement, administer, manage and operate data operations across the entire span of the data within the enterprise across all data activities including data acquisition, transformation, storage, distribution, integration, replication, availability, security, protection, disaster recovery, presentation, analytics, preservation, retention, backup, retrieval, archival, recall, deletion, monitoring, capacity planning across all data storage platforms enabling use by applications to meet the data needs of the enterprise • Mesh enabling the movement of data around the enterprise • Provides access to all data assets • Supports the flow, processing, distribution, management and exchange of data throughout the enterprise • Provide coherent data framework for use by custom and acquired applications • Independent of specific applications • Independent of specific data platforms 18 February 2018 2
- 3. Building An Enterprise Data Fabric 18 February 2018 3
- 4. Core Data Fabric Conceptual Model 18 February 2018 4
- 5. Data Fabric Conceptual Model – Components - 1 of 2 18 February 2018 5 Component Description External Interacting Parties These are the range of external parties that supply data to and access data from the enterprise External Party Interaction Zones, Applications, Channels and Facilities These are the set of applications and data interface and exchange points provided specifically to External Interacting Parties to allow them supply data to and access data from the enterprise These can be hosted internally or externally or a mix of both External Third Party Applications These are third-party applications (such as social media platforms) that contain information about the enterprise or that are used by the enterprise to present information to or interact with External Interacting Parties or where the enterprise is referred to, affecting the perception or brand of the enterprise External Data Sensors Sources of remote data measurements External Party Interaction Zones Data Stores These are applications and sets of data created by the enterprise to be externally facing where external parties can access information and interact with the enterprise External Devices These are devices connected with services offered by the enterprise (such as ATMs and Kiosks) Date Intake/Gateway This is the set of facilities for handling data supplied to the enterprise including validation and transformation including a possible integration or service bus This can be hosted internally or externally or a mix of both Line of Business Applications This represents the set of line of business applications deployed on enterprise owned and managed infrastructure used by business functions to operate their business processes Organisation Operational Data Stores These are the various operational data stores used by the Line of Business Applications
- 6. Data Fabric Conceptual Model – Components - 2 of 2 18 February 2018 6 Component Description Line of Business Applications Hosted Outside the Organisation This represents the set of line of business applications deployed on external infrastructure used by business functions to operate their business processes This includes cloud facilities such as external data storage and XaaS facilities and an integration service to connect external data to internal data External Application Operational Data Stores These are the various operational data stores used by the Line of Business Applications used by Line of Business Applications Hosted Outside the Organisation Data Mastering These are facilities to create and manage master data and data extracted from operational data to create a data warehouse and data extracts for reporting and analysis. This includes an extract, transformation and load facility These can be hosted internally or externally or a mix of both Data Reporting and Analysis Facilities This represents the range of tools and facilities to report on, analyse, mine and model data These can be hosted internally or externally or a mix of both Document Sharing and Collaboration These are tools used within the enterprise to share and collaborate on the authoring of documents Document Management Systems These are systems used to manage transactional and ad hoc structured and unstructured documents in a formal and controlled manner, including the metadata assigned to documents Desktop Applications These are applications used by individual users to view and author documents Document and Information Portal This provides structured access to documents and information including externally hosted applications providing these facilities Unstructured Data Stores These are storage locations for enterprise documentation
- 7. Zones Within Data Fabric Conceptual Model • Sets of components of conceptual data fabric model can be grouped into zones: − Internal – within the enterprise’s boundary − Cloud Extension – extensions to enterprise applications and data held in external cloud platforms − Interface – set of components responsible for getting data into and out of the enterprise and presenting data and applications externally − Externally Located Extension – infrastructure and applications that are connected to the wider enterprise network − External Controlled – components outside the enterprise but under the control of the enterprise − External Uncontrolled – components outside the enterprise and not under the direct control of the enterprise 18 February 2018 7
- 8. Why Create A Conceptual Data Fabric Model? • Conceptual data fabric model represents a rich picture of the enterprise’s data context − Embodies an idealised and target data view • Detailed visualisations represent information more effectively than lengthy narrative text − More easily understood and engaged with • Show relationships, interactions • Capture complexity easily • Provides a more concise illustration of state • Better tool to elicit information • Gaps, errors and omissions more easily identified • Assists informed discussions • Evolve and refine rich picture representations of as-in and to-be situations • Cannot expect to capture every piece of information – focus on the important elements • A rich picture is not a data management process map (yet) 18 February 2018 8
- 9. Differences Between Current And Target Conceptual Data Model • Use the conceptual data fabric model to identify gaps between the current and desired target 18 February 2018 9
- 10. Core Data Fabric Conceptual Model • Conceptual level is one representation of data related components and their interactions within, across and outside the enterprise • Not all components apply to all enterprises • Useful as a basis for understanding the enterprise’s ideal data architecture − Creating an inventory of components in each conceptual area − Defining an idealised target data fabric • Just one dimension of defining, detailing and describing data infrastructure • Other dimensions include: − Data types − Data volumes − Individual data flows − Individual applications − Individual data platforms and applications 18 February 2018 10
- 11. Responding To Interrelated Data Trends 18 February 2018 11 Data Trends Cloud Offerings and Services Analytics Capabilities Data Regulations Internal and External Digital Expectations,
- 12. Responding To Interrelated Data Trends • Designing a data fabric enables the enterprise respond to and take advantage of key related data trends − Internal and External Digital Expectations • External actors expect to be able to interact digitally • Within the enterprise there is an imperative to offer digital interactions and extensions • Gives rise to large amounts of direct and indirect data that may or may not be processed − Cloud Offerings and Services • There are multiple providers of cloud-based services that enable the enterprise invest in and avail of application and data capabilities with low cost and time of entry • Data location changes and data must be integrated across platforms − Data Regulations • The data regulation landscape is changing - GDPR, ePrivacy Regulation Digital Single Market, eIDAS, NIS Directive • This requires greater data compliance and governance effort • Uncontrolled data platforms and storage represent a significant and real risk to the enterprise − Analytics Capabilities • New analytics capabilities across dimensions of data volumes and complexity enables more complex analysis 18 February 2018 12
- 13. IT Function Data Leadership • Enables the IT function demonstrate positive data leadership • Shows the IT function is able and willing to respond to business data needs 18 February 2018 13
- 14. What Are The Data Challenges? • More and more data of many different types • Increasingly distributed platform landscape with data movement, integration and management across multiple service providers and cloud-based services • Compliance and regulation requiring greater control of personal data • Newer data technologies and facilities outside the core competence of the enterprise • Shadow IT occurs when the IT function cannot deliver IT change and new data facilities quickly 18 February 2018 14
- 15. Data Fabric Is Much More Than A Move To The Cloud • Enterprise data fabric should enables appropriate and seamless move to multiple cloud/XaaS platforms - public, private and hybrid - across the entire data infrastructure − Storage − Business applications − Data management − Reporting and analytics tools • Cloud impacts the enterprise’s approach to data − Enterprises cannot ignore cloud and XaaS options • Enterprise data fabric needs to encompass the diversity of data storage infrastructures • Design an open and flexible data fabric that improves the responsiveness of the IT function and reduces shadow IT 18 February 2018 15
- 16. Why Have An Enterprise Data Fabric? • Enables adoption of new data technologies, platforms, systems and infrastructures within an overall data context • Enables move to simplification of data infrastructure • Enables scalability of data infrastructure • Enables industrialisation and automation of data operations, administration, management, governance and common security model • Reduce the effort and cost of management and administration • Focus on extracting data value • Improve the reliability of data operations • Manage risk of mixed data platforms, uncontrolled data on uncontrolled platforms • Allows benefits of scalable data infrastructures that are located anywhere to be achieved 18 February 2018 16
- 17. Why Have An Enterprise Data Fabric? • Focus on achieving benefits from data rather than on data operations − Reduce time to manage, find, combine and curate data − Reduce wasted time, capacity, resources, cost • Abstract data infrastructure from data usage • Enable use of data in currently unanticipated ways through flexible and adaptable facilities • Reduce time to achieve insights 18 February 2018 17
- 18. Creating A Data Vision • Data fabric is concerned with creating a data vision for the enterprise − Data capabilities, competencies − Where the enterprise is and where it wants to be • Define the future target landscape and define the required journey to achieve it • Ensures the vision can be executed • Allows the delivery effort and resources to be quantified • Permits the enterprise to move away traditional approaches to managing data 18 February 2018 18
- 19. Creating A Data Vision – Making The Enterprise Data Focussed • Enable value to be derived from data − Shorten the distance between business and analytics • Facilitate data initiatives by removing the barriers to data enablement • IT needs to understand the data needs and associated data business processes of the business and deliver results − IT showing data leadership • Top-down visualisation that is then implemented by appropriate components are different layers 18 February 2018 19
- 20. Current Data Fabric State 18 February 2018 20
- 21. Target Data Fabric Future State 18 February 2018 21
- 22. Achieving The Target Data Fabric State • Identify the steps needed to achieve the vision • Data fabric is linked to the applications that generate and use data • Use the data fabric as a model to describe the target future state • Articulate the future state vision 18 February 2018 22
- 23. Data Fabric And Digital Enablement • One element of digital business transformation is being able to handle and process large amounts of data and numbers of data sources • The data environment changes very quickly while at the same time becoming more distributed • Traditional data management approaches, toolsets and infrastructures fail to scale • Analytics tools tend to be linked to individual business function and data silos 18 February 2018 23
- 24. Key Design Principles Of A Data Fabric 18 February 2018 24 Administration, Management and Control – Keep control of and be able to manage and administer data irrespective of where it is located Security – Common security standards across entire fabric, automate governance and compliance and manage risk Automation – Management and housekeeping activities automated Integration – All components interoperate together across all layers Stability, Reliability and Consistency – Common tools and facilities used to delivery stable and reliable fabric across all layers Openness, Flexibility and Choice – Ability to choose and change data storage, data access, data location Performance, Retrieval, Access and Usage – Applications and users can get access to data when it is needed, as soon as it is needed and in a format in which it is usable
- 25. Business And IT Drivers For Data Fabric 18 February 2018 25 Reduce Cost of Change and Reaction React and Move Quickly React and Move Substantially Business IT Enable Growth Opportunities Balance Cost of Maintenance and Cost of Change Have A Choice Of And Be Able To Adopt New Technologies Offer Innovative Facilities and Functions React Quickly To New Requirements
- 26. Data Fabric Is A Basic Building Block Of An Enterprise Data Strategy 18 February 2018 26 Data Operations Management Data Quality Management Data Development Metadata Management Document and Content Management Reference and Master Data Management Data Security Management Data Warehousing and Business Intelligence Management Data Governance Data Architecture Management Reporting Insight/ Forecast Monitoring Analysis Solid Data Management Foundation and Framework } You Cannot Have This ... ... Without This
- 27. Why It Happened? Why Is Likely To Happen In The Future? What Is Currently Happening? What Happened? Every Enterprise Aspires To Data Driven Insights ... February 18, 2018 27 Reporting Insight/ Forecast Monitoring Analysis
- 28. Data Driven Trailing And Leading Indicators Reporting • Report on Gathered Information On What Happened To Understand Pinch Points, Quantify Effectiveness, Measure Resource Usage And Success Monitoring • Gather Information In Realtime To Understand Activities, Respond And Make Reallocation Decisions Analysis • Understand Reasons For Outcomes and Modify Operation To Embed Improvements Insight and Forecast • Quantify Propensities, Forecast Likely Outcomes, Identify Leading Indicators, Create Actionable Intelligence February 18, 2018 28 Trailing Indicators Leading Indicators
- 29. Objective Of Designing An Enterprise Data Fabric • Understanding all the data flows throughout the enterprise • Understanding yields insight into what is needed and what will generate a benefit 18 February 2018 29
- 30. Administration, Management Monitoring, Alerting, Event Management Archival, Recall Logging Extended Data Fabric Conceptual Model 18 February 2018 30
- 31. Extended Data Fabric Conceptual Model • Extended data fabric considers operating principles across core fabric components and their interactions 18 February 2018 31 Administration, Management • Ability to manage and administer the entire data fabric • Have a single view of the data fabric Utility, Usability • Be usable and be able to be used Operations • Support the automation of data fabric operations, perform capacity planning and management Monitoring, Alerting, Event Management • Provide monitoring of data fabric and support event management and alerting of problems Governance, Compliance, Risk Management • Support data governance principles and enforcement of regulatory compliance • Manage data risks Security, Protection • Enforce data security and ensure protection of data Archival, Recall • Support necessary and appropriate data archival and recall if required Preservation, Retention, Deletion • Provide facilities to enforce and automate data preservation, retention and deletion policies Capacity Planning • Manage capacity across all dimensions of data storage and I/O volumes and throughput Logging • Log and maintain details on data activities for reporting and analysis Installation, Upgrade. Reconfiguration • Support the seamless installation, upgrade and reconfiguration of new hardware and software components Backup, Recovery, Replication, Continuity, Availability • Implement backup and recovery, including business continuity, availability and replication across infrastructure components
- 32. Data Fabric Needs To Support Entire Data Lifecycle 18 February 2018 32
- 33. Data Lifecycle View • The stages in this generalised lifecycle are: − Architect, Budget, Plan, Design and Specify - This relates to the design and specification of the data storage and management and their supporting processes. This establishes the data management framework − Implement Underlying Technology- This is concerned with implementing the data-related hardware and software technology components. This relates to database components, data storage hardware, backup and recovery software, monitoring and control software and other items − Enter, Create, Acquire, Derive, Update, Integrate, Capture- This stage is where data originated, such as data entry or data capture and acquired from other systems or sources − Secure, Store, Replicate and Distribute - In this stage, data is stored with appropriate security and access controls including data access and update audit. It may be replicated to other applications and distributed − Present, Report, Analyse, Model - This stage is concerned with the presentation of information, the generation of reports and analysis and the created of derived information − Preserve, Protect and Recover- This stage relates to the management of data in terms of backup, recovery and retention/preservation − Archive and Recall - This stage is where information that is no longer active but still required in archived to secondary data storage platforms and from which the information can be recovered if required − Delete/Remove - The stage is concerned with the deletion of data that cannot or does not need to be retained any longer − Define, Design, Implement, Measure, Manage, Monitor, Control, Staff, Train and Administer, Standards, Governance, Fund - This is not a single stage but a set of processes and procedures that cross all stages and is concerned with ensuring that the processes associated with each of the lifestyle stages are operated correctly and that data assurance, quality and governance procedures exist and are operated February 18, 2018 33
- 34. Using The Core Conceptual Model • Understand the true complexity of data requirements within and across the enterprise • Use this complexity to derive a simplified an integrated data fabric 18 February 2018 34
- 35. Data As A Realisable Asset • Raw data must be refined into a format that can be used in order to be viewed as an asset with realisable value • For data to be an asset it must be: − Have its underlying value extracted − Accessible − Usable • Data has physical and tangible characteristics: − Mass – it has bulk and requires resources to store, process and move − Heat – it gets cold over time with different levels of dissipation − Energy – data has different levels of energy based on its movement and value − Volatility – the underlying value of the data can be lost at differing rates − Complexity – the content and structure of the data is variable − Motion – data moves from location to location as it is generated, stored, process − Structure – data may be structured, semi-structured or high-structured − Size to Value Ratio – the usable value with the data may be large or small relative to the volume of the raw data 18 February 2018 35
- 36. External Interacting Parties 18 February 2018 36
- 37. External Interacting Parties • Enterprises typically operate in a complex environment with multiple interactions with different communication with many parties of many different types over different channels • Many types of external party the enterprise interacts with • There will be multiple interactions with different communications with many parties of many different type over different channels • Every interaction will involve data being accessed, presented, transferred and processed • Business Customer • Client • Collaborator • Competitor • Contractor • Counterparty • Dealer • Distributor • Franchisee • Intermediary • Licensee • Licensor • Outsourcer • Partner • Provider • Public • Regulator • Regulated Entity • Representative • Retail Customer • Service • Shareholder • Sub-Contractor • Supplier 18 February 2018 37
- 38. External Party Interaction Zones, Applications, Channels and Facilities 18 February 2018 38
- 39. External Party Interaction Zones, Applications, Channels and Facilities • This is the range of application-based modes and methods of interaction between the enterprise and the External Interacting Parties (rather than pure email) 18 February 2018 39
- 40. External Party Interaction Zones Data Stores 18 February 2018 40
- 41. External Party Interaction Zones Data Stores • The data belonging to and data about the interactions with External Interacting Parties using External Party Interaction Zones, Applications, Channels and Facilities will be stored and managed 18 February 2018 41
- 42. Date Intake/Gateway 18 February 2018 42
- 43. Date Intake/Gateway • Generalised representation of the set of facilities for enabling and managing all communications between the enterprise (and its systems) and external parties − Broker and integration facilities for centralising all external communications – messaging, file transfer, web services − Allows two-way communications – send/receive and to/from internal and external − Supports multiple external channels and protocols − Supports multiple authentication schemes and standards − Provides asynchronous messaging − Includes application programming interface − Allows the exposure of endpoints which external parties can access such as SFTP − Provides management and administration facilities to define how communications should operate and for support and problem identification and resolution − Delivers facilities for orchestration, transformation, development and deployment management, traffic management − Ensure data quality − Provides workflow definition, implementation and operation − Maintains an audit trail of all messages and communications − Delivers high performance, resilience and availability 18 February 2018 43
- 44. External Third Party Applications 18 February 2018 44
- 45. External Third Party Applications • The enterprise may use external applications (such as social media platforms) as sources of external party data, as routes to advertise or direct a message to external parties or as channels to interact with external parties − Information and content stored directly on applications − Information about usage and interactions available from applications • The enterprise may also use external applications for collaboration and information sharing either within the enterprise or with external parties 18 February 2018 45
- 46. External Data Sensors 18 February 2018 46
- 47. External Data Sensors • These represent measurement infrastructure and applications owned by the enterprise, located externally on some wide area network or other communications facility that generate data that is transmitted to the enterprise − Telemetry units 18 February 2018 47
- 48. External Devices 18 February 2018 48
- 49. External Devices • These represent infrastructure and applications owned by the enterprise, located externally on some wide area network or other communications facility that are accessed and used by external parties to interact with the enterprise − ATMs − Kiosks − Point of sale devices 18 February 2018 49
- 50. Line of Business Applications 18 February 2018 50
- 51. Line of Business Applications • This represent the applications used by individual business functions or across the enterprise that are hosted on internal enterprise infrastructure or are hosted externally by application or platform service providers 18 February 2018 51
- 52. Data Storage Platforms 18 February 2018 52
- 53. Data Storage Platforms • These represent the various structure data stores and associated database management software used by applications that are hosted on internal enterprise infrastructure or are hosted externally by application or platform service providers 18 February 2018 53
- 54. Data Reporting and Analysis Facilities 18 February 2018 54
- 55. Data Reporting and Analysis Facilities • This represents the set of facilities to extract operational data from business applications, create, store and manage reference and master data, create and store enduring data and analyse the data including reporting, visualisation, mining and modelling 18 February 2018 55
- 56. Document Management Systems And Document Sharing and Collaboration 18 February 2018 56
- 57. Document Management Systems And Document Sharing and Collaboration • This represents the facilities to store structure and unstructured document-oriented data including document metadata, extract information from documents and support ad hoc and formal workflows related to these documents 18 February 2018 57
- 58. Desktop Applications 18 February 2018 58
- 59. Desktop Applications • These are the suite of desktop applications including email to create, update, distribute and collaborate on documents 18 February 2018 59
- 60. Many Data Types 18 February 2018 60 Transactions and Application Data Unstructured Data Documents Document Images Videos Sound Usage Logs Third-Party Data Files Messages Reports Derived Data Data Models Web Content Telemetry Data Data Warehouse and Data Marts Emails Reference and Master Data Metadata
- 61. Data Fabric As Data Plumbing And A Data Refinery • Data fabric should enable the flow of data throughout the enterprise and the refinement of data to create appropriate refined and derived data products from raw data 18 February 2018 61
- 62. 18 February 2018 62 Data Layers Across Data Fabric Layer Components Data Scope Layer 8+ Data Operations, Usage, Management, Control, Governance, Analysis, Modelling Unified management across all environments and all layers and ensure performance, availability, reliability, scalability, maintainability and supportability Layer 7 Data Presentation, Platforms, Applications, Systems and Business Processes Set of data accessing and data using business applications Layer 6 Data Security and Governance Implement common data security policies across all environments and platforms Layer 5 Data Logical Access and Integration Insulate and abstract access from knowledge of environments and platforms and integrate data systems and data management Layer 4 Data Transportation Provide a common data transport that connects all environments Layer 3 Data Network and Connectivity Connections to storage and physical access irrespective of location across entire network Layer 2 Data Physical Access Provide physical access to data on storage layer Layer 1 Data Storage and Transmission Infrastructure Store data transparently on multiple environments and move data between environments
- 63. Building A Comprehensive Data Vision 18 February 2018 63 Comprehensive Data Vision Enterprise Data Strategy Strategy Area … Strategy Area Core Data Fabric Conceptual Model Components Component Type Component … Component … Component Type Component … Component Extended Data Fabric Conceptual Model Data Management and Operations Facility … Data Management and Operations Facility Data Lifecycle Stage … Stage Data Types Type … Type
- 64. Extending Conceptual Model To Additional Levels Of Detail To Build A Comprehensive Data Vision • Individual data views can be combined to articulate a comprehensive data vision − Enterprise Data Strategy • Individual strategy areas − Core Data Fabric Conceptual Model Components • Individual elements within each component − Extended Data Fabric Conceptual Model • Operating principles and interactions − Data Lifecycle • Individual stages within lifecycle − Data Types • Individual data types • Builds an understanding of how the enterprise wants and needs to handle and use data 18 February 2018 64
- 65. Extending Conceptual Model To Additional Levels Of Detail To Build A Comprehensive Data Vision 18 February 2018 65 Data Fabric Landscape Additional Data Dimensions and Views
- 66. Summary • Data fabric is concerned with creating a data vision for the enterprise • The conceptual data fabric model represents a rich picture of the enterprise’s data context − Detailed visualisations represent information more effectively than lengthy narrative text • Use the conceptual data fabric model to identify gaps between the current and desired target • Data fabric provides a basis for understanding the enterprise’s ideal data architecture • Designing a data fabric enables the enterprise respond to and take advantage of key related data trends − Shadow IT occurs when the IT function cannot deliver IT change and new data facilities quickly − Uncontrolled data platforms and storage represent a significant and real risk to the enterprise • Enterprise data fabric should enables appropriate and seamless move to multiple cloud/XaaS platforms - public, private and hybrid - across the entire data infrastructure • Enables the enterprise focus on achieving benefits from data rather than on data operations 18 February 2018 66