FROM BIG DATA TO ACTION: HOW TO BREAK OUT OF THE SILOS AND LEVERAGE DATA GOVERNANCE FOR YOUR ORGANIZATION

Editor's Notes

• #4: 3 minute video gives a great intro into the why/how… let this lead us forward.
• #7: AUTOMATED – Metadata is created by application at the same as the data is created in a standard manner easily consumable for all with necessary permissions Device that took the picture / name of picture / settings picture was taken at / location geo tag of picture etc – all automatic – all done at creation of data time
• #15: Egeria is an Open Source framework that can be used to provide a distributed, unified view of metadata from different sources, including different stores and tools from different vendors. Egeria creates a unified view of metadata residing in those tools and stores, so users can collaborate and share metadata, without needing to visit multiple tools or stores. Egeria does not attempt to consolidate the metadata into one repository or tool – it’s better to leave it in place - the current owners stay in control of their metadata, and it stays local to its native store or tool. Egeria provides an open type system, plus APIs, protocols, connectors and local metadata repositories.
• #16: The internal architecture of Egeria has two distinct layers. The Open Metadata Access Services layer supports the different types of user and use case. The Open Metadata Repository Services layer provides the unified view of metadata across distinct systems, using protocols and repositories for access and exchange of metadata objects. Egeria’s OMRS layer includes the ability to refer to remote objects or replicate cached copies of remote objects for performance and availability Egeria can store this distributed model in its own local repositories, which support the storing of: local objects, replicas of remote objects and proxy-references to remote objects.
• #17: This slide shows a physical embodiment of a cohort of OMAG Servers. An OMAG Server is a deployable unit of function and each OMAG Server can be configured to either run a set of OMAS services or support a repository, or a combination of these roles. An Egeria cohort is a collection of cooperating OMAG Servers. An OMAG Server may belong to multiple cohorts. The OMAS services are local to a server Each server runs the set of OMAS services listed in its configuration – it is OK to run 0, 1 or multiple OMAS services in a server Each OMAS is for a specific purpose or persona The OMRS protocol layer is supported by all servers The OMAG Servers use OMRS to access/exchange metadata across the cohort A server shares its metadata over OMRS – sending an event each time a change occurs, or sending a query to other servers A server may optionally maintain a local Egeria repository A server may optionally connect to a 3rd party metadata repository In a few slides we’ll see that the OMRS itself is composed of distinct layers that focus on cross-cohort (“Enterprise”) functions and Local functions.
• #18: The role of OMRS is to provide a location transparent, unified view of metadata within a cohort. Cross-cohort operations are supported by the OMRS ‘Enterprise Connector’, including sending queries to the cohort and receiving the results, as well as receiving replicated metadata and saving copies via the local connector. Meanwhile the ‘Local Connector’ handles interactions with an (optional) local repository and provides a default event mapper that sends events when the local state changes. The OMRS protocol uses publish/subscribe over Kafka topics, but the communication/messaging system is pluggable so different transports could be used. The interface to the repository connector is the MetadataCollection API _ which is described on the next slide….
• #19: Egeria’s model of metadata is graph-oriented, both at the business layer and beneath that in the structural metadata Business metadata describes the data that the business needs, what it means and how it should be classified and protected. Structural metadata describes how the data is actually stored and labelled in the data store. The linkages within and between the business and technical metadata forms a graph, that can be used to switch between these two perspectives. One of the built-in repositories in Egeria is a graph repository,; a natural fit for the metadata graph that also accommodates the distributed nature of OMRS. The Egeria local graph repository is built on the open-source JanusGraph graph database.
• #23: It may not always be practical to replicate an instance There are 2 occasions where using a proxy is advantageous: An OMAS wants to save a relationship in a repository and the replication has not happened yet (or the set up is such that replication of that type is not enabled). 2. The repository does not support the full entity type but does support proxies (all proxies have the same storage requirement). A key point about the distributed graph is that whether the relationship refers to a replica entity or uses an entity proxy – it is location transparent. The Enterprise OMRS layer can select which repository into which to save an instance – based on capability and proximity.
• #57: This is ambitious.
• #58: Beyond this is where we put stretch-goal material and deeper dive information.
• #62: ODPi
• #69: Business metadata describes the data that the business needs, what it means and how it should be classified and protected. Structural metadata describes how the data is actually stored and labelled in the data store. The linkage between the business and technical metadata allows our technology to switch between these two perspectives. For example, A request for data expressed in business terminology can be translated into a query for data from a data store. An integration engine copying data into a sand box can discover which are the fields that the business classifies as sensitive and then mask these values dynamically.
• #78: We’re not going to describe this interface in detail – but it’s worth being aware of it, especially as we’re going to talk later about the graph-queries in Group 3.
• #79: Egeria provides a persistent graph repository It’s built using JanusGraph and currently uses version 0.3.1 JanusGraph is an open source project hosted by the Linux Foundation that supports the Apache Tinkerpop 3.3 interface. The Egeria graph repository is built using the Egeria ‘plugin’ repository pattern – in which the repository connector is both the connector and the implementation of the repository. The graph repository supports instances originating locally, instances replicated from a remote server and proxy instances.
• #80: This slide shows (some of) the layers within an OMAG Server. We talked earlier about the access services and about the Enterprise Connector and Local Connectors within OMRS. Now we want to focus on the relationship between the Egeria graph repository connector and repository implementation (both in aqua-blue) and the JanusGraph code (in green) As far as possible the repository uses Apache Tinkerpop for graph operations. This is simply that – while we like JanusGraph – it is probably sensible to stay as far as possible with the Tinkerpop interface for possible future portability. There are some aspects of interacting with a graph database that are inherently implementation-specific – things like the configuration (e.g. of backends), schema and indexing. For these types of interaction it is necessary to use the JanusGraph Management interface.
• #83: Whilst you could look inside the graph for debugging or development – please don’t write code that relies on the schema as it is very likely to evolve The graph does not contain type information – Egeria provides a repository helper that manages types. The graph is used to store instance data - as described in mode detail on the following slides…
• #84: Here is an example of a number of OMRS instance objects – there are two entities, that are connected by a relationship. Also, one of the entities has two classifications. All of the instances have attributes – some will be core attributes used for type or control information; others will be attributes that are specific to the instance type (known as type-defined attributes). You don’t need to remember this picture – we’ll stick a copy of it in the top corner so we can refer back to it…..
• #85: Entities and classifications are vertices. Relationships and classifiers are edges. The graph schema defines labels for Entity, Relationship, Classification and Classifier. Vertex and edge properties are used to store OMRS instance data, which includes type, control and property information: Type is referenced by name – not linked by an edge; types are held in the repository content manager, not stored in the graph Control information is stored in ‘core attribute’ properties Instance properties are stored in serialized form and under unique custom keys to support search
• #86: Entities and classifications are vertices. Relationships and classifiers are edges. The graph schema defines labels for Entity, Relationship, Classification and Classifier. Vertex and edge properties are used to store OMRS instance data, which includes type, control and property information: Type is referenced by name – not linked by an edge; types are held in the repository content manager, not stored in the graph Control information is stored in ‘core attribute’ properties Instance properties are stored in serialized form and under unique custom keys to support search
• #89: Within Group 3 of the MDC API ….
• #92: Experts in a field with their own jargon and ways of doing things. Search report writer interested in assets and not security policies. Security policy author not interested in assets Goals tasks associated artifacts for a role.
• #94: 1 OMAS only e,g Subject area, the UI only uses the OMAS interfaces to communicate with Egeria 2 OMAS and connector e.g. VDC metadata is obtained from Egeria using OMAs calls, the actual date is accessed using an RDB connector 3 OMRS oriented UIs – e.g. Tex used to explore Egeria types 4 Daemon UIs – displaying Lineage
• #95: 1 OMAS only e,g Subject area, the UI only uses the OMAS interfaces to communicate with Egeria 2 OMAS and connector e.g. VDC metadata is obtained from Egeria using OMAs calls, the actual date is accessed using an RDB connector 3 OMRS oriented UIs – e.g. Tex used to explore Egeria types 4 Daemon UIs – displaying Lineage
• #96: For this to work we need to know hostname and ports and url structures. Configuration for tomcat is via application.properties Configuration of the server is held in a file and authored via admin rest calls.
• #99: Example here is the glossary grid. A grid for authoring glossaries in the subject area UI. Work in progress