Sören Eickhoff, Informatica GmbH, "Informatica Intelligent Data Lake – Self Service for Data Analysts"

Editor's Notes

• #3: If your customer thinks of Informatica as an ETL company, this is a chance to change their perception. We are the #1 leader in 6 important data categories: First, cloud data management – we have a full portfolio of data management services for all the major cloud ecosystems – either cloud only or hybrid Data integration – our bread and butter – we have been the best at it for a long time and we continue to set the bar Big Data Management – we are the leader in data management for Big Data platforms. We work closely with all the major Hadoop, NoSQL ecosystems and with all the latest Big Data technologies like Spark Master Data Management – we are the leader in MDM for customer data and any other data that is important to their business. Our secret sauce is our matching engine, ability to discover relationships, and scalability. We can do this on any data platform, either on-premise or in the Cloud. Data Quality – we are setting the bar in DQ. Whether it is for stand-alone initiatives like data governance or for embedding data quality into their business processes Data Security – we are pioneering a new approach in security. Security remains an unsolved problem, and we can address it at the data level
• #4: Most organization are building out some version of a data lake or enterprise data hub concept. Really they are looking to get all their data into one place for next generation of analytics, ability for all people to have access to information They are usually divided up into multiple types of zones.
• #6: To serve these market trends best Informatica developed a Big Data solution that addresses each of the trends. The EIC module helps people understand the data they are looking at providing context The IDL module allows business to be more self service by providing self service data preparation capabilities, yet also helps IT operationalize the data preparation steps at scale in a managed and governed way. Secure@Source gives insight in potential risks around privacy sensitive data, by providing insight in where this data is located, how it is proliferated across the Data Lake (and surrounding applications) and what the associated risks are. Big Data Management helps customers ingest, parse, cleanse, integrate and deliver big data at scale. Intelligent Streaming finally allows processes to use realtime and streaming data sources. All this fucntionality is built as part of the Intelligent Data Platform where we try to use as much open source tools as possible leveraging the power of the ecosystem. We use Hbase to store different types of metadata, we use Titan as a graph database to store the relationship information between data assets. We use Spark (incl Spark Streaming) and Blaze to process data at Scale, we use Kafka as a high speed data transfer mechanism and finally we use Solr to index metadata so it can be searched using a google like search interface.
• #7: The Enterpsie Information Catalog (EIC) application allows business users to quicly find all information around the collection of data assets in their data lake. Since EIC can leverage the metadata provided by Cloudera Navigator we can even show Hive/Impala scripts and Pig scripts that are being used to process data.
• #8: Intelligent data lake provides capabilities to enable business users to do data preparation
• #9: Secure@Source gives insight in sensitive data risks.
• #10: Secure@Source gives insight in sensitive data risks.
• #11: Dynamic Mappings Build a template once – automate mapping execution for 1000’s of sources with different schemas automatically Mapping self adjusts dynamically to external schema changes and column characteristics Ability to process flat files with changing order (a,b,c or c,a,b) and number of columns dynamically Re-use PowerCenter and SQL logic Many customers have existing investments done in traditional powercenter and/or SQL scripts. To allow re-use of these components Informatica provides capabilities to migrate existing PowerCenter logic to run in Hadoop and to convert existing SQL code to Big Data mapping logic that can be executed at scale. Smart Optimizer In-built mapping optimizer that automatically tunes and re-arranges the mapping for high performance Early selection, Early projection, Mapping pruning, Semi-join, Join re-ordering, etc Automatic partitioning support based on statistics and other heuristics Advanced full pushdown optimization support including data ship join
• #12: Intelligent streaming aims to bring the following capabilities into the Informatica Platform: Real-time data ingestion from streaming data sources Rule evaluation and event triggering on a real-time data stream Real-time Data Integration: complex transforms, lookups, joins etc. in real time
• #15: Data Stewards are responsible for strategically managing data assets in the data lake and the enterprise ensuring high levels of data quality, integrity, availability, trustworthiness, and data security while emphasizing the business value of data. By building a catalog, classifying metadata and data definitions, maintaining technical and business rules and monitoring data quality, data stewards ensure data in the lake is consistent for use in the discovery zone and enterprise zone. As the inventory of technical and business metadata is established and data sets available, data architects must design robust scalable data lake architecture to meets the business goals of the marketing data lake.
• #16: Before we dive into the demo, lets look at some terminology, I will be using these terms quite a bit in the demo: Data Lake A data lake is a centralized repository of large volumes of structured and unstructured data. A data lake can contain different types of data, including raw data, refined data, master data, transactional data, log file data, and machine data. In Intelligent Data Lake, the data lake is a Hadoop cluster. Data Asset A data asset is data that you work with as a unit. Data assets can include items such as a flat file, table, or view. A data asset can include data stored in or outside the data lake. Project A project is a container stores data assets and worksheets. Data Preparation The process of combining, cleansing, transforming, and structuring data from one or more data assets so that it is ready for analysis. Recipe A recipe includes the list of input sources and the steps taken to prepare data in a worksheet. Data Publication data publication is the process of making prepared data available in the data lake. When you publish prepared data, Intelligent Data Lake writes the transformed input source to a Hive table in the data lake. Other analysts can add the published data to their projects and create new data assets. Or analysts can use a third-party business intelligence or advanced analytic tool to run reports to further analyze the published data.