Modern Data Platforms
1 like278 views
The document outlines the characteristics and reference architecture of a modern enterprise data and AI platform, emphasizing its discoverability, trustworthiness, interoperability, and security. It highlights the significance of an event-driven architecture using Apache Kafka as a key component and provides a real-world example related to car distribution. The platform aims to provide scalable, governed data services while utilizing Microsoft Azure services for data management and analytics.
Modern Data Platforms
- 1. What does a modern data platform for analytics in the company look like? München, 2021-02-10, Arne Roßmann
- 2. Arne Roßmann Education: - Computer Science (BSc) with Focus on Business Intelligence - International Management (MBA) Key Competences: - Enterprise Architecture - DevOps / DataOps - Big Data / Cloud Projects / Roles: - Chief Architect - DevOps Coach - Solution Architect - (before: Developer, Scrum Master, Project Manager, …)
- 3. § What are the characteristics of an Enterprise Data & AI Platform § How does a Reference Architecture for an Enterprise Data & AI Platform look like? § Why Kafka is a perfect key component in the modern platform? § A real world example § Q & A Agenda
- 4. 4 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 Enterprise Data & AI Platform Characteristics Discoverable Addressable Trustworthy (defined & monitored SLO) Self Service Inter Operable (governed by open standards) Secure
- 5. Reference Architecture for an Enterprise Data & AI Platform 5
- 6. Concept_Templates_April_2018.pptx System of Insights Approach 6 NEXT-GEN ENTERPRISE DATA & AI PLATFORM DATA CENTRIC MODELS BUSINESS DATA HUB DATA PROCESSING DATA CURATION STREAMING SOURCES BATCH SOURCES AI & ANALYTICS FOUNDATION SEARCH, AI & ANALYTICS OLAP PLATFORM MANAGEMENT & OPERATIONS PLATFORM SECURITY ACTIVE DIRECTORY ORCHESTRATION KEY VAULT MFA SECURITY CENTER OMS MONITOR STREAM PROCESSING STREAM SERVING STORE UNIVERSAL DATA LAKE DATA TRANSFORMATION DATA TRUST AZURE DEVOPS DATA INGESTION LANDING RAW DATA INGESTION TRANSFORMED DATA INGESTION STREAM INGESTION AI & ANALYTICS EXECUTION INTELLIGENT APPs SELF SERVICE & ENTERPRISE BI REAL-TIME APPs DATA SERVICES APP DATA SERVICES CUSTOM AI APPs AI & ANALYTICS SERVICES Multiple sources converge to a common Data & AI platform using Microsoft Azure centric services Goals: Consistent, repeatable methods and processes that can scale Govern locally where it matters, manage centrally AZURE DATA FACTORY AZURE DATA FACTORY AZURE DATABRICKS AZURE DATABRICKS AZURE DATABRICKS EVENT HUB KAFKA AZURE DATA LAKE AZURE DATA LAKE AZURE DATA LAKE COSMOS DB DATA LAKE SEARCH AZURE ML AAS ADLS POWER BI BOT FRAMEWORK COGNITIVE SERVICES APP SERVICE LOGIC APP FUNCTION SYNAPSE ANALYTICS MGMT & SECURITY AZURE INFORMATION PROTECTION AXON SECURE@SOURCE ENTERPRISE DATA CATALOG POLYBASE AZURE SYNAPSE ANALYTICS API APP
- 7. Kafka as a key component in the architecture 7
- 8. 8 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 Event Driven Architecture – an approach to solve the Big Ball of Mud problem https://www.confluent.io/blog/using-apache-kafka-drive-cutting-edge-machine-learning/ https://www.confluent.io/blog/changing-face-etl/
- 9. 9 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 Event Driven Architecture – “clean” architecture and AI integration https://www.confluent.io/blog/using-apache-kafka-drive-cutting-edge-machine-learning/ https://www.confluent.io/blog/changing-face-etl/ https://www.confluent.io/blog/importance-of-distributed-tracing-for-apache-kafka-based-applications/
- 10. A real world example 10
- 11. 11 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 A real world example on Car Distribution https://github.com/arossmann/metrics_exporter
- 12. 12 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 Summary & Q&A
- 13. 13 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 A Modern Enterprise Data & AI Platform The AI Engineering Data Centricity Platform Reference Architecture Discoverable Addressable Trustworthy (defined & monitored SLO) Self Service Inter Operable (governed by open standards) Secure AI & ANALYTICS EXECUTION DATA CENTRICITY FOUNDATION AI & ANALYTICS FOUNDATION SOURCE DATA PLATFORM FOUNDATION DATA TRUST Architecture & Advisory Modern Enterprise Data & AI Platform AI & ANALYTICS EXECUTION AI & Analytics Execution AI & ANALYTICS FOUNDATIONS DATA CENTRICITY FOUNDATION DATA TRUST Data Engine Virtual Semantic Layer & Caching Data Ingestion Data Landing Zone Raw Data Files Batch Ingestion Streaming Ingestion Data Governance Strategy and Process Reference & Master Data Data Lifecycle Management Data Privacy Data Quality & Profiling Data Discovery & Lineage Data Catalogue Entity Relationship Management Business Glossary Data Lake, Data Hub, EDW Business Data Hub Modeled data store Universal Data Lake Curated Data Store Data Transformation Data Enrichment Data Curation Custom AI Solutions Intelligent Apps AI & Analytics - Data Services Internal Data External Data CRM ERP ANALYTICAL OTHER THIRD PARTY SOCIAL DATA OTHER SENSOR IOT AI Monitoring Data Science Workbench Model Catalogue Analytics Orchestration KPI Catalogue Application - Data Services Traditional BI Reporting Data Centricity - Data Services PLATFORM FOUNDATION Cloud Foundations I&D Platform Foundations & Run Platform Security & Governance Network Connectivity Cyber Security Cloud Infrastructure Platform Management & Operations DevOps/DataOps Automation Hybrid Cloud Services Edge Integration Data Modeling Aggregated Fact APIs Visualization Data Exploration & Search Analytics & Dashboards Self-Serve BI Reporting, BI & Visualization
- 14. 14 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 SEC1 © 2020 Capgemini. All rights reserved. 14 Abendveranstaltung mit Capgemini Save the Date und sei bei unserem virtuellen Abendprogramm am 11.02.2021 von 18:00 Uhr bis 20:00 Uhr dabei. Es erwartet dich unter anderem eine Keynote „Coding at Capgemini – Revolution or Evolution for Clients” von unserem Head of Innovation Thilo Hermann. Erhalte spannende Einblicke und Beispiele, wie Capgemini seine Kunden revolutionär aber auch evolutionär mit Software unterstützt. Schnapp dir außerdem einen Drink für ein ausgelassenes Get-Together: lerne dabei unsere Bereiche kennen und vernetze dich mit Capgemini-Mitarbeitern. Du hast Interesse? Dann melde dich bis zum 11.02.2021 unter https://capgemini- events.de/abendveranstaltung_mit_capgemini/ an. Wir freuen uns auf dich!
- 15. 15 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 SEC1 © 2020 Capgemini. All rights reserved. 15 Let‘s get in contact and visit us now at our virtual booth/258 here on the OOP! Architecting Our Future
- 16. 16 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 Contact Roßmann, Arne Head of AI & Data Engineering Germany, Insights & Data Chief Architect Capgemini Nuremberg Bahnhofstr. 30 90402 Nuremberg, Germany arne.rossmann@capgemini.com https://www.linkedin.com/in/arnerossmann/
- 17. Capgemini is a global leader in consulting, digital transformation, technology, and engineering services. The Group is at the forefront of innovation to address the entire breadth of clients’ opportunities in the evolving world of cloud, digital and platforms. Building on its strong 50-year heritage and deep industry-specific expertise, Capgemini enables organizations to realize their business ambitions through an array of services from strategy to operations. A responsible and multicultural company of 265,000 people in nearly 50 countries, Capgemini’s purpose is to unleash human energy through technology for an inclusive and sustainable future. With Altran, the Group reported 2019 combined global revenues of €17 billion. About Capgemini Learn more about us at www.capgemini.com This presentation contains information that may be privileged or confidential and is the property of the Capgemini Group. Copyright © 2020 Capgemini. All rights reserved.