A Kafka journey and why migrate to Confluent Cloud?
1 like292 views
The document describes the development and features of frshub, an event-driven data streaming platform created by Fischer Retail Solutions, with a focus on integrating Apache Kafka for data processing and decision-making. It outlines the company's journey from initial implementation challenges with legacy systems to migrating to Confluent Cloud for enhanced capabilities. Additionally, it highlights the benefits of using Kafka and the ongoing development of the platform to improve business intelligence and operational insights.
A Kafka journey and why migrate to Confluent Cloud?
- 1. › frshub platform & services for decision-making A Kafka journey and why migrating to Confluent Cloud? Taucha / 21.04.2021
- 2. frshub | platform for decision-making 2 › Kevin Begrow › Chief Operating Officer | Kafka Expert at Fischer Retail Solutions GmbH › 35 years | Leipzig › 13 years in IT-Business › Working in DevOps & CloudOps environments › 16 Kafka clusters implemented in the last 5 years for enterprise and midmarket customers › Certified Confluent Developer › Since 2019 he and his team is developing the FRSHUB, an event driven data streaming platform for BI analyses and omnichannel retailing.
- 3. frshub | platform for decision-making 3 › Fischer Retail Solutions GmbH › Fischer Retail Solutions GmbH was founded in 2017 in Leipzig / Taucha › Started with 2, now having 16 employees across europe › Confluent RSI Partner since 2022 › 2 pilot customers from the retail sector (Medium-sized business) › MEIN FISCHER: With 21 stores in central Germany and a 190-year history, MEIN FISCHER is one of the best-known fashion houses in central Germany. › Implementation of a digitalization strategy, including performance management, business analyses, online store strategy and more…
- 4. frshub | platform for decision-making 4 › What is frshub › In addition to our consulting services for optimal digital transformation, we offer frshub, a proprietary data platform that interprets data from multiple sources to show real-time and predictive views of operations. Using dynamic targets and dashboards, this platform gives your team a clear picture of operations across your network so they can make course corrections to positively impact performance. Our data services empower your staff at all levels to make informed and accurate decisions. Also additional modules for fullfilment solutions (ISTA), product information management (PIM) and more using FRSHUB. ERP, CRM, LEGACY SYSTEMS DATABASES, DATA WAREHOUSE, OLAP CUBES APPLICATION ENDPOINTS & SERVICES, DATA STREAMS CLOUD DATA, BIG DATA, IoT FLAT FILES (EXCEL, TXT, CSV) Business Intelligence KPI‘s Dashboard Reporting Analysis Performance Management Budgeting, Planning & Forecasting Profitability Analysis Strategy Management Consolidation
- 5. frshub | platform for decision-making 5 › Our Clients › companies 10 who use frshub solutions › in the following departments › Customer Relationship Management › E-commerce › Finance › HR › Marketing › Operations › Procurement › Sales › Gastronomy › Site / Channel planning › Supply Chain Management › to achieve a turnover of ≈300M Euro ( companie sizes from 1-150 M Euro )
- 6. frshub | platform for decision-making 6 › 2017 – 2018: How it started… there was no Apache Kafka at all › Azure migration project started › Implemtation of a Java based ETL Dataclient + Middleware for product data and stock management from legacy datasources › Development ISTA App (Fulfillment / Stock Management App) › CSV exports have been used for Dashboards to generate sales reportings, customer data, product informations etc. à no near-, or real-time at all. › Onlineshop: CSV export/import for stock quantity and product updates, as well as orders à Update once per hour
- 7. frshub | platform for decision-making 7 › 2019 – 2021 FRSHUB – Kafka as the backbone was introduced › Infrastructure as code with Terraform + configuration management with Ansible › All components are container based (docker containers on scale sets later moved to K8s) › Apache Kafka (2.2.0) was introduced in 2019, Blue/Green deployment followed in 2020 › 3 x Brokers › 3 x Kafka Connect workers › 3 x Schema Registry nodes › Later in 2020 - Kafka REST Proxy, ksqlDB were introduced › Custom Development of Kafka REST Connector › ESB: WSO2 was implemented as Open REST API
- 8. frshub | platform for decision-making 8 › 2022 – migration to Confluent Cloud › Short Facts of PROD Environment › Autoscalable nodes for Broker, ksqlDB, Kafka Connect, Rest Proxy › 275 Topics › 8056 Partitions › 23932 Replicas › Multiple Availability Zones › DEV / STAGING à already in Confluent Cloud › PROD › Blocker: Terraform Provider (solved soon)! › Goal: fully migrated by mid 2023
- 9. frshub | platform for decision-making 9 › FRSHUB Infrastructure
- 10. frshub | platform for decision-making 10 › FRSHUB uses Domain Driven Design FRSHUB Event Streaming Platform ERP | CRM Integration Legacy Integration BI Analytics Kafka Connect ETL Data Client Kafka Streams Kafka Cluster Schema Registry CRM | ERP Domain Legacy Domain Sales Domain Open API | REST Endpoints
- 11. frshub | platform for decision-making 11 › FRSHUB uses Domain Driven Design DDD calls for an anti-corruption layer: • a layer that maps the data used in a domain model to the data • transmitted between different microservices or different bounded contexts • This pattern can be used regardless of whether the services communicate through events or with a request-response protocol. • In both cases, there is typically a wire format like REST endpoint or the schema used to describe an event which will be stored in Schema Registry
- 12. frshub | platform for decision-making 12 › Challenges we have faced in our journey › ETL / ESB Middelware vs. Apache Kafka – What do we need, what can we implement? › Dealing with a lot of legacy systems without any API or in best cases SOAP or REST › No direct access to the systems › Implementing event driven streaming ( a sale, a new customer etc.) where ever we have direct access › Offering a REST API to push data to our platform when ever a event occurs (cash desk partner integration) › Using exiting REST API of Systems with REST Connector › Still customers want to export CSV manually and upload data to specific folders › CSV / Flat Files – nobody wants it, everybody has it › Huge CSV files, created overnight › CSV files from different customers, using different delimiters, data formats or changing header à a lot of transformation was needed to fit into defined schemas › The powerful file-pulse connector of streamthoughts was a huge improvement to handle those issues.
- 13. frshub | platform for decision-making 13 › Challenges we have faced in our journey › REST Services – handling schemas › REST connector only give a JSON formatted string à schema validation wasn‘t possible in this case › Defining schemas in ksqlDB for further transformations and aggregations of data › Since 2021 ksqlDB is heavily used with joining streams with lookup tables, merging streams, aggregating and filter out unneccesary data
- 14. frshub | platform for decision-making 14 › Why do we love Confluent Cloud? › Kafka is the De Facto Standard API for Event Streaming › Focus on our actual USP of event streaming and business usecases › Get free resources for other infrastructure tasks › Managable with our exiting IAC * › The main contributor to the Apache Kafka project with 80% of Kafka commits › Always the latest Kafka version (without limitations) and full support › Rich Kafka ecosystem (connectors, governance, security, etc.) › Hybrid architectures (including the only true fully-managed and complete Kafka service) › Hosting in Azure
- 15. Thank you.