Kafka Case by getting data from SAP
0 likes31 views
The document outlines a deployment training for Apache Kafka focused on integrating on-premises and cloud environments, specifically addressing challenges faced by Compass Minerals in Brazil. It details the solution leveraging Kafka with Debezium connectors to automate the movement and joining of data for generating invoices from an SAP system to a cloud customer portal. The results include reduced manual workload, minimized data loss risks, and streamlined data integration processes.
Kafka Case by getting data from SAP
- 2. Apache Kafka deployment Training based on Apache Kafka Healthcheck services for Kafka Open Source Consulting Service for analyzing Kafka environment TECHROM : Monetizing the database through data streaming Techrom is a Brazilian startup focused on Apache Kafka technology service
- 3. Business Case Setting a bridge between on-premises and cloud-computing environments by using Kafka
- 4. Scenario Compass Minerals is an American producer of minerals company which has a subsidiary in Brazil, has started to create the customer portal to simplify the information accesses such as invoice, delivery schedule, payments due, production status, etc. Challenge • Getting the invoice from SAP system which is created on- premises and to be available at the customer portal on cloud- computing (Microsoft Azure) • Joining three tables which have the customer information to generate the invoice • Integrating legacy systems • Avoiding the data lost Solution • Deploying Kafka with Debezium connector to get data from SQL Server ( on-premises ) and be available thru Webservice on Cloud-Computing • Using the KSQL to check (query) updates from the three topics and join in a new topic to emit the invoice • Setting the Acknowledge (Acks) resource to make sure the message would be processed Results • Simplifying the data movement between on-premises and cloud-computing environments • Automatizing the joining data between tables and eliminating the manual workload • Creating the datahub to integrate the legacy system • Decreasing the data lost risk
- 5. 1 2 SAP generates the invoice files thru three tables It is creates 3 topics KSQL Consumer generates the invoiceProducer read PDF file and send the file serialized One new topic is created after KSQL transformation 1 topic (join) KSQL Kafka Architecture Notes 1 - SAP sends the invoice files to SQL Server. Debezium connector writes the data to Kafka Broker via the producer. KSQL resource makes a query of invoice data on the three topics and joins it in a new topic to generate the invoice 2 - Kafka Consumer reads the data from Kafka Broker and the Webservice generates the invoice.