Breaking Down a SQL Monolith with Change Tracking, Kafka and KStreams/KSQL
0 likes1,758 views
The document discusses the breakdown of a SQL monolith using change tracking, Kafka, and KStreams/KSQL, focusing on various aspects like monitoring database changes and Kafka topic configurations. It includes specific technical details about implementing change tracking with Microsoft SQL Server and Kafka integration strategies. The presentation concludes with a summary of the methodologies presented and offers contact information for Wanny Morellato, Director of Engineering at SAP Concur.
![© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "8
MSSQL Change Tracking
SELECT CT.user_id,
U.uuid user_uuid,
U.is_active is_active,
CT.SYS_CHANGE_VERSION,
CASE CT.SYS_CHANGE_OPERATION
WHEN 'I' THEN 'UserCreated'
WHEN 'U' THEN 'UserUpdated'
WHEN 'D' THEN 'UserDeleted'
END
CT.SYS_CHANGE_COLUMNS
FROM CHANGETABLE( CHANGES [dbo]. [users], @last_version )
AS CT
LEFT OUTER JOIN [dbo].[users] AS U WITH (NOLOCK)
ON CT.user_id = U.user_id
WHERE CT.SYS_CHANGE_VERSION < @max_version
ORDER BY CT.SYS_CHANGE_VERSION](https://image.slidesharecdn.com/07wannymorellato-181022184802/85/Breaking-Down-a-SQL-Monolith-with-Change-Tracking-Kafka-and-KStreams-KSQL-8-320.jpg){kind=tracking}
Breaking Down a SQL Monolith with Change Tracking, Kafka and KStreams/KSQL
- 1. Wanny Morellato, SAP Concur Director of Engineering October 16, 2018 Breaking Down A SQL Monolith with Change Tracking, Kafka and KStreams / KSQL
- 2. NOTE: Delete the yellow stickers when finished. Samples of a title slide with a pictogram are also available in this template. See SAP Image Library for other available images you can sort using the drop down menu and selecting illustration options. Wanny Morellato Director of Engineering @ SAP Concur @mrwanny Wanny.Morellato@sap.com #I am Hiring
- 3. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "3 Agenda
- 4. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "4 The Why MTTRHealth Scale
- 5. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "5 The What ReadsReads + Writes
- 6. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "6 The What Reads + WritesReads
- 7. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "7 The How Change Tracking Kafka Connect Kafka Topic Kafka Streams Micro Service
- 8. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "8 MSSQL Change Tracking SELECT CT.user_id, U.uuid user_uuid, U.is_active is_active, CT.SYS_CHANGE_VERSION, CASE CT.SYS_CHANGE_OPERATION WHEN 'I' THEN 'UserCreated' WHEN 'U' THEN 'UserUpdated' WHEN 'D' THEN 'UserDeleted' END CT.SYS_CHANGE_COLUMNS FROM CHANGETABLE( CHANGES [dbo]. [users], @last_version ) AS CT LEFT OUTER JOIN [dbo].[users] AS U WITH (NOLOCK) ON CT.user_id = U.user_id WHERE CT.SYS_CHANGE_VERSION < @max_version ORDER BY CT.SYS_CHANGE_VERSION
- 9. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "9 Kafka Connect public class MssqlSourceTask extends SourceTask { @Override public List<SourceRecord> poll() { ResultSet result = stmt.executeQuery(); return convert(result); }
- 10. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "10 Kafka Topic • One Topic per Table • Partition Size aligns to the Table change rate • Consistent Hashing partitioning based on a Table unique key • Replication Factor : 3 • Min InSync Replica : 2
- 11. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "11 Kafka KSQL CREATE TABLE users (user_id VARCHAR, user_uuid VARCHAR, is_active BOOLEAN, . . . ) WITH (KAFKA_TOPIC='users', VALUE_FORMAT='JSON', KEY = ‘user_uuid') WHERE is_active = 'true' ;
- 12. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "12 Kafka Stream Topology topology = new Topology(); topologyBuilder .addSource("SOURCE", stringDeserializer, deserializer, getTopic()) .addProcessor("FILTER",getFilter(),"SOURCE") .addProcessor("PROCESSOR", getWebhookProcessor(), "FILTER") .addSink(“SINK",getDeadLetterTopic(),serializer,"PROCESSOR"); StreamsConfig config = new StreamsConfig(env.getStreamProperties()); KafkaStreams streaming = new KafkaStreams(topology, config); streaming.start();
- 13. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "13 Summary Change Tracking Kafka Connect Kafka Topic Kafka Streams Micro Service
- 14. Thank You. Contact information: Wanny Morellato Director of Engineering SAP Concur @mrwanny Wanny.Morellato@sap.com
- 15. © 2018 SAP SE or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. The information contained herein may be changed without prior notice. Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation, and SAP SE’s or its affiliated companies’ strategy and possible future developments, products, and/or platforms, directions, and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, and they should not be relied upon in making purchasing decisions. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. All other product and service names mentioned are the trademarks of their respective companies. See www.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices. Learn more at concur.com Follow all of SAP Concur