Verifying Apache Kafka-Based Data Pipelines With Subhangi Agarwala | Current 2022
0 likes415 views
This document discusses the importance of testing streaming applications in finance, specifically focusing on Apache Kafka-based data pipelines at Bloomberg. It covers different testing methodologies, including unit testing and integration testing, and outlines a testing framework using Docker containers to ensure the reliability of Kafka components. The document concludes with an acknowledgment of the challenges involved in testing and provides resources for further engagement with Bloomberg's derivatives team.
Verifying Apache Kafka-Based Data Pipelines With Subhangi Agarwala | Current 2022
- 1. © 2018 Bloomberg Finance L.P. All rights reserved. © 2022 Bloomberg Finance L.P. All rights reserved. Verifying Apache Kafka-based Data Pipelines Current 2022 October 5, 2022 Subhangi Agarwala Senior Software Engineer
- 2. © 2022 Bloomberg Finance L.P. All rights reserved. • Bloomberg • Derivatives Data Engineering — Data ingestion, storage and aggregation • We build real-time financial market data pipelines About Us
- 3. © 2018 Bloomberg Finance L.P. All rights reserved. © 2022 Bloomberg Finance L.P. All rights reserved. Agenda ● Background and our use case ● Why we need to test streaming applications in finance ● Integration test framework ● Conclusion
- 4. © 2022 Bloomberg Finance L.P. All rights reserved. Real-Time Data Pipelines in Finance – and at Bloomberg
- 5. © 2022 Bloomberg Finance L.P. All rights reserved. Application Architecture Config Publisher Trading Data Publisher Kafka Connect Persistent data store Kafka Streams Client App Analytical Engine Real Time analysis Config Updates Config Changes Trading Data
- 6. © 2018 Bloomberg Finance L.P. All rights reserved. © 2022 Bloomberg Finance L.P. All rights reserved. ● Unit Testing ○ Easily test new functional logic in isolation ○ Insufficient for larger workflows ● Integration Testing ○ Handle exceptions and failures where different moving parts are involved ○ Reliability Unit Testing vs. Integration Testing
- 7. © 2018 Bloomberg Finance L.P. All rights reserved. © 2022 Bloomberg Finance L.P. All rights reserved. Testing Framework Test Validator TestContainers using Docker (Producer) TestContainers using Docker (Consumer) TestContainers using Docker (test Kafka Connect) Kafka Kafka KConnect
- 8. © 2018 Bloomberg Finance L.P. All rights reserved. © 2022 Bloomberg Finance L.P. All rights reserved. ● Asynchronous nature of Test Containers ○ Increase in test runtime ● Ensuring consistency between components ○ Add a marker at the end of the input record Challenges m0 Producer m0 m0 c m1 m2 m1 c c
- 9. © 2018 Bloomberg Finance L.P. All rights reserved. © 2022 Bloomberg Finance L.P. All rights reserved. ● Testing interaction between a Kafka producer and a Kafka consumer. ● Producing and consuming raw records and JSON records. ● Combining REST API testing with Kafka testing. ● Spinning up Kafka components - Kafka, KConnect and KStreams in a TestContainer. Conclusion
- 10. © 2018 Bloomberg Finance L.P. All rights reserved. © 2022 Bloomberg Finance L.P. All rights reserved. Thank you! https://www.bloomberg.com/careers Learn more about our Derivatives team: https://www.bloomberg.com/company/stories/meet-the-team-derivativ es-engineering/ Subhangi Agarwala: sagarwala10@bloomberg.net