Kafka and Kafka Streams in the Global Schibsted Data Platform
0 likes196 views
This document discusses Schibsted's use of Kafka and Kafka Streams in its global data platform. It describes how Schibsted processes streaming data using Kafka Streams to build a new streaming pipeline called Yggdrasil that provides low latency and delivery guarantees. It also discusses how Kafka Connect is used to get data into and out of the platform and how third party analytics tools can connect. The document outlines some challenges experienced in scaling up the platform.
Kafka and Kafka Streams in the Global Schibsted Data Platform
- 2. •Data Platform Architect City of Oslo •Former Data Engineer Data Platform FredrikVraalsen WHOAMI?
- 3. SCHIBSTED
- 4. SCHIBSTED 22 countries 200 million users/month 20 billion pageviews/month
- 10. THEISSUES • “High” latency (~30 seconds) • Delivery guarantees • On-boarding • Manual configuration • Homegrown solution
- 11. VISION • Data-driven applications • Self-serve • GDPR • Performance • State of the art
- 12. WHY ?
- 13. STREAMPROCESSING
- 14. • Lightweight library • Streams and Tables • High-level DSL • Low level API KAFKASTREAMS
- 15. YGGDRASILWASBORN
- 22. DATAQUALITY
- 24. GROWINGPAINS
- 25. SCALINGUP
- 29. SELFSERVE
- 30. SELFSERVE • Challenge: Transformations and routing • Multiple configurations • Who maintains? • Required Scala knowledge
- 31. { "time": round(parse-time(.published, "yyyy-MM-dd'T'HH:mm:ssX") * 1000), "device_manufacturer": .device.manufacturer, "device_model": .device.model, "language": .device.acceptLanguage, "os_name": .device.osType, "os_version": .device.osVersion, "platform": .device.platformType, "user_properties": { "is_logged_in" : boolean(.actor."spt:userId") } } SELFSERVE • JSLT – DSL for JSON transformation & queries https://github.com/schibsted/jslt
- 32. SELFSERVE
- 35. Data Platform – Oslo Origo
- 36. Oslo Origo • Digital transformation • Smarter services • Data driven
- 38. Data Platform – Oslo Origo https://www.flickr.com/photos/boaski/8079390195
- 39. Data Platform – Oslo Origo • Creating value from our data • Awareness of opportunities • Simple and safe data access • Insights and decision making • Collaboration and sharing