The Future of Data Pipelines
0 likes140 views
The document discusses the future of data pipelines, highlighting trends and predictions following the rise of pub-sub technologies like Kafka. Key topics include scalability, functionality, compliance, and design considerations for efficient data management. The emphasis is on adapting to growing data volumes and the need for real-time processing capabilities.
The Future of Data Pipelines
- 1. The Future of Data Pipelines | All Things Open 2020 | 2020-02-24 The Future of Data Pipelines John Hammink Developer Advocate, Evangelist @ Aiven john@aiven.io All Things Open Meetup 2020, Raleigh NC
- 2. The Future of Data Pipelines ● Let’s start with where the future looked from about a year ago ● Let’s assume that pub-sub and particularly technologies like Kafka have created a pivot point ● Let’s follow up with a discussion of trends we’re all seeing The Future of Data Pipelines | All Things Open 2020 | 2020-02-24
- 3. The Future of Data Pipelines ● Background ● Learnings ● Some numbers on data ● Functionality ● Design ● Compliance ● Reliability/Performance/Usability/Other ● Conclusions ● Q & A The Future of Data Pipelines | All Things Open 2020 | 2020-02-24
- 4. An arbitrary timeline of data The Future of Data Pipelines | All Things Open 2020 | 2020-02-24
- 5. What have we learned? The Future of Data Pipelines | All Things Open 2019 | 2019-10-14 Pipeline data has grown/transformed in terms of:
- 6. The Future of Data Pipelines | All Things Open 2019 | 2019-10-14
- 7. The Future of Data Pipelines | All Things Open 2019 | 2019-10-14 Core-to-edge computing
- 8. The Future of Data Pipelines | All Things Open 2019 | 2019-10-14 Functionality ● can autoscale, shard, tolerate partitions ● can capture, fix and requeue errored events ● can be troubleshot and configured on fly ● data can be used for AI/ML ● agnostic: accomodates all possible formats ● can self-perpetuate and automate continuous improvements
- 9. source: kafka.apache.org Source: Martin Kleppmann’s Kafka Summit 2018 Presentation, Is Kafka a Database? The Future of Data Pipelines | All Things Open 2019 | 2019-10-14
- 10. Other design considerations ● Kill switch - when things go wrong; a way to stem the data flow. ● Query on the streams - like KSQL, Apache Flink, SQL on Amazon Kinesis, etc. Ultimately: one query interface for all data. ● Need to accomodate - core-to-edge, growing data volume, near real-time velocity, bi-directionality ● Distributed ledgers - pipelines can support these as tunably consistent distributed datastores The Future of Data Pipelines | All Things Open 2019 | 2019-10-14
- 11. Compliance The Future of Data Pipelines | All Things Open 2019 | 2019-10-14
- 12. Reliability/Performance/Usability/Other The Future of Data Pipelines | All Things Open 2019 | 2019-10-14 ● MTTF → ∞ ● Components: from interpreted to native? ● Metadata: will continue to transform, to offer even more space savings. ● Pipelines: from hard-coded to drag and drop designs? ● Which data is left volatile and which is stored?
- 13. Wrapping up The Future of Data Pipelines | All Things Open 2019 | 2019-10-14 Check out The Future of Data Pipelines at https://aiven.io/blog/the-future-of-data-pipelines/
- 14. Discussion The Future of Data Pipelines | All Things Open 2019 | 2019-10-14