SlideShare a Scribd company logo

Flink Snapshots: A Comprehensive Guide for New Users

HostedbyConfluent, profile picture
HostedbyConfluent

The document is a comprehensive guide on 'Flink Snapshots' aimed at new users, detailing the processes related to stateful processing, checkpoints, and state backends. It covers essential topics such as exactly-once processing, checkpoint lifecycles, and configuration settings to optimize performance. Common problems and solutions are also discussed, along with practical examples and concepts like buffer debloating and incremental checkpoints.

Technology
Related topics:
Kafka Summit • Kafka Summit London • Apache Flink
1 of 48
Downloaded 11 times
1
2
3
Most read
4
5
6
Most read
7
8
9
10
11
12
13
Most read
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Flink Snapshots A Comprehensive Guide for New Users
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Flink Snapshots A Comprehensive Guide for New Users Danny Cranmer Principal Engineer at AWS Apache Flink PMC Member
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda 1. Stateful processing recap 2. Flink Checkpoints 3. State backends 4. Common problems 3
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 4 Simple Flink Example SELECT SUM(clicks) FROM MyKafkaTopic 2 5 7 3 4 2 2 Input Output 7 14 17 21 23
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 5 Simple Flink Example 3 4 2 7 14 2 14 o6 o5 o4 o3 o2 o1 o4 In flight state
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 6 Exactly once processing
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 7 At least once processing - Duplicates
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 8 At least once processing - Duplicates
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 9 At most once processing – Dropped Records
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 10 At most once processing – Dropped Records
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda 1. Stateful processing recap 2. Flink Checkpoints 3. State backends 4. Common problems 11
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 12 Flink Checkpoints
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 13 Flink Checkpoints
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 14 Flink Checkpoints
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 15 Flink Checkpoints
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 16 Barrier Alignment
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 17 Barrier Alignment
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 18 Barrier Alignment
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 19 Checkpoint Lifecycle
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 20 Checkpoint Lifecycle
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 21 Checkpoint Lifecycle
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 22 Checkpoint Lifecycle
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 23 Checkpointing Configuration execution.checkpointing.interval execution.checkpointing.min-pause execution.checkpointing.timeout
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 24 Checkpoint Statistics Demo
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 25 Checkpoint vs Savepoint
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda 1. Stateful processing recap 2. Flink Checkpoints 3. State backends 4. Common problems 26
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 27 State Backends
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. State Backend Selection 28 state size < sum(Task Manager memory) / ? ? HashMap : RocksDB
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda 1. Stateful processing recap 2. Flink Checkpoints 3. State backends 4. Common problems 29
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 30 Bottlenecks
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 31 Buffer Debloating
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 32 Buffer Debloating - Disabled
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 33 Buffer Debloating - Disabled
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 34 Buffer Debloating - Disabled
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 35 Buffer Debloating - Disabled
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 36 Buffer Debloating - Disabled
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 37 Buffer Debloating - Disabled
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 38 Buffer Debloating - Enabled
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 39 Buffer Debloating - Enabled
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 40 Buffer Debloating - Enabled
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 41 Buffer Debloating taskmanager.network.memory.buffer-debloat.enabled: true taskmanager.network.memory.buffer-debloat.target: 1s
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 42 Unaligned Checkpoints - Checkpoint barriers jump the queue - Records in buffers stored in the checkpoint - Not supported for savepoints
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 43 Unaligned Checkpoints env.getCheckpointConfig().enableUnalignedCheckpoints(); execution.checkpointing.unaligned: true
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 44 Incremental Checkpoints
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 45 Incremental Checkpoints env.setStateBackend(new RocksDBStateBackend(filebackend, true)); state.backend.incremental: true
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Load Test/Skew 46
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Load Test/Skew 47
FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thank you! © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Danny Cranmer Principal Engineer at AWS Apache Flink PMC Member

More Related Content

PDF
Making Sense of Apache Flink: A Fearless Introduction
HostedbyConfluent
 
PDF
Getting Data In and Out of Flink - Understanding Flink and Its Connector Ecos...
HostedbyConfluent
 
PDF
Deeply Declarative Data Pipelines
HostedbyConfluent
 
PDF
What's the time? ...and why? (Mattias Sax, Confluent) Kafka Summit SF 2019
confluent
 
PDF
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Databricks
 
PDF
Introducing the Apache Flink Kubernetes Operator
Flink Forward
 
PDF
Apples and Oranges - Comparing Kafka Streams and Flink with Bill Bejeck
HostedbyConfluent
 
PDF
Spark shuffle introduction
colorant
 
Making Sense of Apache Flink: A Fearless Introduction
HostedbyConfluent
 
Getting Data In and Out of Flink - Understanding Flink and Its Connector Ecos...
HostedbyConfluent
 
Deeply Declarative Data Pipelines
HostedbyConfluent
 
What's the time? ...and why? (Mattias Sax, Confluent) Kafka Summit SF 2019
confluent
 
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Databricks
 
Introducing the Apache Flink Kubernetes Operator
Flink Forward
 
Apples and Oranges - Comparing Kafka Streams and Flink with Bill Bejeck
HostedbyConfluent
 
Spark shuffle introduction
colorant
 

What's hot (20)

PDF
Improving Apache Spark's Reliability with DataSourceV2
Databricks
 
PPTX
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Flink Forward
 
PPTX
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
Flink Forward
 
PDF
Understanding Data Consistency in Apache Cassandra
DataStax
 
PDF
Virtual Flink Forward 2020: Autoscaling Flink at Netflix - Timothy Farkas
Flink Forward
 
PDF
Flink 2.0: Navigating the Future of Unified Stream and Batch Processing
HostedbyConfluent
 
PPTX
Evening out the uneven: dealing with skew in Flink
Flink Forward
 
PPTX
Real-time Stream Processing with Apache Flink
DataWorks Summit
 
PDF
The Power of SPL
Splunk
 
PDF
Introduction To Flink
Knoldus Inc.
 
PDF
Changelog Stream Processing with Apache Flink
Flink Forward
 
PDF
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Databricks
 
PDF
Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...
DataStax
 
PPTX
Stream Processing Frameworks
SirKetchup
 
PDF
Dive into PySpark
Mateusz Buśkiewicz
 
PDF
Tzu-Li (Gordon) Tai - Stateful Stream Processing with Apache Flink
Ververica
 
PDF
Apache flink
pranay kumar
 
PPTX
Where is my bottleneck? Performance troubleshooting in Flink
Flink Forward
 
PDF
Hardening Kafka Replication
confluent
 
PDF
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Databricks
 
Improving Apache Spark's Reliability with DataSourceV2
Databricks
 
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Flink Forward
 
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
Flink Forward
 
Understanding Data Consistency in Apache Cassandra
DataStax
 
Virtual Flink Forward 2020: Autoscaling Flink at Netflix - Timothy Farkas
Flink Forward
 
Flink 2.0: Navigating the Future of Unified Stream and Batch Processing
HostedbyConfluent
 
Evening out the uneven: dealing with skew in Flink
Flink Forward
 
Real-time Stream Processing with Apache Flink
DataWorks Summit
 
The Power of SPL
Splunk
 
Introduction To Flink
Knoldus Inc.
 
Changelog Stream Processing with Apache Flink
Flink Forward
 
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Databricks
 
Apache Cassandra Multi-Datacenter Essentials (Julien Anguenot, iLand Internet...
DataStax
 
Stream Processing Frameworks
SirKetchup
 
Dive into PySpark
Mateusz Buśkiewicz
 
Tzu-Li (Gordon) Tai - Stateful Stream Processing with Apache Flink
Ververica
 
Apache flink
pranay kumar
 
Where is my bottleneck? Performance troubleshooting in Flink
Flink Forward
 
Hardening Kafka Replication
confluent
 
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Databricks
 
Ad

Similar to Flink Snapshots: A Comprehensive Guide for New Users (20)

PDF
Hands-On Lab: Test Drive the Enhanced HTML5 Dashboard Designer With Release 8...
CA Technologies
 
PDF
Eclipse-Based User Interfaces for CA Endevor SCM and CA Testing Tool
CA Technologies
 
PDF
robsable_Enhancing DevOps Practices with CloudWatch APM FINAL.pdf
AWS Chicago
 
PDF
Apache JMeter from the Ground Up
CA Technologies
 
PDF
AWS Reinvent 2020 - Recap Amazon Builder's Library session
Guillaume Marchand
 
PDF
Airheads dallas 2011 rap troubleshooting
Aruba, a Hewlett Packard Enterprise company
 
PPTX
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
elesangwon
 
PDF
Hands-On Labs: Introduction to CA Unified Infrastructure Management
CA Technologies
 
PDF
Hands-On Lab: From Zero to Compliance Using CA Software Asset Management
CA Technologies
 
PDF
Pre-Con Ed: Build Your Own Apps for an Enhanced Network Management Experience...
CA Technologies
 
PDF
What's new in AWS for developers? - AWS reInvent recap 2025
Matthew Lewis
 
PPTX
Generating a custom Ruby SDK for your web service or Rails API using Smithy
g2nightmarescribd
 
PDF
Real World Problem Solving Using Application Performance Management 10
CA Technologies
 
PDF
Implementation and Use of Generic VTAM Resources with Parallel SYSPLEX Features
CA Technologies
 
PDF
Hands-On Lab: Tune CA Performance Management for an Optimal Network Performan...
CA Technologies
 
PDF
Continuous Delivery on AWS with Zero Downtime
Casey Lee
 
PDF
Extending Jenkins to the Mainframe. A Simpler Approach.
DevOps.com
 
PPTX
All levels of performance testing and monitoring in web-apps
Andrii Skrypnychenko
 
PDF
Case Study: VF Corporation Takes a Practical Approach to Improving its MOJO w...
CA Technologies
 
PPTX
apidays LIVE New York 2021 - APIOps: automating API operations for speed and ...
apidays
 
Hands-On Lab: Test Drive the Enhanced HTML5 Dashboard Designer With Release 8...
CA Technologies
 
Eclipse-Based User Interfaces for CA Endevor SCM and CA Testing Tool
CA Technologies
 
robsable_Enhancing DevOps Practices with CloudWatch APM FINAL.pdf
AWS Chicago
 
Apache JMeter from the Ground Up
CA Technologies
 
AWS Reinvent 2020 - Recap Amazon Builder's Library session
Guillaume Marchand
 
Airheads dallas 2011 rap troubleshooting
Aruba, a Hewlett Packard Enterprise company
 
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
elesangwon
 
Hands-On Labs: Introduction to CA Unified Infrastructure Management
CA Technologies
 
Hands-On Lab: From Zero to Compliance Using CA Software Asset Management
CA Technologies
 
Pre-Con Ed: Build Your Own Apps for an Enhanced Network Management Experience...
CA Technologies
 
What's new in AWS for developers? - AWS reInvent recap 2025
Matthew Lewis
 
Generating a custom Ruby SDK for your web service or Rails API using Smithy
g2nightmarescribd
 
Real World Problem Solving Using Application Performance Management 10
CA Technologies
 
Implementation and Use of Generic VTAM Resources with Parallel SYSPLEX Features
CA Technologies
 
Hands-On Lab: Tune CA Performance Management for an Optimal Network Performan...
CA Technologies
 
Continuous Delivery on AWS with Zero Downtime
Casey Lee
 
Extending Jenkins to the Mainframe. A Simpler Approach.
DevOps.com
 
All levels of performance testing and monitoring in web-apps
Andrii Skrypnychenko
 
Case Study: VF Corporation Takes a Practical Approach to Improving its MOJO w...
CA Technologies
 
apidays LIVE New York 2021 - APIOps: automating API operations for speed and ...
apidays
 
Ad

More from HostedbyConfluent (20)

PDF
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
HostedbyConfluent
 
PDF
Renaming a Kafka Topic | Kafka Summit London
HostedbyConfluent
 
PDF
Evolution of NRT Data Ingestion Pipeline at Trendyol
HostedbyConfluent
 
PDF
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
HostedbyConfluent
 
PDF
Exactly-once Stream Processing with Arroyo and Kafka
HostedbyConfluent
 
PDF
Fish Plays Pokemon | Kafka Summit London
HostedbyConfluent
 
PDF
Tiered Storage 101 | Kafla Summit London
HostedbyConfluent
 
PDF
Building a Self-Service Stream Processing Portal: How And Why
HostedbyConfluent
 
PDF
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
HostedbyConfluent
 
PDF
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
HostedbyConfluent
 
PDF
Navigating Private Network Connectivity Options for Kafka Clusters
HostedbyConfluent
 
PDF
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
HostedbyConfluent
 
PDF
Explaining How Real-Time GenAI Works in a Noisy Pub
HostedbyConfluent
 
PDF
TL;DR Kafka Metrics | Kafka Summit London
HostedbyConfluent
 
PDF
A Window Into Your Kafka Streams Tasks | KSL
HostedbyConfluent
 
PDF
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
HostedbyConfluent
 
PDF
Data Contracts Management: Schema Registry and Beyond
HostedbyConfluent
 
PDF
Code-First Approach: Crafting Efficient Flink Apps
HostedbyConfluent
 
PDF
Debezium vs. the World: An Overview of the CDC Ecosystem
HostedbyConfluent
 
PDF
Beyond Tiered Storage: Serverless Kafka with No Local Disks
HostedbyConfluent
 
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
HostedbyConfluent
 
Renaming a Kafka Topic | Kafka Summit London
HostedbyConfluent
 
Evolution of NRT Data Ingestion Pipeline at Trendyol
HostedbyConfluent
 
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
HostedbyConfluent
 
Exactly-once Stream Processing with Arroyo and Kafka
HostedbyConfluent
 
Fish Plays Pokemon | Kafka Summit London
HostedbyConfluent
 
Tiered Storage 101 | Kafla Summit London
HostedbyConfluent
 
Building a Self-Service Stream Processing Portal: How And Why
HostedbyConfluent
 
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
HostedbyConfluent
 
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
HostedbyConfluent
 
Navigating Private Network Connectivity Options for Kafka Clusters
HostedbyConfluent
 
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
HostedbyConfluent
 
Explaining How Real-Time GenAI Works in a Noisy Pub
HostedbyConfluent
 
TL;DR Kafka Metrics | Kafka Summit London
HostedbyConfluent
 
A Window Into Your Kafka Streams Tasks | KSL
HostedbyConfluent
 
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
HostedbyConfluent
 
Data Contracts Management: Schema Registry and Beyond
HostedbyConfluent
 
Code-First Approach: Crafting Efficient Flink Apps
HostedbyConfluent
 
Debezium vs. the World: An Overview of the CDC Ecosystem
HostedbyConfluent
 
Beyond Tiered Storage: Serverless Kafka with No Local Disks
HostedbyConfluent
 

Recently uploaded (20)

PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
KodekX | Application Modernization Development
KodekX
 
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
PPTX
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
PDF
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
Encapsulation theory and applications.pdf
gurumoop
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
KodekX | Application Modernization Development
KodekX
 
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 

Flink Snapshots: A Comprehensive Guide for New Users

  • 1. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Flink Snapshots A Comprehensive Guide for New Users
  • 2. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Flink Snapshots A Comprehensive Guide for New Users Danny Cranmer Principal Engineer at AWS Apache Flink PMC Member
  • 3. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda 1. Stateful processing recap 2. Flink Checkpoints 3. State backends 4. Common problems 3
  • 4. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 4 Simple Flink Example SELECT SUM(clicks) FROM MyKafkaTopic 2 5 7 3 4 2 2 Input Output 7 14 17 21 23
  • 5. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 5 Simple Flink Example 3 4 2 7 14 2 14 o6 o5 o4 o3 o2 o1 o4 In flight state
  • 6. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 6 Exactly once processing
  • 7. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 7 At least once processing - Duplicates
  • 8. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 8 At least once processing - Duplicates
  • 9. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 9 At most once processing – Dropped Records
  • 10. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 10 At most once processing – Dropped Records
  • 11. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda 1. Stateful processing recap 2. Flink Checkpoints 3. State backends 4. Common problems 11
  • 12. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 12 Flink Checkpoints
  • 13. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 13 Flink Checkpoints
  • 14. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 14 Flink Checkpoints
  • 15. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 15 Flink Checkpoints
  • 16. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 16 Barrier Alignment
  • 17. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 17 Barrier Alignment
  • 18. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 18 Barrier Alignment
  • 19. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 19 Checkpoint Lifecycle
  • 20. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 20 Checkpoint Lifecycle
  • 21. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 21 Checkpoint Lifecycle
  • 22. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 22 Checkpoint Lifecycle
  • 23. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 23 Checkpointing Configuration execution.checkpointing.interval execution.checkpointing.min-pause execution.checkpointing.timeout
  • 24. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 24 Checkpoint Statistics Demo
  • 25. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 25 Checkpoint vs Savepoint
  • 26. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda 1. Stateful processing recap 2. Flink Checkpoints 3. State backends 4. Common problems 26
  • 27. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 27 State Backends
  • 28. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. State Backend Selection 28 state size < sum(Task Manager memory) / ? ? HashMap : RocksDB
  • 29. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda 1. Stateful processing recap 2. Flink Checkpoints 3. State backends 4. Common problems 29
  • 30. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 30 Bottlenecks
  • 31. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 31 Buffer Debloating
  • 32. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 32 Buffer Debloating - Disabled
  • 33. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 33 Buffer Debloating - Disabled
  • 34. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 34 Buffer Debloating - Disabled
  • 35. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 35 Buffer Debloating - Disabled
  • 36. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 36 Buffer Debloating - Disabled
  • 37. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 37 Buffer Debloating - Disabled
  • 38. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 38 Buffer Debloating - Enabled
  • 39. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 39 Buffer Debloating - Enabled
  • 40. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 40 Buffer Debloating - Enabled
  • 41. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 41 Buffer Debloating taskmanager.network.memory.buffer-debloat.enabled: true taskmanager.network.memory.buffer-debloat.target: 1s
  • 42. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 42 Unaligned Checkpoints - Checkpoint barriers jump the queue - Records in buffers stored in the checkpoint - Not supported for savepoints
  • 43. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 43 Unaligned Checkpoints env.getCheckpointConfig().enableUnalignedCheckpoints(); execution.checkpointing.unaligned: true
  • 44. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 44 Incremental Checkpoints
  • 45. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 45 Incremental Checkpoints env.setStateBackend(new RocksDBStateBackend(filebackend, true)); state.backend.incremental: true
  • 46. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Load Test/Skew 46
  • 47. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Load Test/Skew 47
  • 48. FLINK SNAPSHOTS: A COMPREHENSIVE GUIDE FOR NEW USERS © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thank you! © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Danny Cranmer Principal Engineer at AWS Apache Flink PMC Member