Apache Ratis - A High Performance Raft Library

Bratislava, Slovakia. June 3-5, 2024
Apache Ratis
A High Performance Raft Library
Tsz-Wo Nicholas Sze
2024.6.3

About Me
Tsz-Wo Nicholas Sze, Ph.D.
● Software Engineer / Amateur Mathematician
● PMC member of Apache Hadoop, Ozone & Ratis
● Fun facts
○ Used Hadoop to create a π computation record (2010)
■ https://www.bbc.com/news/technology-11313194
○ Discovered a deterministic primality proving algorithm for Proth numbers (2008)
■ https://en.wikipedia.org/wiki/Proth_prime#Primality_testing

Agenda
● A Brief Introduction of Raft
● Apache Ratis Community
● Apache Ratis Features

5
© 2024 Cloudera, Inc. All rights reserved.
Consensus
● What is consensus?
○ Multiple servers to agree a value
● Typical use cases
○ Log replication
○ Replicated state machines (high availability)

6
Consensus Algorithms
● Paxos (1990)
○ Works but hard to understand
○ Hard to implement (correctly)
● Raft (2014) – “In Search of an Understandable Consensus Algorithm”
○ Easy to understand
○ Easy to prove
○ Easy to implement

7
Raft Basic – Leader Election
● Servers are started as Followers
● Randomly timeout to become a Candidate and start a leader election
○ Candidate sends requestVote to other servers
○ A server votes for a Candidate only if the candidate has a up-to-date state.
● The Candidate becomes the Leader once it gets a majority of the votes
○ Impossible to have two Leaders at the same time.
RPC: requestVote

8
Raft Basic – Log Replication
● Clients send requests to the Leader
○ A non-leader will reject client requests.
● Leader forwards requests as log entries to the Followers via appendEntries.
○ Once the Leader has obtained a majority, a request is committed
○ Committed requests are safe regardless of failures or restarts.
● Heartbeat to maintain leadership
○ The Leader sends empty appendEntries to Followers to when idle.
RPC: appendEntries

9
Raft Library
● Our Motations (2016)
○ Use Raft in Hadoop Ozone – HA and Write Pipeline
● “In Search of a Usable Raft Library”
○ A long list of Raft implementations is available
○ None of them a general library ready to be consumed by other projects.
○ Most of them are tied to another project or a part of another project.
● We need a Raft library!

11
Apache Ratis — A Brief History
● 2016-03: Started at Hortonworks.
● 2017-01: Entered Apache incubation.
● 2017-05: Released ﬁrst version 0.1.0-alpha.
● 2017-04: Hadoop Ozone branch started using Ratis!
● 2020-07: Released ﬁrst GA version 1.0.0.
● 2021-02: Became a top level Apache project.
● 2021-03: Released the version 2.0.0.
● 2023-11: Released the version 3.0.0.
● 2024-01: Releases the latest version 3.0.1.

12
Who uses Apache Ratis?
● Apache Ozone (Object Store)
○ SCM/OM high availability
○ Write Pipeline
● Alluxio (Data Orchestration for the Cloud)
○ Originally called Tachyon
○ Managing all of Alluxio's journaled state
● Apache IoTDB (Database for Internet of Things)
○ Replicating application state (HA)
● Apache Celeborn (Intermediate Data Service)
○ Replication for high availability

13
General Ratis Use Case
● You already have a service running on a single server
● You want to
○ (1) replicate the server data to multiple machines
■ The replication number/cluster membership can be changed in runtime
■ It can tolerate server failures.
or
○ (2) have a HA (highly available) service
■ When a server fails, another server will automatically take over.
■ Clients automatically failover to the new server.
Apache Ratis is for you!

14
Apache Ratis — A Raft Library
● Open source, open development, community driven
● Apache License 2.0
● Written in Java 8
Contributions are welcome!

16
Apache Ratis — Standard Raft Features
● Leader Election + Log Replication
○ Automatically elect a leader among the servers in a Raft group
○ Randomized timeout for avoiding split votes
○ Log is replicated in the Raft group
● Membership Changes
○ Members in a Raft group can be re-conﬁgured in runtime.
○ Replication factor can be changed in runtime.
● Log Compaction
○ Snapshots are taken periodically
○ Send snapshot instead of a long log history.

17
Apache Ratis — Pluggability
● Pluggable state machine – the application logic
○ An application must deﬁne its state machine
● Pluggable metrics
○ Default is implemented using Dropwizard metrics v4
○ Provided an alternative using Dropwizard metrics v3
○ Applications may provide a custom metrics implementation

18
Apache Ratis — Pluggability
● Pluggable RPC
○ Applications may provide their own RPC implementation
○ Provided implementations: gRPC, Netty, Hadoop RPC
● Pluggable Raft log
○ Users may provide their own log implementation
○ The default implementation stores log in local ﬁles

19
Apache Ratis — High Performance
● Asynchronous Event Driven Architecture
○ Implemented by
■ gRPC bi-directional stream API
■ Netty asynchronous event-driven network
application framework
■ Java CompletableFuture & Executor APIs
○ Server-to-server: asynchronous appendEntries
○ Client-to-server: asynchronous client requests
■ Also support blocking APIs in client

20
Apache Ratis — Client APIs
● Async API
CompletableFuture<RaftClientReply> send(Message message);
CompletableFuture<RaftClientReply> sendReadOnly(Message message);
● Blocking API (simply call CompletableFuture.get())
RaftClientReply send(Message message) throws IOException;
RaftClientReply sendReadOnly(Message message) throws IOException;

21
Apache Ratis — High Performance
● Data Intensive Applications
○ In Raft,
■ All transactions and the data are written in the log
● Log entry: data + metadata
● State machine: data
■ Write amplification
● A single write may become two or more internal writes
○ In Ratis
■ Application could choose to not written all the data to log
● Log entry: only metadata
● State machine: data

22
Apache Ratis — Streaming API
● Allow a client writing to the closest peer, instead of the Leader
● The first peer forwards the data to other peers
● The client may provide a routing table to tell how to forward the data.
● Netty zero buffer copy – never create/copy buffers in the code
○ Not necessarily the OS level zero buffer copy

23
Apache Ratis — Streaming Performance
● The performance can be 3x compared to writing to the Leader
○ Streaming can use the full power of all three machines.

24
Apache Ratis — Read from Followers
● Linearizable Read
○ Clients can read updated data from Followers
○ Read-index algorithm & Leader lease
● Stale Read
RaftClientReply sendStaleRead(Message message, long minIndex, RaftPeerId server) throws IOException;
CompletableFuture<RaftClientReply> sendStaleRead(Message message, long minIndex, RaftPeerId server);
○ minIndex is the minimum log index already committed in the given server
○ The returned data can be outdated (stale)

25
Apache Ratis — Read After Write Consistency
● Suppose x = 1, a client sends the following operations asynchronously:
○ Write x = 2
○ Write x = 3
○ Read x
● What will it get for “Read x”?
○ Intuitively, it should get x = 3.
○ However, read and write are both asynchronous and executed in diﬀerent paths.
■ Although the same client sends all ops, it is like
that read and write are sent by two clients.
○ “Read x” may return 1, 2 or 3.
● In Ratis, it support read-after-write consistency (conﬁgurable)
○ “Read x” must return 3.

26
Apache Ratis — Watch Requests
● Watch an index to satisfy the given replication level.
RaftClientReply watch(long index, ReplicationLevel replication) throws IOException;
CompletableFuture<RaftClientReply> watch(long index, ReplicationLevel replication);
■MAJORITY (default) – committed at the leader and replicated to a majority of peers (Raft)
■ALL – committed at the leader and replicated to the all peers.
■MAJORITY_COMMITTED – committed at a majority of peers.
■ALL_COMMITTED – committed at all peers.
Client API

27
Apache Ratis — Multi-Raft
● A server can join multiple Raft groups

28
Apache Ratis — Security
● Support TLS connections in gRPC and Ratis Streaming
○ Applications may pass a Ratis TlsConf to build a Ratis client/server
○ The client/server uses the TlsConf to build a Netty SslContext
○ The SslContext is used to establish a secure connection to provide
■Authentication
■Encryption

29
Apache Ratis — Leadership
● Transfer Leadership
○ Change the leader from a server to another server
● Leader Election Management
○ Pause and resume leader elections

30
Apache Ratis — Snapshot Management
● Standard Raft Snapshot
○ Auto-triggering snapshot
○ Conﬁgurable snapshot retention policy
● Install Snapshot Notiﬁcation
○ Notify a follower to install a snapshot from an external source
● Manual Snapshot Creation
○ API to trigger creating a snapshot

31
Apache Ratis — Server Priority
● Priority
○ Servers can be assigned to a priority
○ A higher priority server won’t vote for a lower priority server
■ unless the higher priority server is outdated
○ A lower priority server won’t become the Leader
■ unless all higher priority servers voted for it

32
● Listeners – for listening log entries
○ They receive appendEntires.
○ They do not vote.
○ They are not counted for majority.
○ They can be served as hot standbys.
Apache Ratis — Non-voting members

33
Apache Ratis — CLI
● Group: info, list
● Peer: add, remove, setPriority
● Snapshot: create
● Election: transfer, stepDown, pause, resume
● Local: raftMetaConf

Thank you
Contributions are welcome!

35
Apache Ratis - Driven by Community
● 93 contributors (according to github)
● 30 committers
● 19 PMC members

36
● We are currently voting for the 3.1.0 release
○ Thanks William Song from Apache IoTDB
for being the Release Manager
● Ratis Thirdparty Release 1.0.6
○ Vote passed on 2024.5.14
○ Thanks also Attila Doroszlai from Apache Ozone
for being the Release Manager
● Thanks everyone for voting, verifying and
testing the releases!
Apache Ratis - New Releases

Agenda
● A Brief Introduction of Raft
● Apache Ratis Community
● Apache Ratis Features (selected)
● A Short Example

38
Example: FileStore
● Maintain a file map (key -> file)
○ Supported operations: Read, Write, Delete
○ Unsupported operations: List, Rename, etc.
● Asynchronous & In-order
○ Client may submit multiple write requests to
■ Write to multiple files at the same time
■ Each file may have multiple write requests
● File data is managed by the state machine
○ but not store in the raft log
● Performance (depending on hardware)
○ Write throughput: 1.8 GB/s (large files, old results)
○ IOPS: 16,000 txns/s (small files, recent results by Siyao Meng)

Apache Ratis - A High Performance Raft Library

More Related Content

What's hot (20)

Similar to Apache Ratis - A High Performance Raft Library (20)

Recently uploaded (20)

Apache Ratis - A High Performance Raft Library