Internals of how an Http Client works (Final) (3).pdf

A look into the
internals of an
HTTP Client
How Event Loops, Decorators, and Reactive Streams
are used to create an HTTP Client.

Modern day backend systems
- Network communication is an important part of modern day
systems.
- External services
- Databases
- Clients
- Especially, with the popularity of microservices
architecture, inter-component communication is becoming
more important.

Case Study 1
- A well configured HTTP client can go a long way.
- Imagine a request per thread server uses an HTTP client with
a 10 second response timeout, without a circuit breaker.
- The server may not be able to serve important requests.
A thread-per-request
server
Push
Service
Payment
Service
User Requests
- All threads are busy waiting for a
push response
HTTP Client

Case Study 2
final MyResponse response;
try {
response = client.get("/");
} catch(IOException e) {
// log the exception and pray this doesn’t happen more often
}
It would be nice if we actually could understand the
stacktrace and know what the issue is.

Before diving in
- I’d like to share how a performant, extensible client is implemented
by analyzing Armeria’s WebClient.
- I imagine other http clients are implemented in a similar way.
- I hope this will be useful even for those who won’t be using
Armeria.
- Based on the most recent Armeria tree at the moment of writing
- Version 1.25.2
- It is not realistic to describe all code paths, I’ll be assuming only the
most general code path with a few details left out.

So what is Armeria
- An asynchronous microservice framework
- Supports HTTP/1, HTTP/2, PROXY, TLS
- Built on top of Netty
- Compliant with Reactive Streams
- https://github.com/line/armeria

A simple example
ClientFactory cf = ClientFactory.builder().build();
HttpRequest req = HttpRequest.of(HttpMethod.GET, "/hello");
HttpResponse res = WebClient.builder("http://foo.com")
.factory(
cf)
.build()
.
execute(req);
AggregatedHttpResponse aggRes = res.aggregate().join();
System.out.println(aggRes.contentUtf8());
I will focus on what happens when HttpClient#execute is called

A simple example (illustrated)
DefaultWebClient
HttpClientDelegate
Decorators
Name Server
DNS Query
Target Server
Acquire a connection
HttpRequest HttpResponse
subscribe()
read(), write()
ClientFactory
DefaultWebClient
Executed from an Event Loop starting from this point

High level view of what we’ll be looking at
1. Figure out where exactly we want to send
our request
2. Open and prepare a TCP connection
3. Send our request and receive the response
GET /hello HTTP/1.1
Host: foo.com
HTTP/1.1 200 OK
Request
Response

DNS
Eventually we need to know where to send the request
- foo.com -> 10.0.0.0
Some questions over the past years
- There are too many DNS queries which are ﬂooding the DNS server. Is there a way to reduce
the number of DNS queries.
- The target server IP will be replaced. How can we replace the servers gracefully so that
requests aren’t failed.
- If a DNS resolution fails, will the request fail? (should it fail?)
The starting point is calling resolveAddress , which eventually calls AddressResolver#resolve to
resolve a host name to an InetSocketAddress.

DNS: The Basic Resolver
Name Server
Armeria
foo.com
123.123.123.123
TTL: 1 day
- Basically, we ask the Name Server if there is a record “foo.com”
- If there is, an IP address is returned.

Name Server
Armeria
foo.com
123.123.123.123
1. DelegatingDnsResolver
- Delegates DNS resolution to
the Netty layer resolver
- Resolver.conf
- Google
- …

What about DNS query ﬂooding?
- Too many DNS queries may degrade a DNS server’s performance.
- If N clients request to M domains, that means N * M DNS resolutions may
occur per each TTL
We may need a cache layer which respects the DNS record TTL

Query Name Server
123.123.123.123
1. CachingDnsResolver
- Caches entries for each DNS
query
123.123.123.123
Armeria
foo.com
Checks cached entries

- What about search domains?
Wikipedia: A search domain is a domain
used as part of a domain search list.The search
list, as well as the local domain name, is used by
a resolver to create a fully qualified domain
name (FQDN) from a relative name
If search domains are “a.com”, “b.com”, the
searched domains will be:
- mydomain.a.com.
- mydomain.b.com.
- mydomain.

Query Name Server
123.123.123.123
1. SearchDomainDnsResolver
- Queries for each search
domain if exists
query
- foo.com.a.com.
- foo.com.b.com.
- foo.com.
123.123.123.123
Armeria
Expands query to each search
domain
foo.com

Sometimes, users would like to
take more control over dns
resolution results
- /etc/hosts

Query Name Server
123.123.123.123
1. HostsFileDnsResolver
- Checks whether the hosts ﬁle
can be used for DNS resolution
- Queries for each search domain if
exists
query
- Delegates DNS resolution to the
Netty layer resolver
Check if hosts file has the entry
- foo.com.a.com.
- foo.com.b.com.
- foo.com.
123.123.123.123
Armeria
domain
foo.com

Query Name Server
123.123.123.123
1. DefaultDnsResolver
- Puts everything together
- Checks whether the hosts ﬁle can
be used for DNS resolution
- Queries for each search domain if
exists
- Caches entries for each DNS query
- Delegates DNS resolution to the
Netty layer resolver
- foo.com.a.com.
- foo.com.b.com.
- foo.com.
123.123.123.123
Armeria
domain
foo.com

- Q: If a request is made on “foo.com” but a DNS resolution fails, should we fail
the request overall?
- A: It depends! Obviously, the user should be able to conﬁgure this.
- But what is the most reasonable default behavior?
The current default implementation:
1. If there are no previously cached entries, the request will fail.
2. If there is a previously cached entry, try to use the previous value. Meanwhile,
try to refresh the cached entry in the background.
Note: We actually recommend using the EndpointGroup API for this scenario, but this is a separate
topic.

Query Name Server
123.123.123.123
1. RefreshingAddressResolver
- Refreshes addresses in the background
and serves cached dns entries.
2. DefaultDnsResolver
- Puts everything together
- Checks whether the hosts ﬁle can be
used for DNS resolution
- Queries for each search domain if exists
- Caches entries for each DNS query
- Delegates DNS resolution to the Netty
layer resolver
- foo.com.a.com.
- foo.com.b.com.
- foo.com.
123.123.123.123
Armeria
domain
foo.com
Refreshes cached entries in
the background

DNS
FAQ
- There are too many DNS queries which are flooding the DNS server. Is there a way to
reduce the number of DNS queries.
- We can check if the number of search domains (ndots) is causing too many DNS
queries
- We can check if the dns cache is configured correctly
- The target server IP will be replaced. How can we replace the servers gracefully so that
requests aren’t failed.
- We can temporarily clamp the DNS TTL to a smaller value while migration takes
place
- We could also check if the RefreshingAddressResolver cache configurations.
- If a DNS resolution fails, will the request fail? (should it fail?)
- By default, it won’t fail. The previous DNS query result will be used by default even
if the TTL passed.

Connection Pooling
How many connections should be opened in the above scenario?
- HTTP1.1: If keep alive is enabled, a connection handle requests sequentially without closing
the connection. (assuming pipelining is disabled)
- HTTP2: A connection handle handle MAX_CONCURRENT_STREAMS number of requests
concurrently
Opening a connection consumes resources/latency, so Armeria reuses connections as much as
possible.
WebClient client = WebClient.of();
client.get("http://1.2.3.4");

Connection Pooling
Endpoint
- A connection to (10.0.0.0) cannot be shared with a connection to (10.0.0.1)
Protocol
- A connection using HTTP2 cannot be shared with a connection using HTTP1
Event Loop
- Armeria uses Netty under the hood for network IO
- Netty assumes a single connection is handled by a single event loop
- This means a connection for EventLoopA cannot be shared with a connection for
EventLoopB
- We want to share connections as much as possible
- A connection is sharable per (Endpoint, Protocol, Event Loop) for Armeria.

Event Loops
Event Loop (Wikipedia)
the event loop is a programming construct or design
pattern that waits for and dispatches events or
messages in a program

Event Loops
class MyEventLoop {
final Queue<Runnable> taskQueue = new ArrayDeque<>();
final Thread thread = new Thread() {
@Override
public void run() {
while (true) {
Runnable runnable = taskQueue.poll();
if (runnable != null) {
runnable.run();
}
}
}
};
MyEventLoop() {
thread.start();
}
public void schedule(Runnable runnable) {
taskQueue.add(runnable);
}
}
MyEventLoop myEventLoop = …
myEventLoop.schedule(() -> System.out.println(“Hello”))
A simple event loop
implemented in Java

Event Loops
Without event loops, a simple solution is locking
int inc = 0;
ReentrantLock lock = new ReentrantLock();
private void increment() {
lock.lock();
try {
inc++;
} finally {
lock.unlock();
}
}
void main() {
for (int i = 0; i < 100; i++) {
ForkJoinPool.commonPool().submit(() -> increment());
}
await().until(() -> inc == 100);
assert inc == 100;
}

Event Loops
Provides a much simpler way to do concurrent programming.
int inc = 0;
EventLoop eventLoop = ...;
private void increment() {
if (!eventLoop.inEventLoop()) {
eventLoop.execute(() -> increment());
return;
}
inc++;
}
void main() {
for (int i = 0; i < 100; i++) {
ForkJoinPool.commonPool().submit(() -> increment());
}
await().until(() -> inc == 100);
assert inc == 100;
}
Note: although the queue
add operation may also
lock, it will do so for a
short time only

Event Loops
How do we handle two concurrent requests to a single TCP connection?
1. We can acquire a lock before writing to a connection
2. A single channel (connection) is handled by a single EventLoop
- We don’t have to worry about race conditions when writing to the wire
POST /a HTTP/1.1
Host: foo.com
GET /b HTTP/1.1
Host: foo.com
Content-Length: 5
Hello
POST /a HTTP/1.1
Host: foo.com
Content-Length: 5
Hello
GET /b HTTP/1.1
Host: foo.com

Connection Pooling
- Each ClientFactory maintains a connection pool
- Each request is pre-assigned an Event Loop
- Before a connection is acquired, the call is executed from an event
loop
- In order to acquire a connection
- We check if an existing connection can be used
- We try to acquire a new (or pending) connection
EventLoop
EventLoop
EventLoop
HttpChannelPool
HttpChannelPool
HttpChannelPool
HTTP2 10.0.0.1
HTTP2 10.0.0.2
HTTP1 10.0.0.3
Connections
Connections
Connections
ClientFactory
—
HttpClientDelegate
WebClient WebClient WebClient

Connection Pooling
How many connections should be opened in the above scenario? (given that the request is made concurrently)
- 0 new connections if there are reusable connections available
- Otherwise:
- 3 connections if HTTP1
- 1 connection if HTTP2
What if there are too many connections being created?
- Check which protocol we are using
- If HTTP2, try tuning the MAX_CONCURRENT_STREAMS value
- Check how many endpoints are behind foo.com
WebClient client = WebClient.of();

Session Protocol Negotiation
This step prepares each new connection so that we can start sending HTTP requests.
The trigger is when a new channel connection is attempted, which eventually invokes
HttpClientPipelineConﬁgurator#connect.
- It mostly involves adding the appropriate Netty handlers to the pipeline.
Each protocol has different steps for preparing a connection.
- h1c://foo.com
- h2c://foo.com
- http://foo.com
(Note that I won’t be discussing https, but it is a simple extension of cleartext
negotiation)

h1c://foo.com
- Add HTTP message serializers/deserializers to the Netty Pipeline
- There is no need for protocol negotiation for HTTP1
- Open a connection, send a request, and receive a response.
- Session protocol negotiation is done immediately and the session
future is completed.

h2c://foo.com
- Add HTTP2 message serializers/deserializers to the Netty Pipeline
- Send a connection preface and a settings frame
- Receive a connection preface and a settings frame
- The expected frames have been received and the session future is
completed - we’re now free to send HTTP2 frames

http://foo.com
What if server-side doesn’t support HTTP2?
- We fall back to HTTP1
Steps
1. Send HTTP2 preface and settings frame assuming prior knowledge that the server
supports HTTP2
2. The server sends an error response because it actually doesn’t support HTTP2
3. Close the connection, and store a cache entry that the server doesn’t support
HTTP2
4. Open a new connection and start using HTTP1

- In the end, a sessionPromise is completed and we have a new
connection ready for use
- All we now have to do is send our HttpRequest and receive a
HttpResponse

Sending requests and receiving responses
HttpResponse res = WebClient.builder("http://foo.com" )
.factory( cf)
.build()
. execute(req);
System.out.println(res);
^ This prints immediately even though we didn’t actually send the request
out!
Going back to the original code

Sending requests and receiving responses
HttpRequestWriter requestWriter =
HttpRequest.streaming(HttpMethod.POST, "/");
HttpResponse response = WebClient.builder("foo.com")
.build()
.execute(
requestWriter);
requestWriter.write(HttpData.ofAscii("Hello "));
Thread.sleep(1000)
requestWriter.write(HttpData.ofAscii("World"));
requestWriter.close();
Actually, we can even stream data to the HttpRequest.
- In the following sample code, we acquire a HttpResponse object
even though we didn’t write the request fully yet.

Reactive Streams
From the website: Reactive Streams is an initiative to provide a standard for
asynchronous stream processing with non-blocking back pressure
Publisher Subscriber
subscribe()
request(1)
onNext(message)
1. A subscriber subscribes to a publisher
2. The subscriber can process 1 request
3. The publisher sends 1 message when available
request(1)
onNext(message)
4. The subscriber can process 1 request
5. The publisher sends 1 message when available

Reactive Streams
Subscriber<Integer> subscriber = new
Subscriber<Integer>() {
private Subscription s;
@Override
public void onSubscribe(Subscription s) {
this.s = s;
s.request(1);
}
@Override
public void onNext(Integer integer) {
submitAsyncTask(integer).thenRun(() ->
s.request(1));
}
...
};
Flux.just(1,2,3).subscribe(subscriber);
onSubscribe(s)
request(1)
onNext(1)
request(1)
onNext(2)
request(1)
onNext(3)

Reactive Streams
10G file
on disk
1G RAM
Server
Network
Request some data
Send a small chunk of data
Send a small chunk of data
Ready to receive more data
Request some more data
We can apply reactive
streams principles to
serve a large ﬁle by a
server.
- Obviously, we can’t
just load the entire
ﬁle and serve the data
all at once.

Reactive Streams
HttpRequest HttpRequestSubscriber
HttpResponseWrapper
DecodedHttpResponse
StreamMessageCollector
HTTP/1.1 200 OK
wo
Netty APIs
Hello
POST /hello HTTP1.1
Host: foo.com
rld
HttpObject
Encoder
HttpRespo
nseDecode
r
POST /hello HTTP/1.1
Host: foo.com
Hello
HTTP/1.1 200 OK
world

Reactive Streams
HttpRequest HttpRequestSubscriber
HttpResponseWrapper
DecodedHttpResponse
StreamMessageCollector
subscribe()
Netty APIs
subscribe()
onNext() {
writeToNetty()
}
tryWrite()
Netty’s
encoder
invokes write
HttpObject
Encoder
HttpRespon
seDecoder

Reactive Streams
.factory(
cf)
.build()
.
execute(req);
- HttpResponse and HttpRequest implement Reactive Streams Publisher
- The HttpResponse#aggregate() call internally subscribes to the HttpResponse
and collects the full HttpResponse contents

High level view of what we’ll be looking at
1. Figure out where exactly we want to send
our request
2. Open and prepare a TCP connection
3. Send our request and receive the response
4. A small detour into decorators
GET /hello HTTP/1.1
Host: foo.com
HTTP/1.1 200 OK
Request
Response

Decorators
- What if we want to add more functionality to our client?
- What is an HttpClient
- Basically sends an HttpRequest, and returns an
HttpResponse
- HttpClientDelegate also implements HttpClient
public interface HttpClient {
HttpResponse execute(ClientRequestContext ctx, HttpRequest req)
throws Exception;
}

Decorators
Say that we want to log every request before sending it out
class MyLoggingClient implements HttpClient {
final HttpClient delegate;
MyLoggingClient (HttpClient delegate) {
this.delegate = delegate;
}
@Override
public HttpResponse execute(ClientRequestContext ctx, HttpRequest req) throws
Exception {
logger.info("req: {}", req);
return delegate.execute(ctx, req);
}
}
HttpClient actualClient = ...;
HttpClient myLoggingClient = new MyLoggingClient( actualClient );
myLoggingClient .execute(ctx, req);
Although slightly different, the implementation is similar to Armeria’s built in LoggingClient

Decorators
- Makes use of the decorator design pattern.
- Provides a point of customization before the request is sent
- There are many decorators provided by default (to name just a few)
- RetryingClient
- MetricCollectingClient
- LoggingClient
- CircuitBreakerClient
- AuthClient

Decorators
.factory(cf)
.decorator(LoggingClient.newDecorator())
.decorator(RetryingClient.newDecorator(RetryRule.failsafe()))
.build()
.execute(req);
We can add built-in decorators as well as custom decorators to our clients
for additional functionality

So what is Armeria
- An asynchronous microservices framework
- Supports HTTP/1, HTTP/2, PROXY, TLS
- Built on top of Netty
- Compliant with Reactive Streams
- https://github.com/line/armeria

Internals of how an Http Client works (Final) (3).pdf

More Related Content

What's hot (20)

Similar to Internals of how an Http Client works (Final) (3).pdf (20)

Recently uploaded (20)

Internals of how an Http Client works (Final) (3).pdf