Usenix lisa 2011

Editor's Notes

• #3: Worked on Traffic Server both at Yahoo, and at Apache.Before we go on, lets do a show of hands. How many have or are using a Proxy server of some sort?How many of you are or were using Squid ?There’s still hope for you.
• #4: Traffic Server started as a commercial product, developed and sold by Inktomi way back in the days Yahoo! Acquired Inktomi in 2003, and the software was shelved. Yahoo! needed a reverse proxy to build their own CDN, and squid was not a viable option. So, we took ATS off the shelves again, and made it run on our RHEL distros. In 2009, Y! donated the Traffic Server source code to the Apache Software Foundation. In April of 2010, Apache Traffic Server became a TLP.
• #5: Traffic Server is obviously not the only HTTP intermediary in the Open Source community. Existing servers include Apache mod_proxy, Squid, NGINX, Varnish and Haproxy. This makes the task of choosing a Proxy server an interesting, but challenging task. You really need to understand your problem space, your requirements, and any restrictions (like, budget). Easy for me to pick, but lets discuss some of the considerations you should take.
• #6: I hate this slide, because by the time I do the presentation, this table is already wrong, and you can bet that someone in the audience will point it out! So, please look at this for the next 10 seconds, and then come to me after the presentation and complain.
• #7: The point here is that performance of modern proxy servers is already ridiculous. Other things are more important, such as latency, or features.
• #8: The point here is that performance of modern proxy servers is already ridiculous. Other things are more important, such as latency, or features.
• #9: * Before we go into details of what drives Traffic Server, and how we use it, let me briefly discuss the three most common proxy server configurations.* In a forward proxy, the web browser has to be manually (or via auto-PAC files etc.) configured to use a proxy server for all (or some) requests. The browser typically sends the “full” URL as part of the GET request.The forward proxy typically is not required to be configured for “allowed” destination addresses, but can be configured with Access Control List, or blacklists controlling what requests are allowed, and by whom. A forward proxy is typically allowed to cache content, and a common use case scenario is inside corporate firewalls.
• #10: A reverse proxy, aka a web accelerator, does not require the browser to cooperate in any special way. As far as the user (browser) is concerned, it looks like it’s talking to any other HTTP web server on the internet. The reverse proxy server on the other hand must be explicitly configured for what traffic it should handle, and how such requests are properly routed to the backend servers (aka. Origin Servers). Just as with a forward proxy, many reverse proxies are configured to cache content locally. It can also help load balancing and redundancy on the Origin Servers, and help solve difficult problems like Ajax routing.
• #11: An intercepting proxy, also commonly called a transparent proxy, is very similar to a forward proxy, except the client (browser) does not require any special configuration. As far as the user is concerned, the proxying happens completely transparently. A transparent proxy will intercerpt the HTTP requests, modify them accordingly, and typically “forge” the source IP before forwarding the request to the final destination. Transparent proxies usually also implements traffic filters and monitoring, allowing for strict control of what HTTP traffic passes through the mandatory proxy layer. Typical use cases include ISPs and very strictly controlled corporate firewalls. I’m very excited to announce that as of a few days ago, code for transparent proxy is available in the subversion tree.
• #12: * The obvious use case of a proxy is as a cache.
• #13: Sorry, not cash, content cache. The idea is to offload traffic from the origin servers, your web servers, to the proxy cache.This has obvious advantages, such as easier to scale, easier to manage, and higher availabilityAnd finally, what most people associate a web cache for, you can move content closer to the user, on the “edge”.
• #14: Before we go into the next use case scenario, lets discuss a couple of common problems that can severely affect your web sites performance.There are several reasons why a Web page might be slow to render, three common problems that I personally have to deal with are TCP 3-way handshake, TCP congestion control, and DNS lookups. We’ll discuss the first two here, and explain what we’ve done at Yahoo! using Traffic Server to alleviate these two problems.
• #15: Before we go into the next use case scenario, lets discuss a couple of common problems that can severely affect your web sites performance.There are several reasons why a Web page might be slow to render, three common problems that I personally have to deal with are TCP 3-way handshake, TCP congestion control, and DNS lookups. We’ll discuss the first two here, and explain what we’ve done at Yahoo! using Traffic Server to alleviate these two problems.
• #16: In TCP, every new connection has to go through a setup phase, typically referred to as the 3-way handshake. As you can see from the picture above, this means that there’s a full round-trip worth of latency before the client can even send the first HTTP request to the server. Since latency introduced by 3-way handshake is associated with network round-trip time, it goes without saying that the longer the distance between the client and the server, the longer the latency until the HTTP request can be sent. The solution to this problem is generally to use HTTP keep-alive between the client and server, which is a major reason why it is so critical for Traffic Server, and other intermediaries, to be able to handle tens of thousands of concurrent connections.
• #17: Before we go into the next use case scenario, lets discuss a couple of common problems that can severely affect your web sites performance.There are several reasons why a Web page might be slow to render, three common problems that I personally have to deal with are TCP 3-way handshake, TCP congestion control, and DNS lookups. We’ll discuss the first two here, and explain what we’ve done at Yahoo! using Traffic Server to alleviate these two problems.
• #18: TCP congestion avoidance is a mechanism for TCP to avoid congestion on our networks. There are many different implementations of TCP congestion control mechanisms, but the general idea is to start out ‘slow’, and increase the amount of packets we have outstanding on the wires before we wait for an acknowledgement from the receiver. This is why sometimes congestion control is referred to as “slow start”. Similar to 3-way handshake, this will introduce latencies which are directly related to the round-trip time between client and server. Keep-alive generally doesn‘t solve this, since by default (sometimes configurable) an idle connection will force the restart of another slow start.
• #19: Before we go into the next use case scenario, lets discuss a couple of common problems that can severely affect your web sites performance.There are several reasons why a Web page might be slow to render, three common problems that I personally have to deal with are TCP 3-way handshake, TCP congestion control, and DNS lookups. We’ll discuss the first two here, and explain what we’ve done at Yahoo! using Traffic Server to alleviate these two problems.
• #20: Looks familiar?
• #21: Looks familiar?
• #22: Alright, so lets talk about what problems a good HTTP (and/or proxy server) can solve. There are two primary concurrency problems for the server software developers to consider:How can the software handle tens of thousands of concurrent TCP connections?How can the software take advantage of modern multi-core CPUs? Commodity hardware today has 2, or 4 and even 8 or more cores in each server.* Additionally, while solving these two problems, we have to make sure we don’t introduce other resource starvations, for example, memory pressure.
• #23: Multithreading allows a process to split itself, and run multiple tasks in “parallel”. There is significantly less overhead running threads compared to individual processes, but threads are still not free. They need memory resources, and incur context switches. It’s a known methodology for solving the concurrency problem, and many, many server implementations relies heavily on threads. Modern OS’es have good support for threads, and standard libraries are widely available.
• #24: Deadlocks, where two threads (or processes) need to acquire the same two resources (e.g. locks), which can cause the application to completely stall (unrecoverable) Race conditions can occur, where the outcome is not deterministic, but depends on timing or scheduling of threads execution. Difficult to code and ‘get right’.
• #25: Events are scheduled by the event loop, and event handlers execute specific code for specific events This makes it easier to code for, there’s no risk of deadlock or race condition Can handle a good number of connections (but not unlimited) Squid is a good example of an event driven server.
• #26: Can not block on I/O or the entire event processor stall CPU intensive processing in an event handler can starve pending events for CPU (solution: Cooperative multi-tasking, where the event handler yields the CPU) Latency can become an issue if an event processor has to handle a lot of events Can only use one CPU / core
• #27: There are n worker threads per core, typically 2. This gives around 16 – 24 threads of execution threads on typical modern hardware, each running an event loop There are m I/O threads per disk spindle. This is used to deal with disk I/O outside of the worker threads, and the default is 4. A critical configuration decision here is to scale this appropriately, particularly if a “disk” is raided, and might have more than 1 spindle under the hood. There are also a small number of “helper” threads, to do tasks like accepting new connections, produce log output and stats aggregation and presentation* All threads share resources, such as RAM and Disk cache, configurations, stats and logs.
• #28: When people first look at the ATS configurations, this is usually their reaction.But, it’s actually the complete oppositeGood defaults, in most cases you don’t have to change anythingMost configurations can be reloaded without restartMost features are readily configurable in readable formats.
• #29: Traffic Server comes with a fairly large number of configuration files. Most of the configurations have reasonable “defaults”. Most applications wouldn’t use even a fraction of the available configuration options and features.
• #30: Getting started, only two or maybe three configuration files are necessary to tweak. records.config is a key-value configuration format, holding most global application configuration settings storage.config is used to specify disk storage configurations remap.config is used to specify mapping rules for rewriting requests, typically in a reverse proxy setup
• #41: * Now, the big question, was this a huge waste of your time at the end of a long day?I really hope that it was not, and here’s why
• #42: * Apache Traffic Server is a great, versatile tool readily available, and it’s free!
• #43: * Apache Traffic Server is fast, ridiculously fast. If it’s not fast enough for your needs, please let us know, and please hire some developers, because you probably can afford it.
• #44: * And of course, Traffic Server has a very vibrant community, as a top-level Apache project.
• #47: * And finally, I’d like to thank Go Daddy to not only let me play with fun technology all day, they also pay my bills
• #50: First time I met some of the original developers of Traffic Server, John Plevyak and George Paul both mentioned the “four horsemen”Foundational belief held by some of the original developers of Traffic Server
• #51: * RAW disks, striped and monitored internally* Very low overhead on CPU and disk I/O, since there is no LRU to manage Very low memory overhead (10 bytes per object stored on cache, compare to 60-70 bytes for Squid) Can support very large disks (0.5PB per “partition”)