Проксирование HTTP-запросов web-акселератором / Александр Крижановский (Tempesta Technologies)
1 like464 views
The document discusses HTTP requests proxying and related topics, including load balancing, connection management, and security issues. It covers the functionalities and considerations involved in high-performance network traffic processing, specifically in relation to Tempesta Technologies' open-source application delivery controller (ADC). Additionally, it outlines various techniques for optimizing HTTP message handling, such as zero-copy socket proxying and TCP/IP stack management.
Проксирование HTTP-запросов web-акселератором / Александр Крижановский (Tempesta Technologies)
- 1. HTTP Requests Proxying Alexander Krizhanovsky Tempesta Technologies, Inc. ak@tempesta-tech.com
- 2. Who am I? CEO & CTO at Tempesta Technologies Developing Tempesta FW – open source Linux Application Delivery Controller (ADC) Custom software development in: ● high performance network traffic processing e.g. WAF mentioned in Gartner magic quadrant ● Databases e.g. MariaDB SQL System Versioning https://github.com/tempesta-tech/mariadb https://m17.mariadb.com/session/technical-preview-temporal-queryi ng-asof
- 3. HTTP requests proxying Load balancing Backend connections management SSL/TLS termination Protocol downgrade
- 4. HTTP proxying issues How many server connections? What to do if an upstream resets a connection? How to manage HTTP message queues? Is it safe to resend a request to another upstream? TLS is expensive Sockets proxying is expensive HTTP multiplexing - HTTP/2?
- 5. Server connections New connections or persistent connections HTTP keep-alive connections Keep-Alive: timeout=5, max=10000 Reestablish closed KA connection New connections if all are busy Tempesta FW: connections provisioning for DDoS and flash crowds
- 6. How many connections? There is optimal connections number for particular Web server and hardware Large connections number: Larger latency Smaller RPS
- 7. Connection resets & server failure Default for Apache HTTPD & Nginx: reset each 100 requests Dirty hack for messy dynamics Server failure ● Connection reset or I/O timeout ● HTTP error responses ● Backend healthcheck • TCP connection • HTTP response code • HTTP response body
- 8. HTTP message pipelining Mostly unused by proxies Squid, Tempesta FW, Polipo Messages multiplexing Forwarding and reforwarding issues Security issues ● Breaking authentication ● HTTP Response splitting
- 9. Connection closing and Keep-Alive Client Connection: close + TCP active close (RFC 7230 3.3.3) no Content-Length no Transfer-Encoding: chunked Backend ● Connection: keep-alive ● Add Content-Length ● Now the message can be pipelined
- 10. HTTP Response Splitting attack (Web cache poisoning) /redir_lang.jsp?lang=foobar%0d%0aContent-Length:%200%0d%0a%0d%0a HTTP/1.1%20200%20OK%0d%0aContent-Type:%20text/html%0d%0a Content-Length:%2019%0d%0a%0d%0a<html>Shazam</html> HTTP/1.1 302 Moved Temporarily Date: Wed, 24 Dec 2003 15:26:41 GMT Location: http://10.1.1.1/by_lang.jsp?lang=foobar Content-Length: 0 HTTP/1.1 200 OK Content-Type: text/html Content-Length: 19 <html>Shazam</html> Proxy must count number of requests and responses (later detection!) Validate allowed URI alphabet (also for injection attacks)
- 11. (Non-)indempotent requests RFC 7231 4.2.2: Nonidempotent request changes server state Idempotent (safe) methods: GET, HEAD, TRACE, OPTIONS ● Admin responsibility: GET /action?do=delete ● Can be POST (e.g. web-search) Let a user to specify idempotence of methods and resources: nonidempotent GET prefix “/action”
- 12. Pipeline queues Tempesta FW: switch off pipelining server_queue_size 1 Use a queue with NI-request only if all connections are busy
- 13. Resend HTTP requests? Tainted requests: server_forward_retries and server_forward_timeout Request-killers – RFC 7230 6.3.2: “a client MUST NOT pipeline immediately after connection establishment” ● Connection with re-forwarded requests is non-schedulable Nonidempotent requests aren’t reforwarded server_retry_nonidempotent Error messages keep order of responses Resending HTTP messages + TCP retransmission Copying is required… ...unless TCP send queue integreated into a proxy
- 14. Requests reforwarding (not for persistent sessions!)
- 15. SSL/TLS: no zero-copying (no sendfile()!) User-kernel space copying ● Copy network data to user space ● Encrypt/decrypt it ● Copy the data to kernel for transmission Kernel-mode TLS ● Facebook, RedHat: https://lwn.net/Articles/666509/ ● Netflix: https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf ● DDoS: TLS handshake is still an issue ● Tempesta: http://www.netdevconf.org/2.1/papers/https_tcpip_stack.pdf
- 16. Buffering vs streaming Buffering ● Seems everyone by default ● Performance degradation on large messages ● 200 means OK, no incomplete response Streaming ● Tengine (patched Nginx) w/ proxy_request_buffering & fastcgi_request_buffering ● More performance, but 200 doesn't mean full response
- 17. HTTP messages adjustment Add/remove/update HTTP headers Build a new HTTP message
- 18. User space HTTP proxying 1. Receive request at CPU1 2. Copy request to user space 3. Update headers 4. Copy request to kernel space 5. Send the request from CPU2 3 data copies Access TCP control blocks and data buffers from different CPUs
- 19. HTTP messages adjustment: zero-copy Add/remove/update HTTP headers w/o copies skb and its head are allocated in the same page fragment or a compound page
- 20. Zero-copy sockets proxying Kernel space or user space TCP/IP stack Socket callbacks call TLS and HTTP processing Everything is processing in softirq (while the data is hot) No receive & accept queues No file descriptors Less locking
- 21. Tempesta FW: kernel HTTPS/TCP/IP stack Alternative to user space TCP/IP stacks HTTPS is built into TCP/IP stack L7 DDoS mitigation and simple WAF out of the box Very fast HTTP parser and strings processing using AVX2 TODO HTTP QoS for asymmetric DDoS mitigation DSL for multi-layer filter rules
- 22. Tempesta FW: zero-copy proxying Socket callbacks call TLS and HTTP processing Everything is processing in softirq (while the data is hot) No receive & accept queues No file descriptors Less locking Lock-free inter-CPU transport => faster socket reading => lower latency
- 23. HTTP/2 Pros ● Responses are sent in parallel and in any order (no head-of-line blocking) ● Compression Cons ● Zero copy techniques aren’t applicable => For client connections (slow network), not for LAN (fast network)
- 24. Thanks! Web-site: http://tempesta-tech.com (Powered by Tempesta FW) Availability: https://github.com/tempesta-tech/tempesta Blog: http://natsys-lab.blogspot.com E-mail: ak@tempesta-tech.com