Dumb Ways To Die: How Not To Write TCP-based Network Applications
The document provides advice on how not to write TCP-based network applications. It discusses several issues: 1) Message delivery over TCP is unreliable and there are no guarantees about when or if a message will arrive. Timeouts need to be implemented at various points. 2) Resources like memory and processing time must be limited for all operations to prevent denial of service attacks. 3) Errors like ECONNRESET can occur for various reasons and applications need to be prepared to handle them gracefully rather than crashing. Overall it emphasizes the need to design applications with the understanding that TCP is a lossy protocol, implement timeouts and resource limits, and handle errors robustly rather than assuming perfect delivery of messages and connections
![HOW[NOT]TO Write TCP-based
Network Applications
Artyom Gavrichenkov
1/x](https://image.slidesharecdn.com/tcpcaveatsengdwtd-121121074635-phpapp02/85/Dumb-Ways-To-Die-How-Not-To-Write-TCP-based-Network-Applications-1-320.jpg)
Dumb Ways To Die: How Not To Write TCP-based Network Applications
- 1. HOW[NOT]TO Write TCP-based Network Applications Artyom Gavrichenkov 1/x
- 2. Based on a True Story • NOT AN AD! • Qrator: distributed network ● Custom TCP/IP at the bottom ● Custom management protocol at the top ● Interacting with plenty of Web servers and Web browsers on a daily basis ● 2 years of continuous debug^W Product ImprovementTM 2
- 3. Issue #1 • Message delivery is unreliable in TCP.
- 4. Issue #1 • Message delivery is unreliable in TCP: there's no estimation on when (and if) the message will arrive at all • Timeouts! • Limit all resources, including time • No action is itself an action
- 5. Timeouts • Between recvfrom() • Between requests • Request timeout • Lifetime of a session • Lifetime of %OBJECTNAME% • Long polling may be a bad idea
- 6. Ex. 1 • Slowloris (Apache): DoS ● (not distributed, just denial of service) • Slow HTTP POST ● Apache, IIS, Lighttpd: DoS ● Nginx: DDoS with a botnet
- 7. Ex. 2 12 rpm AJAX page update ● Backup script switched the server off
- 9. Content-Length – Limit resources for all actions – Custom protocol should define limits on the input length
- 10. errno(3) – The connection may be closed for no good reason – Check errno after recvfrom(), sendto(), etc. ● ENOMEM ● ECONNRESET ● EANYTHING
- 11. Ex. 3 ● Internet Explorer: ECONNRESET means successful connection termination – Download status is being ignored – Content-Length is being ignored
- 12. Memory limits – Resource limits: ● Maximum – ENOMEM ● Minimum – idle wait → ECONNRESET
- 13. Ex. 4 – DNS TTL ● Too big: days of downtime (continuous) ● Too small: days of downtime (total)
- 14. Latency – 3-Way Handshake takes time – Do implement persistent connections! ● Do it from the very beginning
- 15. They haven't listened to me! ● TCP – T/TCP ● HTTP/1.0 – HTTP/1.1
- 16. Optimization – Measure! – Profile! – Emulate packet loss!
- 17. Optimization – Text-based protocols are convenient to debug ● And you will debug – Maybe even in production – Making use of binary protocols is often a premature optimization ● BSON, Google Protocol Buffers
- 18. Optimization ● TCP socket options: – TCP_NODELAY: disables Nagle's algorithm ● Speedup with small portions of data – TCP_CORK (Linux): multiple portions of data in a single TCP segment – "socket corking"
- 19. Optimization ● TCP stack options: – Linux: /proc/sys/net/** ● net.ipv4.tcp_fin_timeout ● net.ipv4.tcp_{,r,w}mem ● net.core.{r,w}mem_max – Windows: HKLMSystemCurrentControlSetServicesTcpipParameters
- 20. IPv6 ● Accidental IPv6 deployment
- 21. • SO_REUSEADDR • sendfile(2) • select(2)/poll(2)/epoll(7) • {n,h}to{n,h}{s,l}() • int64_t vs long 21
- 22. This is it! Artyom Gavrichenkov <ximaera@highloadlab.com>