Multipath TCP Upstreaming
- 1. Multipath TCP Upstreaming Mat Martineau (Intel) and Matthieu Baerts (Tessares)
- 2. Plan ● Multipath TCP Overview ● First Patch Set Upstreaming Roadmap ● Advanced Features Roadmap ● Conclusion and links 2
- 4. Multipath TCP (MPTCP) ● Exchange data for a single connection over different paths, simultaneously ● RFC-6824 and supported by IETF Multipath TCP (MPTCP) working group 4Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain
- 5. Multipath TCP (MPTCP) ● Exchange data for a single connection over different paths, simultaneously ● RFC-6824 and supported by IETF Multipath TCP (MPTCP) working group ● More bandwidth: 5Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain
- 6. Multipath TCP (MPTCP) ● Exchange data for a single connection over different paths, simultaneously ● RFC-6824 and supported by IETF Multipath TCP (MPTCP) working group ● More mobility (walk-out): 6Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain
- 7. Multipath TCP (MPTCP) ● Exchange data for a single connection over different paths, simultaneously ● RFC-6824 and supported by IETF Multipath TCP (MPTCP) working group ● More mobility (walk-out): 7Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain
- 8. Multipath TCP (MPTCP) ● Exchange data for a single connection over different paths, simultaneously ● RFC-6824 and supported by IETF Multipath TCP (MPTCP) working group ● More mobility (walk-out): 8Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain
- 9. Multipath TCP (MPTCP) ● Exchange data for a single connection over different paths, simultaneously ● RFC-6824 and supported by IETF Multipath TCP (MPTCP) working group ● More mobility (walk-out): 9Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain
- 10. Multipath TCP Use Cases ● Smartphones (Apple, Samsung, LG, others) ○ Support failover / “walk-out” scenario. ○ More Bandwidth ● Residential Gateways (LTE + DSL, for example) ○ More Bandwidth ● Multipath TCP is part of 5G standardisation: ○ Access Traffic Steering, Switching and Splitting: ATSSS 10
- 11. Multipath TCP Use Cases: ATSSS 11 Steering best network selection OR Switching seamless handover FROM TO and vice versa Splitting network aggregation AND improved end-user experience Defined in 3GPP Release 16, ATSSS is a core network function in 5G networks, playing a key role in managing data traffic between 3GPP (5G, 4G) networks and non-3GPP (Wi-Fi) networks 5G 5G 5G WiFi WiFi WiFi
- 12. Existing Linux implementation ● First implementation for Linux kernel in March 2009 ○ Latest MPTCP out-of-tree Linux kernel version is v0.95 ○ Generally used as a client / server in current deployments, for millions of users ● But not upstreamable ○ Built to support experiments and rapid changes but not generic enough ○ Special purpose implementation of MPTCP 12
- 13. Guidelines for upstream ● New implementation cannot affect existing TCP stack: ○ Without performance regressions. No code size change if CONFIG_MPTCP=n ○ Maintainable and configurable ○ Can be used in a variety of deployments ● Multipath TCP will be "opt-in" ● Proceed in steps: ○ Minimal features set ○ Optimisations and advanced features for later 13
- 14. Protocol Overview: RFC 6824 ● Looks like TCP on the wire, similar usage for apps 14 MPTCP TCP Subflow TCP Subflow Socket Layer IP Layer
- 15. Protocol Overview: RFC 6824 ● Looks like TCP on the wire, similar usage for apps ● Subflow mapping: Data Sequence Number 15 MPTCP TCP Subflow TCP Subflow Socket Layer IP Layer AB CD EF Dseq=0, seq=123,“A” Dseq=1, seq=124,“B” Dseq=2, seq=456,“C” Dseq=4, seq=125,“E” Dseq=5, seq=126,“F” Dseq=3, seq=456,“D” Smartphone icon by Blurred203 under CC-by-sa, others from Tango project, public domain
- 16. ● Looks like TCP on the wire, similar usage for apps ● Subflow mapping: Data Sequence Number Protocol Overview: RFC 6824 16 MPTCP TCP Subflow TCP Subflow Socket Layer IP Layer
- 17. Protocol Overview: RFC 6824 17 MPTCP TCP Subflow TCP Subflow Socket Layer IP Layer ● Looks like TCP on the wire, similar usage for apps ● Subflow mapping: Data Sequence Number
- 18. Protocol Overview: RFC 6824 18 MPTCP TCP Subflow TCP Subflow Socket Layer IP Layer ● Looks like TCP on the wire, similar usage for apps ● Subflow mapping: Data Sequence Number
- 19. Protocol Overview: RFC 6824 19 MPTCP TCP Subflow TCP Subflow Socket Layer IP Layer ● Looks like TCP on the wire, similar usage for apps ● Subflow mapping: Data Sequence Number + ACK
- 20. Protocol Overview: RFC 6824 ● Looks like TCP on the wire, similar usage for apps ● Subflow mapping: Data Sequence Number + ACK ● MP_CAPABLE, MP_JOIN, DATA_FIN 20 MPTCP TCP Subflow TCP Subflow Socket Layer IP Layer In TCP options
- 21. Protocol Overview: RFC 6824 ● Looks like TCP on the wire, similar usage for apps ● Subflow mapping: Data Sequence Number + ACK ● MP_CAPABLE, MP_JOIN, DATA_FIN ● Signaling: Add/Remove Addresses, Fast Close 21 MPTCP TCP Subflow TCP Subflow Socket Layer IP Layer Via TCP ACK
- 22. Protocol Overview: RFC 6824 ● Looks like TCP on the wire, similar usage for apps ● Subflow mapping: Data Sequence Number + ACK ● MP_CAPABLE, MP_JOIN, DATA_FIN ● Signaling: Add/Remove Addresses, Fast Close ● Coupled receive windows across TCP subflows 22 MPTCP TCP Subflow TCP Subflow Socket Layer IP Layer
- 23. Multiple versions of MPTCP ● RFC 6824: Experimental ○ All known implementations support it, only this version ● RFC 6824 bis: Standard ○ Submitted to IESG for publication ○ Behavioral changes: MPTCP v0 → MPTCP v1 ○ Some parts easier to implement ○ Selected by 3GPP for 5G 23
- 24. First Patch Set Roadmap 24
- 25. MPTCP Socket architecture 25 MPTCP TCP Subflow TCP Subflow Socket Layer IP Proto: struct proto SKB 1 SKB 2 ... SKB 1 SKB 2 ... TCP ULP: struct tcp_ulp_ops We start from: tcp_request_sock_ops SKB extension: struct mptcp_ext To store Data Sequence Signal (25 bytes)
- 26. Userspace API ● MPTCP selected when creating the socket: socket(AF_INET(6), SOCK_STREAM, IPPROTO_MPTCP); 26
- 27. Userspace API ● MPTCP selected when creating the socket: socket(AF_INET(6), SOCK_STREAM, IPPROTO_MPTCP); ○ IPPROTO_MPTCP = IPPROTO_TCP | 0x100; /* = 262 */ 27
- 28. Userspace API ● MPTCP selected when creating the socket: socket(AF_INET(6), SOCK_STREAM, IPPROTO_MPTCP); ○ IPPROTO_MPTCP = IPPROTO_TCP | 0x100; /* = 262 */ ● getsockopt() / setsockopt() with MPTCP socket or its TCP subflows? 28
- 29. Userspace API ● MPTCP selected when creating the socket: socket(AF_INET(6), SOCK_STREAM, IPPROTO_MPTCP); ○ IPPROTO_MPTCP = IPPROTO_TCP | 0x100; /* = 262 */ ● getsockopt() / setsockopt() with MPTCP socket or its TCP subflows? ● Security: who can create MPTCP sockets? ○ Initial implementation will not be hardened by broad use yet (syzkaller, etc.) ○ sysctl per network namespace, MPTCP disabled by default: is it enough? 29
- 30. Diagnostics ● MPTCP will have a collection of counters for diagnostic and debug purposes ● Per-socket data will be shared with userspace via sock_diag(7) ○ TCP ULP framework has been extended to enable diag ● Some TCP counters are also found in /proc ○ Should MPTCP add to these as well? 30
- 31. Tests ● Kernel Self Tests ○ Between multiple namespaces (veth) ○ MPTCP ⇔ MPTCP, MPTCP ⇔ TCP, TCP ⇔ MPTCP ○ Various conditions including packet loss, reordering, and variations in routing ● Packetdrill ○ Background project ongoing to add MPTCP support ○ Out-of-tree Packetdrill with MPTCP support but old and limited 31
- 32. Initial use case ● Server role is a good place to start ● Simpler path management ○ Client side handles multiple interfaces (like cellular + Wi-Fi) ● Common server configuration uses one public interface for clients ○ Advertising additional interfaces not required ● Client features all build on what’s needed for servers 32 Server IP1 Client IP2 IP3 NAT NAT Internet
- 33. Code already merged upstream ● SKB extensions ○ Needed to carry MPTCP options that are tied to the data payload ○ Also used to remove sp (sec_path) and nf_bridge pointers from struct sk_buff ○ Suitable for data that can’t fit in sk_buff and justifies memory overhead ● Add inet_diag_ulp_info to socket diag format and ULP get_info hook 33
- 34. Change in TCP Code Git Stat: include/linux/skbuff.h | 11 ++ include/linux/tcp.h | 51 ++++++++++ include/net/sock.h | 6 +- include/net/tcp.h | 20 ++++ include/trace/events/sock.h | 5 +- include/uapi/linux/in.h | 2 + net/Kconfig | 1 + net/Makefile | 1 + net/ax25/af_ax25.c | 2 +- net/core/skbuff.c | 7 ++ net/decnet/af_decnet.c | 2 +- net/ipv4/inet_connection_sock.c | 2 + net/ipv4/tcp.c | 8 +- net/ipv4/tcp_input.c | 29 +++++- net/ipv4/tcp_ipv4.c | 4 +- net/ipv4/tcp_minisocks.c | 6 ++ net/ipv4/tcp_output.c | 62 +++++++++++- net/ipv4/tcp_ulp.c | 12 +++ 34
- 35. Changes to TCP code ● tcp_ulp_clone() ● Export two low-level TCP functions and one struct ● SKBs with MPTCP extensions can’t be coalesced or collapsed ● MPTCP option parsing and writing ● is_mptcp flag in tcp_sock and tcp_request_sock 35
- 36. Changes to TCP code, continued ● One MPTCP-specific branch in TCP minisocks ● Call out to MPTCP from tcp_data_queue to add SKB extension and process ACKs ● Additional members in struct tcp_options_received ● Subflow receive window sharing will introduce changes too 36
- 38. Path Manager Which path to create/remove? Which address to announce? 38 ? ? Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain
- 39. Userspace Path Manager ● Peers share ADD_ADDR and REMOVE_ADDR signals to advertise available addresses for each MPTCP connection ● Path manager runs in userspace and uses generic netlink to track address and local interface updates and request subflow changes ● Can be customized with different policies. ● Multipath TCP Daemon alpha release is available at github.com/intel/mptcpd 39
- 40. Packet Scheduler On which available path packets will be sent? Reinject packets in another path? 40 A A ? ? Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain
- 41. Packet scheduling ● Different connections may optimize for throughput, latency, or redundancy. ● Peers can set a ‘backup’ flag on each subflow to limit transmission on that flow ● Include basic scheduler options in the kernel ● Consider eBPF to define custom schedulers, instead of kernel modules 41
- 42. Using MPTCP with unmodified binaries ● Some organizations want to take advantage of MPTCP without recompiling their userspace ● Can add BPF_CGROUP_SOCKET to attach an eBPF program that rewrites the protocol number passed to socket() ● Similar attachment points exist for bind() and connect() 42
- 43. MPTCP Performance optimizations ● Initial emphasis is on correctness and reasonable MPTCP performance ○ While not disrupting TCP’s optimizations! ● Target performance optimizations based on data ● Protocol optimizations ○ Example: changing scheduler behavior for reinjection of data on different subflows ● TCP Fast Open support 43
- 44. Break-before-make ● MPTCP can keep a connection active even with zero subflows connected ○ Allows the session to continue by adding a subflow with MP_JOIN ● Can be useful to switch between access points ● Will add this capability if there’s demand for it 44
- 45. Subflow socket options ● One MPTCP socket manages a set of in-kernel subflow sockets ● Socket options that use TCP option space or change data flow could interfere ● The MPTCP socket can act as an intermediary for subflow options ● Will need to whitelist specific known-safe options ● Could expose file descriptors only good for getsockopt()/setsockopt() 45
- 46. Kernel TLS and MPTCP ● kTLS is built on top of TCP using ULP framework ● An MPTCP socket is not a TCP socket, so it doesn’t have ULP ● TLS needs to operate on the MPTCP data stream, not subflow streams ○ TLS records could be split across subflows ○ MPTCP DSS mappings are specific to TCP sequence numbers ● TLS_SW appears feasible but would need work to integrate with an MPTCP socket type 46
- 47. Conclusion 47
- 48. Conclusion ● Build around TCP as much as we can. ● We are close to having an initial patch set ready. 48 This project is open to everybody. ● Wiki: https://is.gd/mptcp_upstream ● Mailing list: https://lists.01.org/mailman/listinfo/mptcp ● Git repository: https://github.com/multipath-tcp/mptcp_net-next ● Paper: https://linuxplumbersconf.org/event/4/contributions/435/ ● mathew.j.martineau@linux.intel.com ● matthieu.baerts@tessares.net
- 49. Backup slides 49
- 50. Protocol challenges 50Used with Christoph Paasch’s permission Relations between structures
- 52. Protocol challenges Coupled receive windows across TCP subflows 52Used with Sébastien Barré’s permission
- 53. Multipath TCP (MPTCP) Hybrid access network use-case (BBF TR-348 by Tessares - SwissCom - OVH) 53 Telco CloudDSL CPE 4G CPE TCP TCP MPTCP copper (long line) 4G/LTE network fiber xDSL network MPTCP cloud native HAG multi- platform Agent available capacity needed, for 4G/LTE connectivity Images from Tessares
- 55. Protocol challenges Data sequence numbers and mappings 55 AB C Smartphone icon by Blurred203 under CC-by-sa, others from Tango project, public domain
- 56. Protocol challenges Data sequence numbers and mappings 56 AB CD EF Smartphone icon by Blurred203 under CC-by-sa, others from Tango project, public domain
- 57. Protocol challenges Data sequence numbers and mappings 57 AB C D EF Smartphone icon by Blurred203 under CC-by-sa, others from Tango project, public domain
- 58. Protocol challenges Data sequence numbers and mappings 58 AB C D EF Dseq=0, seq=123,“A” Dseq=1, seq=124,“B” Dseq=2, seq=456,“C” Dseq=4, seq=125,“E” Dseq=5, seq=126,“F” Dseq=3, seq=456,“D” Smartphone icon by Blurred203 under CC-by-sa, others from Tango project, public domain
- 59. Protocol challenges Data sequence numbers and mappings mptcp_sock tcp_sock Socket Layer IP Layer App sends data 59
- 60. Protocol challenges Data sequence numbers and mappings mptcp_sock tcp_sock Socket Layer IP Layer 60 Select the TCP subflow
- 61. Protocol challenges Data sequence numbers and mappings mptcp_sock tcp_sock Socket Layer IP Layer TCP header: What DSS to set? 61
- 62. Protocol challenges Sending of ACKs to signal options, e.g. REMOVE_ADDR in a TCP ACK mptcp_sock tcp_sock tcp_sock Socket Layer IP Layer Subflow ops Subflow ops 62 Notification: one iface is down
- 63. Protocol challenges Sending of ACKs to signal options, e.g. REMOVE_ADDR in a TCP ACK mptcp_sock tcp_sock tcp_sock Socket Layer IP Layer Subflow ops Subflow ops 63 Select the subflow
- 64. Protocol challenges Sending of ACKs to signal options, e.g. REMOVE_ADDR in a TCP ACK mptcp_sock tcp_sock tcp_sock Socket Layer IP Layer Subflow ops Subflow ops 64 Sending a ACK not from TCP stack
- 65. Protocol challenges Reception of ACKs with signaling options, e.g. REMOVE_ADDR in a TCP ACK mptcp_sock tcp_sock tcp_sock Socket Layer IP Layer Subflow ops Subflow ops 65 TCP ACK received
- 66. Protocol challenges Reception of ACKs with signaling options, e.g. REMOVE_ADDR in a TCP ACK mptcp_sock tcp_sock tcp_sock Socket Layer IP Layer Subflow ops Subflow ops 66 TCP ACK is not dropped
- 67. Protocol challenges Reception of ACKs with signaling options, e.g. REMOVE_ADDR in a TCP ACK mptcp_sock tcp_sock tcp_sock Socket Layer IP Layer Subflow ops Subflow ops 67
- 68. Protocol challenges Signaling with MPTCP: ● MP_CAPABLE ● MP_JOIN ● DSEQ / DACK ● FAST_CLOSE ● ADD_ADDR ● REMOVE_ADDR 68 SYN SYN ALL ACK followed by RST ACK ACK