Rlite software-architecture (1)
- 1. A RINA light implementation Vincenzo Maffione 20/02/2017
- 2. Introduction (1) β A Free and Open Source light implementation of RINA for Linux β Implementation splitted between user-space and kernel-space β KISS approach β codebase is clean and essential β Focus: β basic functionality - do few things but to them well β stability and performance - support deployments with hundreds of nodes β minimality - avoid over-engineering β Main goal: a baseline implementation for future RINA products β Code and documentation available at https://github.com/vmaffione/rlite
- 3. Introduction (2) β ~ 27 Klocs (not including blanks) β kernel-space: ~ 9 Klocs β user-space: ~ 18 Klocs β including tools and example applications β Written mostly in C (some parts are C++ for convenience) β C: 14 Klocs β C++: 7 Klocs β Network applications can be written in C β Python bindings available to write network applications in Python
- 4. Introduction (3) β kernel-space is implemented as a set of out-of-tree kernel modules, which run on the unmodified Linux kernel. β Linux Kbuild system is used to build the modules against the running kernel β Build time (no parallel make): 3-15 seconds β user-space is implemented as a set of shared libraries and programs β CMake is used to configure and build libraries and executables β Build time (no parallel make): 15-60 seconds
- 5. Basic features (1) β Applications: β Flow allocation and deallocation, with QoS specification β Application registration and unregistration β Data transfer β Stack administration: β Creation, deletion and configuration of IPCPs β Registration and enrollment among IPCPs β Monitoring and inspection β inspection of IPCPs in the system β inspection of RIBs β per-flow statistics
- 6. Basic features (2) β QoS (supported through DTCP): β Flow control β Retransmission control β Maximum allowable gap β Simple token-bucket rate-limiting β Decent performance (detailed performance plots to come) β About 9.5 Gbps on a 10 Gbit link without flow control and retransmission β About 6 Gbps on a 10 Gbit link with flow control β A lot of room for optimizations β Stability indicators β Done 10 days long VM-based experiments with up 35 nodes, two levels of normal DIFs and 50 flows allocations per second β Done experiments with up to 10 levels of DIFs
- 7. Architecture overview (1) rlite-ctl uipcps daemon application librlite-cdaplibrlite-conf librlite /dev/rlite /dev/rlite-io rlite shim-eth shim-loopback shim-tcp4 normal user-space kernel-space shim-hv shim-udp4
- 8. Architecture overview (2) β kernel-space β Supports control operations β Implements datapath β Keeps state β user-space β Libraries to abstract interaction with kernel-space functionalities β A daemon to implement management part of (many) IPCPs β An iproute2-like command-line tool to administer the stack β Interactions between kernel-space and user-space only happen through character devices β therefore through file descriptors
- 9. Kernel-space architecture (1) β Supported functionalities: β IPCP creation, deletion and configuration (kernel keeps a per-IPCP data structure) β Flow (de)allocation (kernel keeps a per-flow data structure) β Application (un)registration (kernel keeps a data structure for each registered application) β RMT, DTP and DTCP components of the normal IPCP β Shim IPCP processes (e.g. interaction with network device drivers) β State is maintained in kernel-space: β user-space can crash or be restarted at any time β user-space can recover state from kernel
- 10. Kernel-space architecture (2) β User-space interacts with kernel-space only through two character devices β /dev/rlite for control operations β /dev/rlite-io for data transfer and synchronization β Consequently, interactions only happen through file descriptors β Both are βcloning devicesβ β each open() creates a new independent kernel-space instance β Both devices support blocking and non-blocking operation β Standard poll() and select() widely used with the devices
- 11. Kernel-space architecture (3) β /dev/rlite used for control operations β flow (de)allocation β Application (un)registration β IPCP creation, deletion and configuration β Management of PDU forwarding table β interactions between user-space and kernel-space parts of IPCPs β inspection and monitoring operations on flows and IPCPs β ...
- 12. Kernel-space architecture (4) β Control operations follow a request/response paradigm: β write() to the control device to submit a request message β Response messages (not always present) can be read through read() β The control device is used to avoid ioctls() and netlink β Easier porting to other OSes (e.g. FreeBSD) β Request and response messages are represented by packed structs and are serialized/deserialized during the user-space ββ kernel-space transition β support for string (de)serialization β support for (apn, api, aen, aei) name (de)serialization
- 13. Kernel-space architecture (5) β /dev/rlite-io for data transfer and synchronization β read() β write() β select(), poll(), epoll() β Application workflow: β Use the control device to allocate a flow (kernel-space object) β Bind the flow to a newly-created data transfer file descriptor - this is the only task performed by means of ioctl() β Use the data transfer file descriptor to exchange SDUs and/or wait for events β Close file descriptor to deallocate the associated flow β Special binding mode to exchange management SDUs
- 14. Kernel-space architecture (6) β Usual abstract factory pattern to manage different types of IPCPs β normal: implementation of the regular IPCP β shim-loopback: supports system-local IPC, with optional queued mode to decouple TX and RX code-paths, and optional packet drop emulation β shim-eth: uses network device drivers to transmit and receive SDUs, sharing the device with the Linux network stack β shim-udp4: tunnels RINA traffic over UDP socket; mostly implemented in user-space, only data transfer is implemented in kernel-space β shim-tcp4: same as shim-udp4, but using a TCP socket; deprecated, since it duplicates flow-control and congestion control done in higher layers β shim-hv: uses VMPI devices to transmit and receive SDUs
- 15. Some kernel-space internals β Reference counters widely used to manage lifetime of objects (e.g. IPCPs, flows, registered applications, PDUs) β sk_buff-like approach to avoid copies throughout the datapath β dynamic allocation of PDU buffers β The amount of header space to reserve at allocation time is precomputed by the user-space daemon, depending on the local IPCP dependency graph β All PDU queues are limited in size to keep memory usage under control β Deferred work (workqueues) used only when necessary, to keep latency low β Example: driver transmission routine directly executes in the context of an application write() system call, when possible
- 16. Architecture overview rlite-ctl uipcps daemon application librlite-cdaplibrlite-conf librlite /dev/rlite /dev/rlite-io rlite shim-eth shim-loopback shim-tcp4 normal user-space kernel-space shim-hv shim-udp4
- 17. user-space libraries β librlite (written in C) β main library, abstracts interactions with the rlite control device (/dev/rlite) β provides common utilities and helpers (application names, flow specification, control messages, ...) β provides an API for RINA applications β Other libraries β librlite-conf (C): extends librlite with kernel-space IPCP management functionalities β librlite-cdap (C++): CDAP implementation based on Google Protocol Buffer
- 18. librlite - Overview β librlite provides API calls to interact with control device instances β Validation, serialization and deserialization of control messages in both directions (user β kernel, kernel β user) β It defines a POSIX-like APIs for applications: β Reminiscent of the socket API, to ease porting of existing socket applications... β β¦ yet with the full power of RINA API (QoS support and complete naming scheme) β Easy to learn for grown-up network developers! β Documentation available at https://github.com/vmaffione/rlite/blob/master/include/rina/api.h β Other resources: https://github.com/IRATI/stack/wiki/Application-API
- 19. librlite - Application API β Main API calls: β int rina_open() β fd β Opens a control device instance, returning a file descriptor. β int rina_flow_alloc(dif_name, local_name, remote_name, flowspec, flags) β fd β Issues a flow allocation request and possibly wait for the associated response. Returns a file descriptor to be used for data transfer. β int rina_register(fd, dif_name, appl_name, flags) β Register an application into a given DIF. β int rina_register(fd, dif_name, appl_name, flags) β Unregister an application from a given DIF. β int rina_flow_accept(fd, flags) β remote_appl, flowspec β Wait and possibly accept an incoming flow request, where the destination application is one of the ones registered to the control device referred by fd. Returns a file descriptor to be used for data transfer.
- 20. librlite-conf β It is the backend for the rlite-ctl stack administration tool β Exports the management and inspection functionalities: β IPCP creation β IPCP deletion β IPCP configuration β Fetch of current flows (with related statistics) β Dump state of a specific flow β Synchronization with uipcps daemon, to wait for the user-space part of an IPCP to show up β ...
- 21. librlite-cdap β CDAP implementation using Google Protocol Buffer as concrete syntax β Provides CDAP message constructors, serializers and deserializers β Provides CDAP connections object to send and receive CDAP messages β Each CDAP connection wraps a file descriptor β In this way CDAP can be used over arbitrary file descriptors β Primarily meant to be used with /dev/rlite-io file descriptors β No dependencies on other parts of rlite, can be reused as a stand-alone component
- 22. Architecture overview rlite-ctl uipcps daemon application librlite-cdaplibrlite-conf librlite /dev/rlite /dev/rlite-io rlite shim-eth shim-loopback shim-tcp4 normal user-space kernel-space shim-hv shim-udp4
- 23. Uipcps daemon - Overview β A multi-threaded single-process daemon that implements management part of some IPCPs β When an IPCP is created by the kernel, the daemon gets notified, and creates the corresponding user-space IPCP (uipcp) β For regular IPCPs, it implements: β Flow allocation RIB objects β Directory Forwarding Table RIB objects β Enrollment RIB objects and enrollment state machines β Routing RIB objects β Address allocation RIB objects β For shim-tudp4 IPCPs it implements UDP sockets setup and dynamic UDP port allocation β For shim-tcp4 IPCPs it implements TCP connection setup and teardown for both client and server side (connect(), accept(), etc.)
- 24. Uipcps daemon - Internals β A custom event-loop thread for each IPCP β An additional thread that implements a UNIX socket server to serve requests coming from the rlite-ctl tool (or other future agents) β Abstract factory pattern to manage different types of uipcps β Reference counters used to manage uipcps lifetime β Subsystems: β UNIX socket server, written in C β uipcps container for generic uipcp management (creation, deletion, β¦), written in C β shim-udp4 and shim-tcp4 user-space implementation, written in C β normal IPCP user-space implementation, written in C++ manly because of CDAP β C++ code confined inside the uipcp-normal statically linked library.
- 25. Uipcps daemon - Subsystems rlite-ctl uipcp daemon librlite-cdap librlite application unix server uipcps container normal shim udp4
- 26. Uipcp daemon - Event loop β A custom event-loop on top of rlite control devices β The event-loop thread to select() over many file descriptors β rlite control devices: when events happen on the control device, event-specific callbacks get executed β Other file descriptors: when an event is ready on one of those, an user-provided callback gets executed β Supports timers, that can be used to execute a callback after a certain amount of time
- 27. Uipcp daemon - Advanced features β The uipcp-containers module keeps track of the IPCPs in the local system and the flows allocated among them β This information is maintained in a graph of local IPCPs β A node for each IPCP, an edge for each inter-IPCP flow β Graph used for automatic computation of: β per-IPCP Maximum SDU size (using the constraints provided by shim DIFs) β per-IPCP PCI header space to be reserved at kernel buffer allocation β Result of computation is pushed to the kernel for optimized operation β Optional automatic re-enrollment triggers to create N-1 flows where they are missing
- 28. rlite-ctl β An ip-route2-like command-line tool to administer and monitor IPCP processes β Functionalities: β IPCP creation and deletion β IPCP configuration β Registration of an IPCP to a DIF β Enrollment between a local IPCP and a remote IPCP β Show list of IPCPs β Show RIB of a DIF β Show list of flows β Dump state of a specific flow
- 29. Common functionalities β Common code is compiled both in user-space and kernel-space, to ease maintenance: β Serialization and deserialization routines of control messages across user/kernel interface β Table-based serialization/deserialization, adding a new message is straightforward β Helper functions for RINA names - (APN, API, AEN, AEI) tuples.
- 30. Available RINA application β Example applications: β rinaperf: multi-threaded client/server capable of parallel flow allocation, implementing basic connectivity and performance testing: ping, request-response, unidirectional bandwidth β rina-echo-async: single-threaded event-loop based client/server tool, capable of concurrent flow allocation and concurrent flow management β Real application β nginx: RINA port of the popular Nginx server β dropbear: RINA port of the Dropbear ssh client/server β rina-gw: Event-loop application acting as an application gateway between a RINA network and an IP network β It forwards TCP connections over RINA flows and the other way around
- 31. Demo β RINA/TCP gateway, to make TCP/IP world interact with RINA world β Minimally patched Nginx Web Server runs over RINA TCP/IP NETWORK Proxy host Client host 1 Web browser rina-gw Server host 1 patched nginx RINA NETWORK Client host 2 Web browser RINA flow TCP connection
- 32. Demo β RINA/TCP gateway, to make TCP/IP world interact with RINA world β Minimally patched Nginx Web Server runs over RINA VM A patched nginx VM B rina-gw Browser n.1.DIF (normal) Shim-eth (e.1.DIF) TCP