[9-6-2016] Openlab Poster-v3
0 likes381 views
1. The document discusses the development of an Open Fabrics Interface (OFI) transport for the FairMQ message queue library used by the ALICE experiment at CERN. 2. This transport allows FairMQ to bypass the Linux kernel and leverage RDMA capabilities when using the libFabric library for high-performance message passing. 3. Preliminary tests of the OFI transport prototype showed significant performance gains over existing TCP-based transports, achieving throughput of over 30Gbps for large message sizes.
![Ioannis.Charalampidis@cern.ch
Lazaros.Lazaridis@cern.ch
CERN, June 2016
libFabric
ofi://
Transport
nanomsg
ALFA
usNIC
FairMQ
For the ALICE O2 upgrade, the simulation and
reconstruction software for the ALICE
experiment are using the ALFA1 framework.
Among other abstractions, ALFA/FairROOT
framework provides a Message-Queue
abstraction library, the FairMQ2, that is
lightweight wrapper around ØMQ and NanoMsg
libraries.
Since the project’s goal is to implement a new
transport for FairMQ we decided to extend the
functionality of either of these libraries.
We chose NanoMsg3 because of it’s
clean and modular internals.
Introduc)on
ØMQ
fi_send(
endpoint,
buffer, len,
mr_desc,
context
);
buffer … ..
fi_recv(
endpoint,
buffer, len,
mr_desc,
context
);
buffer … ..
RDMA*
Memory Region Memory Region
fi_send(
endpoint,
buffer, len,
mr_desc,
context
);
fi_recv(
endpoint,
buffer, len,
mr_desc,
context
);
Tx CQ Rx CQ Tx CQ Rx CQ
SEND
ACK
fi_cq_read( &event ); fi_cq_read( &event );
* libfabric has custom event polling func?ons
One of the powerful features of the usNIC
fabric is the fact that it can bypass the linux
kernel from user-space when using the
libFabric4 library.
This relieves the kernel from the IP stack
overhead, reclaiming it’s CPU time for more
useful operations.
usNIC + Kernel Bypass
The ofi:// Transport
The project is implemented as a patch to the
NanoMsg sources5 that introduces the Open
Fabrics Interface (OFI) transport.
The transport translates the POSIX-like API of
NanoMsg into an RDMA-like API for libFabric,
transparently to the user. To achieve this it
uses a dynamic memory registration (MR)
mechanism that tries to reduce the amount of
MR performed, while being agnostic of the
user’s intentions.
1. Technical Design Report for the Upgrade of the Online–
Offline Computing System, The ALICE Collaboration
2. https://github.com/FairRootGroup/FairRoot/tree/master/
fairmq
3. http://nanomsg.org
4. http://ofiwg.github.io/libfabric
5. https://github.com/wavesoft/nanomsg-transport-ofi
6. https://github.com/wavesoft/robob
7. https://github.com/ofiwg/libfabric/issues?q=author
%3Awavesoft
8. https://github.com/nanomsg/nanomsg/pull/612
In a similar manner, it uses the high-level
libFabric polling API, instead of the FD - based
NanoMsg polling API, making it possible to
support any fabric without any modification.
True Zero-Copy
One of the implementation requirements of the
OFI transport was to ensure that no memcpy
operations will take place between the user’s
request and the transfer on the wire.
That’s reasonable if you consider
that the message sizes vary
from 50Mb to 1Gb
A useful by-product of this project was the
development of robob6, a fully automated benchmarking
utility, for ensuring the quality of the measured values
By-Products
Outcomes
We frequently encountered roadblocks while
working this project, since we were using
new products and open source components.
We had frequent interactions with NanoMsg
and CISCO developers7 and we contributed
our own modifications8.
Nonetheless we managed to create a
prototype were we demonstrated the
feasibility of the transport and it’s
performance.
On the right we present some preliminary
measurements using the OFI transport
between two Intel Xeon E5-2690 machines
with UCSC-PCIE-C40Q NICs, connected
through switch with a 40Gbit copper cable. 0
5
10
15
20
25
30
35
8192 16384 32768 65536 1048576 2097152 4194304 8388608 16777216 33554432 67108864 134217728
Throughput (GB/s) for Different Message Sizes
OFI [GBit/s] TCP [Gbit/s] ØMQ [Gbit/s]
www.cern.ch/openlab
Poster by Ioannis Charalampidis. Special thanks
to our supervisor, Predrag Buncic, to Artur
Barczyk for his guidance, to Mohammad Al-
Turany and Peter Hristov](https://image.slidesharecdn.com/ea0d2fe3-08c3-4b2e-9128-d34e52f87d5e-160721103318/85/9-6-2016-Openlab-Poster-v3-1-320.jpg)
[9-6-2016] Openlab Poster-v3
- 1. Ioannis.Charalampidis@cern.ch Lazaros.Lazaridis@cern.ch CERN, June 2016 libFabric ofi:// Transport nanomsg ALFA usNIC FairMQ For the ALICE O2 upgrade, the simulation and reconstruction software for the ALICE experiment are using the ALFA1 framework. Among other abstractions, ALFA/FairROOT framework provides a Message-Queue abstraction library, the FairMQ2, that is lightweight wrapper around ØMQ and NanoMsg libraries. Since the project’s goal is to implement a new transport for FairMQ we decided to extend the functionality of either of these libraries. We chose NanoMsg3 because of it’s clean and modular internals. Introduc)on ØMQ fi_send( endpoint, buffer, len, mr_desc, context ); buffer … .. fi_recv( endpoint, buffer, len, mr_desc, context ); buffer … .. RDMA* Memory Region Memory Region fi_send( endpoint, buffer, len, mr_desc, context ); fi_recv( endpoint, buffer, len, mr_desc, context ); Tx CQ Rx CQ Tx CQ Rx CQ SEND ACK fi_cq_read( &event ); fi_cq_read( &event ); * libfabric has custom event polling func?ons One of the powerful features of the usNIC fabric is the fact that it can bypass the linux kernel from user-space when using the libFabric4 library. This relieves the kernel from the IP stack overhead, reclaiming it’s CPU time for more useful operations. usNIC + Kernel Bypass The ofi:// Transport The project is implemented as a patch to the NanoMsg sources5 that introduces the Open Fabrics Interface (OFI) transport. The transport translates the POSIX-like API of NanoMsg into an RDMA-like API for libFabric, transparently to the user. To achieve this it uses a dynamic memory registration (MR) mechanism that tries to reduce the amount of MR performed, while being agnostic of the user’s intentions. 1. Technical Design Report for the Upgrade of the Online– Offline Computing System, The ALICE Collaboration 2. https://github.com/FairRootGroup/FairRoot/tree/master/ fairmq 3. http://nanomsg.org 4. http://ofiwg.github.io/libfabric 5. https://github.com/wavesoft/nanomsg-transport-ofi 6. https://github.com/wavesoft/robob 7. https://github.com/ofiwg/libfabric/issues?q=author %3Awavesoft 8. https://github.com/nanomsg/nanomsg/pull/612 In a similar manner, it uses the high-level libFabric polling API, instead of the FD - based NanoMsg polling API, making it possible to support any fabric without any modification. True Zero-Copy One of the implementation requirements of the OFI transport was to ensure that no memcpy operations will take place between the user’s request and the transfer on the wire. That’s reasonable if you consider that the message sizes vary from 50Mb to 1Gb A useful by-product of this project was the development of robob6, a fully automated benchmarking utility, for ensuring the quality of the measured values By-Products Outcomes We frequently encountered roadblocks while working this project, since we were using new products and open source components. We had frequent interactions with NanoMsg and CISCO developers7 and we contributed our own modifications8. Nonetheless we managed to create a prototype were we demonstrated the feasibility of the transport and it’s performance. On the right we present some preliminary measurements using the OFI transport between two Intel Xeon E5-2690 machines with UCSC-PCIE-C40Q NICs, connected through switch with a 40Gbit copper cable. 0 5 10 15 20 25 30 35 8192 16384 32768 65536 1048576 2097152 4194304 8388608 16777216 33554432 67108864 134217728 Throughput (GB/s) for Different Message Sizes OFI [GBit/s] TCP [Gbit/s] ØMQ [Gbit/s] www.cern.ch/openlab Poster by Ioannis Charalampidis. Special thanks to our supervisor, Predrag Buncic, to Artur Barczyk for his guidance, to Mohammad Al- Turany and Peter Hristov