Introduction
netio3 is a network library used by the phase-2 FELIX project. It supports network communication via TCP and RDMA. The library is split into two parts: netio3-backend (this project) and netio3. The backend provides functionality for sending and receiving data over the network, while the frontend provides data coalescence (buffered sending) and publish/subscribe semantics. The backend is completely agnostic to any FELIX specific functionality and can be used in any project that requires network communication.
TCP is supported using the asyncmsg library developed by ATLAS TDAQ, while RDMA is supported using the libfabric library. Both backends use a connection and message-based model.
Recommended Setup
In order to use the RDMA technology, the host computer shall be equipped with a capable network card such as nVidia Connect-X (from X3 onwards). The drivers can be downloaded from NVIDIA <https://developer.nvidia.com/doca-downloads>_ (Host-Server -> DOCA-Host -> Linux -> x86_64 -> doca-ofed).
Setting the following environment variables is strongly reccomended. All but RDMAV_FORK_SAFE are automatically set in FELIX software; RDMAV_FORK_SAFE has be set in the environment of SW ROD because CERN ROOT libraries use fork().
Variable |
Value |
Reason |
|---|---|---|
RDMAV_FORK_SAFE [1] |
any |
allows the application to use fork() |
FI_VERBS_TX_IOV_LIMIT |
30 |
Max length of TX IOV vector |
FI_VERBS_RX_IOV_LIMIT |
30 |
Max length of TX IOV vector |
FI_VERBS_TX_SIZE |
1024 |
Max number of buffers per socket |
FI_VERBS_RX_SIZE |
1024 |
Max number of buffers per socket |
Distribution, Use, API
netio3-backend is part of the FELIX software distribution but can be compiled standalone as well. It is not supposed to be used to interface felix-star. Use felix-client-thread instead.
Requirements
ALMA9
LCG_106b
Recommended: * MLNX_OFED (NVIDIA drivers)
Minimum for compilation without NVIDIA drivers: * libibverbs * librdmacm-devel