Introduction
============

netio3 is a network library used by the phase-2 FELIX project. It supports network communication via
TCP and RDMA. The library is split into two parts: netio3-backend (this project) and netio3.  The
backend provides functionality for sending and receiving data over the network, while the frontend
provides data coalescence (buffered sending) and publish/subscribe semantics. The backend is
completely agnostic to any FELIX specific functionality and can be used in any project that requires
network communication.

TCP is supported using the `asyncmsg <https://gitlab.cern.ch/atlas-tdaq-software/asyncmsg>`_ library
developed by ATLAS TDAQ, while RDMA is supported using the `libfabric
<https://ofiwg.github.io/libfabric/>`_ library. Both backends use a connection and message-based
model.


Recommended Setup
-----------------

In order to use the RDMA technology, the host computer shall be equipped with a capable network card
such as nVidia Connect-X (from X3 onwards). The drivers can be downloaded from `NVIDIA <https://developer.nvidia.com/doca-downloads>_`
(Host-Server -> DOCA-Host -> Linux -> x86_64 -> doca-ofed).

Setting the following environment variables is strongly reccomended. All but RDMAV_FORK_SAFE are automatically
set in FELIX software; RDMAV_FORK_SAFE has be set in the environment of SW ROD because CERN ROOT libraries use fork().

.. list-table:: Recommended environment variables.
   :widths: 15 5 80
   :header-rows: 1

   * - Variable
     - Value
     - Reason
   * - RDMAV_FORK_SAFE [#]_
     - any
     - allows the application to use fork()
   * - FI_VERBS_TX_IOV_LIMIT
     - 30
     - Max length of TX IOV vector
   * - FI_VERBS_RX_IOV_LIMIT
     - 30
     - Max length of TX IOV vector
   * - FI_VERBS_TX_SIZE
     - 1024
     - Max number of buffers per socket
   * - FI_VERBS_RX_SIZE
     - 1024
     - Max number of buffers per socket

.. [#] see `https://www.rdmamojo.com/2012/05/24/ibv_fork_init <https://www.rdmamojo.com/2012/05/24/ibv_fork_init/>`_.


Distribution, Use, API
----------------------

netio3-backend is part of the `FELIX software distribution <https://gitlab.cern.ch/atlas-tdaq-felix/felix-distribution>`_
but can be compiled standalone as well. It is not supposed to be used to interface felix-star. Use
`felix-client-thread <https://atlas-project-felix.web.cern.ch/atlas-project-felix/user/felix-interface/master/>`_ instead.


Requirements
------------

* ALMA9
* LCG_106b

Recommended:
* MLNX_OFED (NVIDIA drivers)

Minimum for compilation without NVIDIA drivers:
* libibverbs
* librdmacm-devel