Introduction

Felix-star is the suite of applications that interface FELIX hardware with the network. Different applications serve different purposes: felix-tohost publishes data coming from the detector uplinks (ToHost direction) over the network, felix-toflx transfers data in the opposite direction (FromHost), and felix-register allows remote clients to monitor the status and change the configuration of FELIX cards.

All felix-star applications use the netio-next network library, or it newer version dubbed netio3 netio3 to take advantage of the low-latency and high-performance RDMA technology. The network library provides the publish/subscribe pattern, data coalescence and the event loop that drives asynchronous I/O.

Remote clients send and receive data associated to one or more e-links that generally correspond to detector front-end resources. Clients discover what FELIX host and application serves the e-link(s) of interest thanks to felix-bus. In felix-bus, each felix-star application advertises the associated e-links in the form of FelixIDs (fids), each accompanied by the IP, port and network setting. Not all e-links map to the detector: "virtual" e-links are defined to identify additional data streams. For example, felix-register defines e-links do map network messages to FELIX devices.

Felix-star interacts with the FELIX card using the flxcard API and is compatible with all FELIX card models.

Basic Concepts

This section defines some functional components of the FELIX card that are fundamental for felix-star. A complete and technical description can be found in the firmware specification document.

Note

An uplink is an optical link that moves data from the front-end electronics to FELIX. A downlink moves data in the opposite direction.

FELIX device

A FELIX card [1] has a physical 16-lane PCIe interface that is split in two 8-lane logical interfaces or, more precisely, PCIe endpoints. Each PCIe endpoint is in jargon called a device. Therefore if two cards are present in a server, device 0 and 1 are correspond to card 0 and device 1 and 2 correspond to card 1.

[1]

with the exception of FLX-709 that has an 8-lane PCIe physical interface and one corresponding PCIe endpoint.

FELIX registers

FELIX cards are configured via registers. Registers are memory locations defined in the FELIX register map that reside in the PCIe Base Address Register (BAR). Registers are accessible via the flxcard API. Registers can be read-only and report monitoring information, or writable and be used to communicate with the firmware. Notably, registers are used to report the position of read/write pointers used for data transfer.

DMA buffer

A FELIX card can directly read from or write data to specified buffers in the host memory without requiring CPU involvement. This functionality is called Direct Memory Access (DMA) and the buffers allocated in the host memory DMA buffers. The DMA buffers are allocated within a large contiguous memory region allocated at boot time by CMEM. CMEM is a driver and library developed at CERN that is capable of allocating such large buffer. In felix-star's code the acronym VMEM also appears: this is a software emulation of CMEM that does not require the CMEM driver to be installed.

Important

Each FELIX device supports one FromHost DMA buffer (host to downlinks) and multiple ToHost DMA buffers (uplinks to host).

Firmware flavours

For what concern felix-star, there are two families of firmware: lpGBT-like and FULL-like. The first family is allows FELIX to communicate with radiation-hard ASICs such as lpGBT and GBTX by supporting the corresponding data transmission protocols. These protocols define the concept of e-link: one of the many data streams carried by an optical fibre. From the felix-star point of view lpGBT-like firmware provide a number of e-links much larger than the number of links and a large message rate rather than throughput. Firmware flavour that are lpGBT-like are lpGBT, GBT, ITk Pixel, ITk Strips, and HGTDlumi.

FULL-like flavours, such as FULL and Interlaken, use a simpler protocol that does not split the bandwidth of an optical link and are meant to interface FELIX with another FPGA-based hardware platform. In this case each link is an e-link and can support high-throughput (up to the link bandwidth). Some detectors require the support for multiple data streams in FULL-like flavours. Because the protocol has no support for streams [2], the stream identifier (StreamID) is contained in the first byte of the message payload. Up to 256 streams are thus supported. When streams are enabled the StreamID is parsed in software.

In all cases, felix-star applications read the e-link configuration automatically from FELIX registers.

[2]

the Interlaken protocol supports data streams, but FULL does not however a common solution has been preferred.

Network buffers

The use of RDMA requires the allocation and pinning [3] of network buffers. Network buffers are managed by the network library but the numbers and size is configured by felix-star applications. This configuration can have an impact on the memory use: each buffered [4] connection allocated an independent stack of buffers.

[3]

make the buffer non-swappable i.e. the kernel cannot move the data elsewhere.

[4]

see netio-next documentation

Distribution

Felix-star is distributed as part of all FELIX software releases. Information about the releases can be found at download page; the release source code can be found at felix-distribution repository.