felix-tohost
Felix-tohost is the application that reads ToHost DMA buffers and publishes the data over the network. Felix-tohost can read multiple devices and for each device multiple ToHost DMA buffers.
Top-level Class
The top-level class is ToHost, used to build two executables: felix-tohost that works with a FELIX card and felix-file2host that emulates the hardware. ToHost is templated to support both use cases. The template parameters are three: the configuration parameters (emulating hardware requires passing in more parameters), the type of ToHost DMA buffer (that interfaces with the FELIX driver or not) and the type of FELIX device (real or emulated). ToHost owns the devices and the DMA buffers.
-
template<class CFG, class DEV, class BUF>
class ToHost Core of felix-tohost and felix-file2host.
Public Functions
-
inline explicit ToHost(std::unique_ptr<CFG> config)
ToHost constructor: scalable readout of multiple devices and DMA buffers.
This constructor initialises the object; it does not start reading operations.
- Parameters:
config -- configuration parameters.
-
void start()
Open devices, allocate buffers, create publishers and start reading data.
-
void print_monitoring()
retrieve and write monitoring information. To be executed by the thread that creates an instance of this class.
-
void stop()
Stop reading all associated DMA buffers.
-
inline explicit ToHost(std::unique_ptr<CFG> config)
ToHost Buffer
The ToHost DMA buffer is described by the ToHostBuffer class, specialised to the case in which firmware writes and software reads (as opposed to the FromHost case) in a circular buffer. ToHostBuffer is an abstract class from which two implementations derive: one to interface with a FELIX card and another one in which the behaviour of the card is emulated. In the latter case, a dedicated thread is spawned to write in the DMA buffer (reading the input data from disk and eventually generating data on-the-fly), advance the write pointer and, if configure, emulate hardware interrupts. ToHostBuffer is responsible for advancing the read pointer in a thread-safe manner. In fact ToHostBuffer supports multiple buffer readers, each running its own thread.
Note
ToHostBuffer supports both buffered and zero-copy publishers. Buffered publishers copy the data from the DMA buffer into network buffers. Because data has been copied the read pointer can be advanced immediately. Zero-copy publishers pass to the network the address of data in the DMA buffer. In this case, to make sure that data does not get overwritten, the read pointer can be advanced only after the network reported the completion of the send operation (on_send_complete callback). The read pointer is thus advanced using the of bytes sent by each reader. More precisely, the read pointer offset is set to the minimum number of bytes sent across readers modulo the buffer size. A separate counter keeps track of the bytes read to determine if there are data awaiting a reader in the DMA buffer.
-
class ToHostBuffer
Abstract class representing a ToHost DMA buffer. The elinks enabled for the corresponding DMA buffers are read from the device in the constructors.
Subclassed by FileToHostBuffer, FlxToHostBuffer
Public Functions
-
inline void irq_data_enable()
enable on_data interrupt for this DMA buffer.
-
inline void irq_data_disable()
disable on_data interrupt for this DMA buffer.
-
inline void irq_busy_enable()
enable busy interrupt for this DMA buffer.
-
inline void irq_busy_disable()
disable busy interrupt for this DMA buffer.
-
inline std::shared_ptr<Device> get_device()
- Returns:
a shared pointer to the device serving this DMA buffer.
-
inline int get_dmaid() const
- Returns:
identifying number of this DMA buffer.
-
inline std::vector<Elink> get_elinks()
- Returns:
read e-links enabled for this DMA buffer.
-
inline std::vector<Elink> get_elinks_of_type(elink_type_t t)
- Parameters:
t -- the e-link type, determined from the encoding.
- Returns:
read e-links enabled for this DMA buffer of a given type.
-
std::vector<std::vector<Elink>> split_elinks_of_type(elink_type_t t, size_t n)
- Parameters:
t -- the e-link type, determined from the encoding.
- Returns:
fraction of e-links enabled for this DMA buffer of a given type.
-
virtual void allocate_buffer(size_t size, const std::string &name, bool vmem, bool free_previous_cmem = true) = 0
fraction of e-links enabled for this DMA buffer of a given type.
- Parameters:
name -- of the buffers passed to the CMEM driver.
vmem -- if true, do not use CMEM but malloc a buffer in memory.
free_previous_cmem -- if true, re-claim a buffer with the same name if present and not locked.
-
virtual void dma_start_continuous() = 0
inform the FELIX card of the allocated buffer, to be used as a circular DMA buffer.
-
virtual void dma_wait_for_data_irq() = 0
wait for an on-data MSI-X interrupt.
-
virtual size_t dma_bytes_available() = 0
- Returns:
number of bytes written by firmware, to be read by software.
-
virtual size_t dma_bytes_available_nowrap() = 0
- Returns:
number of bytes written by firmware up to the end of the buffer, to be read by software.
-
virtual bool dma_is_full() = 0
- Returns:
true if the DMA buffer is full.
-
virtual uint64_t dma_get_write_ptr() = 0
- Returns:
address in virtual memory of write pointer (moved by firmware).
-
virtual uint64_t dma_get_read_ptr() = 0
- Returns:
address in virtual memory of read pointer (moved by software).
-
virtual void dma_set_read_ptr_vaddr(uint64_t v_addr) = 0
set a new read pointer.
software uses the virtual addresses, firmware uses the physical ones.
- Parameters:
v_addr -- virtual address of new read pointer.
-
virtual void dma_set_read_ptr_paddr(uint64_t p_addr) = 0
set a new read pointer.
software uses the virtual addresses, firmware uses the physical ones.
- Parameters:
p_addr -- physical address of new read pointer.
-
virtual uint64_t dma_get_read_offset() = 0
- Returns:
read pointer offset with respect to the start of buffer.
-
virtual uint64_t dma_get_write_offset() = 0
- Returns:
write pointer offset with respect to the start of buffer.
-
virtual void dma_advance_read_ptr(size_t bytes) = 0
advance read pointer.
- Parameters:
bytes -- number of bytes processed by software.
-
inline uint64_t dma_get_vaddr()
- Returns:
virtual address of start of buffer.
-
inline size_t dma_get_size()
- Returns:
buffer size.
-
uint32_t reader_register(bool zc_flag)
register a new reader to this DMA buffer.
- Parameters:
zc_flag -- true if the reader uses zero-copy.
- Returns:
reader identifier.
-
bool reader_is_data_available(uint32_t reader_id)
- Parameters:
reader_id -- reader identifier.
- Returns:
whether there are data in the DMA buffer for this reader to read.
-
size_t reader_get_available_bytes(uint32_t reader_id)
- Parameters:
reader_id -- reader identifier.
- Returns:
bytes available for reading for the given reader.
-
std::span<Block> reader_get_available_blocks(uint32_t reader_id)
- Parameters:
reader_id -- reader identifier.
- Returns:
blocks to be read by the given reader.
-
void reader_advance_read_ptr(uint32_t reader_id, uint32_t read_bytes, uint32_t sent_bytes)
advance the DMA buffer read pointer according to the readers' progress
single synchronisation point of the different ToHostReaders.
- Parameters:
reader_id -- reader identifier.
read_bytes -- bytes read by the reader.
reader_id -- bytes sent by the reader (sent != read only in zero-copy mode).
-
inline bool has_zero_copy_reader() const
- Returns:
whether a zero-copy reader is present among those registered to this buffer.
-
inline void set_zero_copy_reader()
set flag for the presence of a zero-copy reader.
-
inline void stop()
stop reading operations.
-
inline bool is_stopped() const
- Returns:
whether read operations have been stopped.
-
uint32_t dma_get_free_MB()
- Returns:
free space in the buffer in MB.
-
inline void dma_increase_irq_counter()
increase the on-data interrupt counter.
-
inline uint64_t dma_get_irq_counter() const
- Returns:
the number of on-data interrupts.
-
inline void irq_data_enable()
Buffer Reader
ToHostReader reads and decodes data for a given subset of e-links. Readers own a publisher and share a pointer to the buffer. The read function is registered as callback in the event loop run by the publisher (see Common Components), either as a periodic callback (for polling data) or as an asynchronous callback (to react in response to an hardware interrupt). ToHostReader owns one decoder per e-link. Each decoder decodes the chunks [1] contained in FELIX blocks corresponding to that e-link.
-
template<class BUF>
class ToHostReader A ToHost DMA buffer can be read by multiple ToHostReaders. ToHostReader is configured to read data from a set of elinks, data from other elinks is ignored. ToHostReader includes a number of Decoders equal to the number of elinks and one publisher. The publisher runs an eventloop thread, therefore each ToHostReaders runs on a dedicate thread.
Public Functions
-
ToHostReader(std::shared_ptr<BUF> buffer, const std::vector<Elink> &elinks, std::unique_ptr<Publisher> publisher, int buffer_poll_period_us, bool zerocopy, int l0id_ck)
ToHostReader constructor.
- Parameters:
buffer -- DMA buffer to be read.
elinks -- vector of elinks whose blocks are to be decoded by this reader.
publisher -- network publisher.
buffer_poll_period_us -- polling period in microseconds.
zerocopy -- if zero-copy readout (instead of buffered) is used.
l0id_ck -- flag to enable the L0ID sequentiality check.
-
ToHostReader(std::shared_ptr<BUF> buffer, elink_type_t elinks_type, std::unique_ptr<Publisher> publisher, int buffer_poll_period_us, bool zerocopy, int l0id_ck)
ToHostReader constructor.
- Parameters:
buffer -- DMA buffer to be read.
elinks_type -- type of elinks whose blocks are to be decoded by this reader.
publisher -- network publisher.
buffer_poll_period_us -- polling period in microseconds.
zerocopy -- if zero-copy readout (instead of buffered) is used.
l0id_ck -- flag to enable the L0ID sequentiality check.
-
bool read()
read the DMA buffer: decode blocks and publish chunks.
- Returns:
true if not all blocks available at the beginning of the call were decoded.
-
inline size_t get_read_blocks() const
- Returns:
the number of blocks read by this reader.
-
inline void fire_publisher_async_signal()
fire the signal to invoke the configured callback.
-
std::vector<ToHostElinkStats> get_reader_elink_stats()
- Returns:
per-elink monitoring information.
-
uint32_t get_network_resource_counter()
- Returns:
publisher network resource counter (e.g. number of available buffers or completion objects).
-
uint32_t get_network_subscriptions()
- Returns:
Number of subscriptions to the publisher.
-
inline elink_type_t get_elink_type()
- Returns:
Number of subscriptions to the publisher.
-
uint64_t get_network_resource_available_calls()
- Returns:
number of callbacks invoked when network resources (e.g. buffers) become available after having been exhausted.
-
inline uint32_t get_id() const
- Returns:
id of the reader
-
ToHostReader(std::shared_ptr<BUF> buffer, const std::vector<Elink> &elinks, std::unique_ptr<Publisher> publisher, int buffer_poll_period_us, bool zerocopy, int l0id_ck)
Decoder
Decoder implements the algorithm to decode the chunks contained in blocks. A chunk can be made up of a single subchunk of type WHOLE, or split in sub-chunks across blocks: one FIRST, possible MIDDLE and one LAST. Each subchunk comes with a flag written by firmware that reports possible error conditions: truncation, malformation or CRC error. An example of malformation is the occurrence of of two FIRST subchunks in a row (the second would be marked with error). These flags are OR'ed with possible error conditions in software such as the truncation of a too-large message. The result is encoded in the status_byte passed to the publisher.
Another type of sub-chunk is TIMEOUT. Blocks are subject to a timeout in firmware and have fixed size. It is thus possible that a block contains one or a few chunks and a TIMEOUT chunk that completes the block. If the TIMEOUT has no truncation flag set, the chunk can be discarded as it contains only zeroed bytes; if the truncation flag is set it contains also the start of a chunk that was never completed and shall not be discarded.
-
class Decoder
Block decoder associated to a one e-link. The Decoder translates blocks into chunks and published them over the network. Multiple Decoders share one Publisher.
Public Functions
-
Decoder(const Elink &elink, Publisher &publisher, flx_tohost_format fmt, int l0id_decoder_fmt, unsigned int block_size, uint64_t buf_vaddr)
Decoder class constructor.
- Parameters:
elink -- whose blocks are to be processed by this decoder.
l0id_decoder_fmt -- data format for L0ID sequence check (debug feature for detector integration).
block_size -- size of the blocks read from the DMA buffer.
buf_vaddr -- virtual address of beginning of DMA buffer (used for computation of key needed in zero-copy mode).
- Returns:
Publisher return code
-
inline uint32_t get_last_block()
- Returns:
address of the last processed block. Used in zero-copy mode.
-
inline uint64_t get_fid()
- Returns:
fid of the e-link associated to this decoder
-
inline void set_block_size(unsigned int block_size)
set the block size used by firmware
-
Publisher::Result decode(Block &block)
decode the block. This function will redirect to decode_subchunk_headers or decode_subchunk_trailers
- Parameters:
block -- address of the block to decode
- Returns:
Publisher return code
-
inline ToHostElinkStats get_decoder_stats()
- Returns:
copy of the current monitoring data
-
ToHostElinkStats get_decoder_stats_increment(ToHostElinkStats &previous)
wrapper of ToHostElinkStats::get_increment
- Returns:
a new instance of ToHostElinkStats containing the difference between the current values and the previous ones. It also updates the the "previous" value to the current one.
-
Decoder(const Elink &elink, Publisher &publisher, flx_tohost_format fmt, int l0id_decoder_fmt, unsigned int block_size, uint64_t buf_vaddr)
Monitoring
Monitoring information consists of a JSON string containing statistics regarding decoders, readers, DMA buffers and devices. The JSON string can be written to a UNIX fifo such that another application can read it and forward it where needed. The ToHostMonitor class provides methods to assemble the JSON message from the inputs.
-
class ToHostMonitor : public Monitor
Monitoring information for the felix-tohost application. ToHostMonitor gathers the hierarchical monitoring data (buffers > readers > decoders) and returns a json string.
Public Functions
-
void append_device_stats(const std::string &ts, const std::string &hostname, const ToHostDeviceStats &s)
append to the JSON message device-level information.
- Parameters:
ts -- the timestamp
hostname -- the hostname
s -- data structure containing device statistics
-
void append_dma_stats(const std::string &ts, const std::string &hostname, int deviceid, const ToHostDmaStats &s)
append to the JSON message buffer-level information.
- Parameters:
ts -- the timestamp
hostname -- the hostname
deviceid -- the device identifier.
s -- data structure containing DMA buffer statistics
-
void append_reader_stats(const std::string &ts, const std::string &hostname, int deviceid, int dma_id, const ToHostReaderStats &s)
append to the JSON message reader-level information.
- Parameters:
ts -- the timestamp
hostname -- the hostname
deviceid -- the device identifier.
dma_id -- the dma identifier
s -- data structure containing reader statistics
-
void append_elink_stats(const std::string &ts, const std::string &hostname, int device, int dma_id, int reader_id, const ToHostElinkStats &s)
append to the JSON message decoder-level information.
- Parameters:
ts -- the timestamp
hostname -- the hostname
deviceid -- the device identifier.
dma_id -- the dma identifier
reader_id -- the reader id
s -- data structure containing e-link statistics
-
void append_device_stats(const std::string &ts, const std::string &hostname, const ToHostDeviceStats &s)
Note
Monitoring counters are updated by the reader threads while monitoring information is printed out by the main application thread. Therefore thread synchronisation would be needed. However no synchronisation has been implement to avoid performance degradation. Instead the main thread makes a copy of the counters and computes the difference with respect to the previous reading. There is no need of re-setting the counters thanks to the use of 64-bit unsigned counters. The lack of thread synchronisation is minimal: it might cause two counters belonging to the same data structure to have values corresponding to slightly different times.