Basics

Both the asyncmsg and the libfabric backend use an event-driven model which is driven by an event loop. Events can be callbacks from the underlying network library notifying the application about new connections or incoming data or user defined callbacks added through signals and timers. The event loop guarantees that all events are executed sequentially and that no two events are executed concurrently (within the same instance). Each event is associated with a callback that contains the user code that is executed when the event is triggered.

Event loop implementation

There are two different event loop implementations: One based on the Linux kernel epoll system call (EpollEventLoop), and one based on Boost::ASIO::io_service (AsioEventLoop). Both backends are compatible with either event loop implementation, however, it is recommended to prefer the epoll backend for the libfabric backend and the Boost::ASIO backend for the asyncmsg backend. The event loop is injected into the backend through the constructor. The same event loop can be used to power multiple backends.

Note

The asyncmsg backend will always use an Boost::ASIO::io_service internally. If the epoll backend is used, the asyncmsg backend will create a thread that runs the Boost::ASIO::io_service. All events will be forwarded to the epoll event loop to maintain the guarantee that callbacks are executed sequentially. This forwarding comes at a performance penalty.

Initialization

The event loop must be managed by a shared pointer. It is not automatically started after construction and must be started manually by calling the run() function. This function will block the current thread until the event loop is stopped. To stop the event loop, the stop() function has to be called. The event loop will attempt to finish all pending events before stopping. If the event loop is flooded with events, it will eventually stop even if not all events have been processed and emit a warning. The event loop is not supposed to be restarted after it has been stopped. If the event loop is stopped, it must be destroyed and a new one must be created. If the event loop is only supposed to be run for a given amount of time it can be started using run_for(std::uint64_t s). This function will automatically stop the event loop after s seconds. A callback can be passed that is executed when the event loop is started.

auto evloop = std::make_shared<netio3::EpollEventLoop>([] { std::cout << "Event loop started\n"; });  // or netio3::AsioEventLoop
std::jthread(evloop->run();
auto ev_thread = std::jthread([&evloop]() { evloop->run(); });

// Do something
std::this_thread::sleep_for(1s);

evloop->stop();

explicit netio3::EpollEventLoop::EpollEventLoop(std::function<void()> cb_init = nullptr)

explicit netio3::AsioEventLoop::AsioEventLoop(std::function<void()> cb_init = nullptr)

virtual void netio3::BaseEventLoop::run() = 0

Runs the event loop.

The event loop is executed in the caling thread. The function blocks until the event loop is stopped.

The on_init callback is executed before the event loop starts.

virtual void netio3::BaseEventLoop::run_for(std::uint64_t sec) = 0

Runs the event loop for a given time.

The event loop is executed in the calling thread. The function blocks until the event loop is stopped or the given time has passed.

Parameters:

sec – The time to run the event loop in seconds

virtual void netio3::BaseEventLoop::stop() = 0

Stops the event loop.

Signals

A signal registeres a callback attached to an event that can be triggered by the user at any time. Triggering the signal will cause the event loop to pick it up on the next iteration and execute the callback. Signals are created using the create_signal function which returns a handle that provides a function called fire to trigger the signal. The behavior when the signal is triggered multiple times before the event loop picks it up is determined by a flag that is passed to create_signal. If useSemaphore is true the signal will callback will be executed exactly once for each time the signal is triggered. If useSemaphore is false the callback will be executed only once for all outstanding events. Subsequent triggers will of course still be picked up by the event loop in any case. If a handle is destroyed the signal is unregistered from the event loop. Any outstanding triggers will still be executed. The underlying file descriptor is passed as an argument to the callback. This parameter can be ignored if not needed but must be present in the callback signature.

auto evloop = std::make_shared<netio3::EpollEventLoop>();
auto ev_thread = std::jthread([&evloop]() { evloop->run(); });

auto signal_semaphore = evloop->create_signal([] (int) { std::cout << "Signal triggered\n"; }, true);
signal_semaphore.fire();
signal_semaphore.fire();
signal_semaphore.fire();
// Will print "Signal triggered" three times

auto signal_no_semaphore = evloop->create_signal([] (int) { std::cout << "Signal triggered\n"; }, false);
signal_no_semaphore.fire();
signal_no_semaphore.fire();
signal_no_semaphore.fire();
// Will print "Signal triggered" between one and three times depending on the timing

evloop->stop();

Note

Since all outstanding events are executed when the signal is triggered. A signal can be used as a “fire and forget” mechanism to execute a function on the event loop (synchronously with any other callbacks). This can be benefitial for thread synchronization to avoid the need for mutexes. However, if done frequently it might be more efficient to use a single signal that can be re-used in conjunction with a queue of callbacks to execute.

EventSignalHandle netio3::BaseEventLoop::create_signal(const std::function<void(int)> &cb, bool useSemaphore)

Creates a signal.

A signal is a user defined event that can be fired by the user. The callback takes the corresponding file descriptor as argument. If the callback does not need it, it can be ignored.

A signal not using semaphore logic will only be executed once even if it is fired multiple times before the callback is executed. A signal using semaphore logic will be executed for every time it is fired.

Parameters:

• cb – The callback to execute when the signal is fired

• useSemaphore – Whether to use a semaphore for the signal

Returns:

The handle to the signal

void netio3::EventSignalHandle::fire() const

Fires the signal.

Writes a byte to the file descriptor of the signal. If the write fails, an error is issued.

Timers

If a callback is supposed to be executed at a regular interval a timer can be used. Timers are created using the create_timer function which returns a handle that provides functions to start and stop the timer. If the handle is destroyed the timer is automatically stopped and unregistered. Timers can be started and stopped multiple times.

auto evloop = std::make_shared<netio3::EpollEventLoop>();
auto ev_thread = std::jthread([&evloop]() { evloop->run(); });

auto timer = evloop->create_timer([] (int) { std::cout << "Timer triggered\n"; });
timer.start(1s);
std::this_thread::sleep_for(3s);
timer.stop();

evloop->stop();

EventTimerHandle netio3::BaseEventLoop::create_timer(const std::function<void(int)> &cb)

Creates a timer.

A timer is a user defined event that is executed periodically with a given frequency. The first execution happens after the time has elapsed once (so not immediately). The callback takes the corresponding file descriptor as argument. If the callback does not need it, it can be ignored.

Parameters:

cb – The callback to execute when the timer expires

Returns:

The handle to the timer

inline void netio3::EventTimerHandle::start(const ChronoDuration auto &duration)

Starts the timer.

The timer is started with the given duration. The first call happens after the duration has elapsed once.

Parameters:

duration – The duration to start the timer with

void netio3::EventTimerHandle::stop()

Stops the timer.

General callback tips

Important

All callbacks provided to the event loop (in whatever form) should not block the event loop. Blocking callbacks will prevent the event loop from processing any other events and might lead to a deadlock if the blocking condition will therefore never be resolved. Use callbacks to convert blocking operations to non-blocking operations if necessary and move expensive operations to a separate thread.

Data can be passed to a callback by capturing it in a lambda. This is especially useful when the callback is a member function of a class and needs access to the class instance.

class Receiver {
public:
    Receiver() : m_evloop{std::make_shared<netio3::EpollEventLoop>()}} {
        // create backend, register on_data callback (irrelevant for this example)
        // ...
        auto thread = std::jthread([this]() { m_evloop->run(); });
        auto timer = m_evloop->create_timer([this](int) { on_timer(); });
        timer.start(1s);
    }
    void on_timer() const {
        m_stats.print();
    }
    void on_data(std::span<const std::uint8_t> data) const {
        m_stats.update(data);
    }
private:
    std::shared_ptr<netio3::BaseEventLoop> m_evloop;
    Stats m_stats;  // Updated on data reception
};