Asyncmsg backend

This chapter explains some of the details of the asyncmsg backend implementation. asyncmsg is a library based on top of boost ASIO. It is mainly used for not performance critical applications that can make use of the simplicity of the asyncmsg backend.

Sending data

Buffered sending can be used to improve the performance when sending multiple small messages. Zero copy mode is also supported but will not be as performant as for RDMA as a copy is still made by boost ASIO to transfer data into kernel space. Contrary to the libfabric backend, the data for zero-copy sending does not have to reside inside a registered memory region. The most useful send feature will be the immediate sending which copies the data into a dynamically allocated buffer and sends it immediately. This method does not require the memory overhead of allocating multiple buffers and also puts puts no restrictions on the data size and does not require any guarantees from the user in terms of data stability. It is, of course, also the slowest method of sending data but for high-performance applications the libfabric backend should be used anyway.

Receiving data

Compared to the libfabric backend, the asyncmsg backend does not require the user to register memory regions for receiving data. This reduces the memory footprint of the application at the cost of performance.

Connection management

Opening and closing connections is less expensive than for the libfabric backend but still should be avoided during high load if possible.

Thread safety

The asyncmsg backend is thread-safe.

Recommendations

  • Use the asyncmsg backend for applications that do not require high performance or low latency.

  • Use the asyncmsg backend if you want to avoid the complexity of the libfabric backend.

  • Opening connections is less expensive than for the libfabric backend but still should be avoided during high load.

  • If buffered sending is used, make sure to limit the number of connections to avoid running out of memory.

  • Prefer immediate sending over zero-copy sending if performance is not critical.

  • Prefer the AsioEventLoop backend for the asyncmsg backend.

  • Make sure callbacks never block the event loop. It causes performance degradation and might deadlock the application.