#237 - Add direct/indirect polling

[cliffburdick]

Adds per-queue poll_mode support for the raw ibverbs engine, with existing queues remaining indirect by default. In direct mode, the application thread services RX or submits single-packet TX directly, removing the worker-thread handoff to reduce latency. Direct queues require one application thread per queue, omit worker-specific configuration fields, and can coexist with indirect queues on the same interface. The change preserves zero-copy buffer ownership, hardware timestamps, flow steering, HDS, packet pacing, scheduled transmission, and existing APIs, while adding C++ and Python configuration support plus documentation of the behavior, restrictions, and latency-versus-progress tradeoffs.

Closes #237

[greptile-apps]

Greptile Summary

Adds per-queue poll_mode support (indirect | direct) to the raw ibverbs engine. In direct mode the calling application thread drives CQ polling and TX WQE posting inline, eliminating the worker-thread handoff to reduce latency; indirect queues retain the existing behavior as the default.

• QueuePollMode enum and poll_mode_ fields are added to RxQueueConfig / TxQueueConfig; YAML parsing, validate_config, and the ibverbs engine setup/run paths all gate on this field.
• Direct RX uses try_to_lock to detect concurrent callers and returns NOT_READY; direct TX enforces single-packet-at-a-time with a pending-burst tracker and posts WQEs synchronously.
• Docs and Python bindings are updated; AGENTS.md and docs/tutorials/configuration-walkthrough.md are not yet updated.

Confidence Score: 4/5

Safe to merge after the two unsigned commits are rebased with DCO trailers and titles corrected.

Two commits are missing required Signed-off-by trailers and non-conforming titles. The engine logic is well-structured: direct-mode CQ ownership is guarded with try-lock, the pending-burst tracker prevents double-allocation on direct TX, and validate_config catches misconfigurations before init.

Commits 417d8bc and 3ab4dbe need DCO sign-off and conforming titles. AGENTS.md should be updated to describe the direct polling mode.

Important Files Changed

Filename	Overview
src/engines/ibverbs/daqiri_ibverbs_engine.cpp	Core implementation of direct polling: new rx_poll_queue returns Status, direct RX via try-lock + CQ drain in caller thread, direct TX inline-post with single-packet enforcement and pending-burst tracking. Logic is sound.
src/engines/ibverbs/daqiri_ibverbs_engine.h	Adds poll_mode, mutex, and conflict counter fields to IbvRxQueue and IbvTxQueue; extends rx_poll_queue and get_tx_packet_burst_checked signatures. All additive and consistent with the implementation.
src/engine.cpp	Extends validate_config to check direct-mode queue constraints and propagates NOT_READY alongside NULL_PTR in the port/all-queue fan-out overloads.
src/common.cpp	Config parsing updated to read poll_mode and enforce forbidden-field rules; get_tx_packet_burst now delegates entirely to get_tx_packet_burst_checked.
include/daqiri/types.h	Adds QueuePollMode enum and poll_mode_ fields to queue config structs, defaulting to INDIRECT for backward compatibility.
include/daqiri/common.h	API doc comments updated to document NOT_READY returns and direct-mode ownership semantics.
python/daqiri_common_pybind.cpp	Exposes QueuePollMode enum and poll_mode fields to Python.
docs/api-reference/configuration.md	Documents poll_mode field with valid values, defaults, forbidden-field rules, and one-thread-per-queue restriction.
docs/api-reference/cpp.md	Adds a Direct Polling section covering NOT_READY, one-thread-per-queue contract, and direct TX single-packet restriction.
docs/api-reference/python.md	Mirrors C++ direct-polling documentation for the Python binding.
docs/concepts.md	Adds a Queue Polling Modes section explaining the indirect vs. direct polling tradeoff.

_{Reviews (3): Last reviewed commit: "#237 - Preserve indirect ibverbs empty s..." | Re-trigger Greptile}

[RamyaGuru]

Ran direct-mode tests on IGX Thor (loopback cable): RX, TX+RX, paced TX, and HDS — all worked. Verified bad configs are rejected with a descriptive warning (e.g. reorder on a direct queue) and that runtime misuse is caught (e.g. two threads on one queue).

[RamyaGuru]

Are we planning to do add this functionality to the RoCE or DPDK backends too?

[cliffburdick]

Are we planning to do add this functionality to the RoCE or DPDK backends too?

That's a good question. I don't know what the plan is for DPDK features so we'll have to sync up offline. RoCE is typically not for low latency, but it wouldn't be hard to add it in.