#233 - Add automatic RSS for multi-queue flows

[cliffburdick]

Summary

• allow RX queue actions to use ids: [queue, ...] in YAML, C++, and Python
• keep scalar and single-entry targets as direct queue steering
• automatically use flow-affine IPv4/UDP Toeplitz RSS for two or more queue IDs
• implement DPDK MARK + RSS actions and ibverbs DevX RQT/indirect-TIR destinations
• validate duplicate, unknown, eCPRI, and reorder-incompatible RSS targets
• document RSS behavior, balance expectations, and current striping limitations
• increment DAQIRI_ABI_VERSION to 1 for the public FlowAction layout change

Why

A flow previously targeted only one RX queue, preventing applications from distributing varied IPv4/UDP traffic across multiple queues while preserving per-flow affinity. Multi-queue targets now select hardware RSS automatically without changing existing scalar configurations.

User impact

Existing action.id configurations remain unchanged. Users can specify:

action:
  type: queue
  ids: [0, 1]

A fixed five-tuple remains on one queue. Roughly even packet counts require enough varied tuples with reasonably balanced traffic. Exact per-packet striping is not part of this change.

Validation

• DPDK-only, ibverbs-only, combined-engine, and Python-binding builds
• parsed all 33 existing YAML configurations
• YAML/Python validation for scalar, list, empty, duplicate, unknown, and conflicting targets
• documentation reference checker and git diff --check
• DPDK software loopback regression
• ConnectX-7 physical loopback hardware tests on both engines:
  • dynamic RSS: 440,320/440,320 packets on DPDK and 441,344/441,344 on ibverbs
  • static RSS over 1,024 tuples: 882,176/882,176 packets on both engines
  • one fixed tuple: all packets remained on one queue
  • scalar steering, flow isolation, dynamic deletion, and flow marks passed

Closes #233

[greptile-apps]

Greptile Summary

This PR adds automatic Toeplitz RSS support for multi-queue RX flows: a flow action with ids: [q0, q1, ...] now selects direct queue steering for a single entry and hardware RSS for two or more, across both the DPDK and ibverbs engines. DAQIRI_ABI_VERSION is bumped to 1 to reflect the addition of std::vector<uint16_t> ids_ to FlowAction.

• include/daqiri/types.h: FlowAction gains ids_; new helpers flow_config_actions, flow_queue_ids, and flow_queue_action_uses_rss unify single-action and multi-action flows without changing the call sites that used flow.action_ directly.
• DPDK engine: DpdkRxDestination wraps either RTE_FLOW_ACTION_TYPE_QUEUE or RTE_FLOW_ACTION_TYPE_RSS; RSS flows bypass the template (async) path and use the legacy path instead.
• ibverbs engine: resolve_rx_destination creates a DevX RQT + indirect TIR via the Toeplitz hash configuration and caches them as weak_ptrs per queue-set per port; DynamicFlowEntry and PortSteering::RuleSpec each hold a shared_ptr to keep the TIR/RQT alive until the last referencing rule is removed; shutdown ordering is corrected to release rules before TIR/RQT objects.

Confidence Score: 5/5

Safe to merge. The multi-queue RSS feature is well-scoped: single-entry and scalar configurations are untouched, validation is thorough, and the ibverbs shutdown ordering correctly releases TIR/RQT objects before queue destruction.

All changed code paths are gated by the new ids_.size() > 1 predicate, so existing single-queue flows see no change in behavior. The DPDK and ibverbs implementations use their respective native primitives correctly (RTE_FLOW_ACTION_TYPE_RSS with the Toeplitz key pointer valid for the lifetime of the local struct; DevX RQT+indirect-TIR with proper power-of-two sizing and correct big-endian byte layout). The weak_ptr cache in ibverbs avoids creating duplicate TIR/RQT objects for repeated queue sets. Validation in both the common engine layer and each engine's dynamic-flow path covers duplicates, eCPRI incompatibility, and reorder conflicts. All commits carry DCO sign-offs and follow the #233 - … title format.

No files require special attention.

Important Files Changed

Filename	Overview
include/daqiri/types.h	Adds ids_ to FlowAction after legacy fields (preserving aggregate-init compatibility), plus flow_config_actions, flow_queue_ids, and flow_queue_action_uses_rss helpers; ABI version bumped to 1.
src/rss.h	New header with the fixed 40-byte Microsoft Toeplitz RSS key and rss_table_size (next-power-of-two); shared by DPDK and ibverbs engines.
src/engine.cpp	Validation updated to use flow_config_actions; adds duplicate-queue-id, unknown-queue-id, eCPRI+RSS, and reorder-vs-RSS checks; the old "no actions" early-continue is replaced by a correct fallback through flow_config_actions.
src/common.cpp	YAML parsing extended with id/ids mutual-exclusion check, non-empty-sequence validation, and per-entry range check; backward-compatible with scalar id.
src/engines/dpdk/daqiri_dpdk_engine.cpp	Introduces DpdkRxDestination to emit either a QUEUE or an RSS action; RSS flows are gated out of the template path; add_flow forces IPv4+UDP pattern items when RSS is active; add_flex_item_flow signature changed from uint16_t queue_id to const FlowAction& queue_action.
src/engines/ibverbs/daqiri_ibverbs_engine.cpp	New resolve_rx_destination creates a DevX RQT+indirect-TIR for multi-queue flows and caches via weak_ptr; create_dr_rule_locked now takes a pre-resolved dr_action* and RssDestinationPtr; shutdown correctly clears rules then rule_specs (releasing shared_ptrs) before queue destruction.
src/engines/ibverbs/daqiri_ibverbs_engine.h	Adds RssDestination struct and RssDestinationPtr, updates DynamicFlowEntry and PortSteering::RuleSpec to hold shared ownership, and refactors create_dr_rule_locked / adds resolve_rx_destination declarations.
src/engines/ibverbs/mlx5_prm_min.h	Adds RQT create command opcode, DISP_TYPE_INDIRECT, Toeplitz hash constants, hash-field-selector constants, and the mlx5_ifc_create_rqt_in/out_bits / mlx5_ifc_rqtc_bits structures.
python/daqiri_common_pybind.cpp	Exposes ids on FlowAction to Python via def_readwrite; minimal change, correctly mirrors the C++ field.

_{Reviews (4): Last reviewed commit: "#233 - Default dynamic flow example to l..." | Re-trigger Greptile}

[dleshchev]

Tested PR 234 on the IGX against the CX-7 raw ports.

Built 593390b in daqiri-igx with DAQIRI_ENGINE="dpdk ibverbs" and ran daqiri_example_dynamic_rx_flow for the scalar q0, scalar q1, and RSS [0,1] phases.

Results:

• DPDK legacy flow path passed: RSS q0=440320, q1=440320, relative_delta=0, unexpected_flow_ids=0
• raw ibverbs passed after reducing TX num_bufs to 16384 for this device RAW_PACKET max_qp_wr: RSS q0=441344, q1=441344, relative_delta=0, unexpected_flow_ids=0

One IGX caveat: the checked-in example config with dynamic_flow_capacity: 1024 triggered mlx5 async flow setup and failed before traffic with RING: Cannot reserve memory. Setting dynamic_flow_capacity: 0 exercised the DPDK legacy flow path and verified the RSS behavior successfully.

So the core multi-queue RSS behavior looks good on both raw engines; the async-flow/example portability issue looks separate.

[cliffburdick]

Tested PR 234 on the IGX against the CX-7 raw ports.

Built 593390b in daqiri-igx with DAQIRI_ENGINE="dpdk ibverbs" and ran daqiri_example_dynamic_rx_flow for the scalar q0, scalar q1, and RSS [0,1] phases.

Results:

• DPDK legacy flow path passed: RSS q0=440320, q1=440320, relative_delta=0, unexpected_flow_ids=0
• raw ibverbs passed after reducing TX num_bufs to 16384 for this device RAW_PACKET max_qp_wr: RSS q0=441344, q1=441344, relative_delta=0, unexpected_flow_ids=0

One IGX caveat: the checked-in example config with dynamic_flow_capacity: 1024 triggered mlx5 async flow setup and failed before traffic with RING: Cannot reserve memory. Setting dynamic_flow_capacity: 0 exercised the DPDK legacy flow path and verified the RSS behavior successfully.

So the core multi-queue RSS behavior looks good on both raw engines; the async-flow/example portability issue looks separate.

Does this happen on your spark too?

[dleshchev]

Yes, I can reproduce the IGX caveat on the Spark too.

I tested PR 234 at 593390b over the two-Spark p0-to-p0 link. I used Spark-adjusted configs (host_pinned buffers) with RX on this Spark and TX on the peer Spark.

With dynamic_flow_capacity: 1024, the DPDK RX side failed before traffic with the same failure mode:

Failed to configure DPDK async flow API on port 0: fail to configure port
RING: Cannot reserve memory
Cannot start device err=-17
daqiri_init failed

With dynamic_flow_capacity: 0, the DPDK legacy flow path passed and RSS behavior looked correct:

Scalar q0 phase packets: q0=2924544 q1=0
Scalar q1 phase packets: q0=0 q1=2936832
RSS phase packets: q0=1468416 q1=1468416
RSS queue packets: relative_delta=0
unexpected_flow_ids=0

RX DPDK stats had 0 missed, errored, and out-of-buffer packets. The peer TX side sent 20,585,472 packets cleanly.

So the core DPDK multi-queue RSS behavior still looks good on Spark, but the checked-in example async-flow capacity setting has the same portability issue here as on IGX.

[cliffburdick]

Yes, I can reproduce the IGX caveat on the Spark too.

I tested PR 234 at 593390b over the two-Spark p0-to-p0 link. I used Spark-adjusted configs (host_pinned buffers) with RX on this Spark and TX on the peer Spark.

With dynamic_flow_capacity: 1024, the DPDK RX side failed before traffic with the same failure mode:

Failed to configure DPDK async flow API on port 0: fail to configure port
RING: Cannot reserve memory
Cannot start device err=-17
daqiri_init failed

With dynamic_flow_capacity: 0, the DPDK legacy flow path passed and RSS behavior looked correct:

Scalar q0 phase packets: q0=2924544 q1=0
Scalar q1 phase packets: q0=0 q1=2936832
RSS phase packets: q0=1468416 q1=1468416
RSS queue packets: relative_delta=0
unexpected_flow_ids=0

RX DPDK stats had 0 missed, errored, and out-of-buffer packets. The peer TX side sent 20,585,472 packets cleanly.

So the core DPDK multi-queue RSS behavior still looks good on Spark, but the checked-in example async-flow capacity setting has the same portability issue here as on IGX.

Resolved by defaulting to 0