Add MoRI-EP/MoRI-IO test suite for MI300X + Broadcom Thor2 (RoCE) - #191
Open
raviguptaamd wants to merge 2 commits into
Open
Add MoRI-EP/MoRI-IO test suite for MI300X + Broadcom Thor2 (RoCE)#191raviguptaamd wants to merge 2 commits into
raviguptaamd wants to merge 2 commits into
Conversation
Self-contained MoRI-EP (expert-parallel dispatch/combine) and MoRI-IO
(RDMA KV transfer) test suite for AMD Instinct MI300X + Broadcom BCM57608
"Thor2" 400G RoCE, under scripts/mori_test_mi300_thor2/.
Native MoRI-EP internode was blocked on this NIC by two independent issues,
both documented and fixed here:
1. bnxt driver 235.2.86.0 — the 237/238 CQ path (setup_cq_hwqs/cq_timer)
EIOs on this firmware at bnxt_re_dv_create_cq; 235 does not. Driver-only,
no firmware reflash.
2. async_ll EP kernel — Thor2 has no PCIe atomic-completer
(AtomicOpsCap 32bit- 64bit-), so the default v1 kernel's VRAM AMO_ADD
atomics hang. async_ll uses RDMA WRITE+poll instead.
Includes a portable Dockerfile (FROM rocm/vllm-dev:...), host RDMA lib
collection (v34 vs v59 ABI fix), MoRI source build pinned to 12d1bc32, the
async_ll vLLM patch, EP/IO pair-test runners, the ClusterSphere RDMA env
recommender, a customer build guide, and an efficacy report.
Broadcom driver binaries are not redistributed; see
driver-235.2.86.0/README.md for the download link.
Co-Authored-By: Claude <noreply@anthropic.com>
raviguptaamd
requested review from
Rohan138,
amathews-amd,
coketaste,
gargrahul and
ppalaniappan-amd
as code owners
August 4, 2026 00:18
…l gitignore) - THOR2-NIC-SETTINGS.html: device-level niccli NVM options + validated RoCEv2 fabric params (GID 3 / SL 3 / TC 104), host DCQCN/PFC QoS, and MoRI/NCCL/HSA env for the customer. - MoRI-EP-Report.html: efficacy report (was force-needed; MAD root .gitignore excludes *.html). No private data (no MACs, IPs, hostnames, credentials). Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Self-contained MoRI-EP (expert-parallel dispatch/combine) and MoRI-IO (RDMA KV transfer) test suite for AMD Instinct MI300X + Broadcom BCM57608 "Thor2" 400G RoCE, added under
scripts/mori_test_mi300_thor2/.Why
Native MoRI-EP internode was blocked on this NIC by two independent issues. Both are root-caused, documented, and fixed here.
The two fixes
235.2.86.0— the 237/238 driver CQ path (setup_cq_hwqs/cq_timer) returns EIO on this firmware atbnxt_re_dv_create_cq; the 235 driver does not have that step. Driver-only, no firmware reflash.async_llEP kernel — Thor2 has no PCIe atomic-completer capability (AtomicOpsCap: 32bit- 64bit-), so the defaultv1kernel's RDMAAMO_ADDatomics into GPU VRAM hang.async_llsignals via RDMA WRITE+poll instead. (MoRI-IO always worked because it is pure RDMA WRITE.)Contents
Dockerfile(customer-handoff/) —FROM rocm/vllm-dev:..., builds MoRI from source (pinned12d1bc32), applies the async_ll patch, and bakes the host v34 RDMA userspace so no runtime lib bind-mounts are needed.collect_host_libs.sh— gathers the 5 host RDMA libs; explains the v34 vs v59 libibverbs ABI issue (the base image's v59 rejects the bnxt kernel ABI-8 provider → 0 RDMA devices).build_mori.sh,apply_async_ll_patch.py, EP/IO pair-test runners.ROCm/dist-inf-cookbook) to confirm host libs/env per NIC.ibv_devinfo→ib_write_bw→import mori→ EP pair test) and an efficacy report.Validated
ib_write_bw(near 400G line rate).Notes
scripts/mori_test_mi300_thor2/driver-235.2.86.0/README.mdfor the Broadcom download link.235.2.86.0driver installed + Docker Hub pull access to the base image + build-time internet for the MoRI clone (offline path documented).🤖 Generated with Claude Code