Sync with Microsoft ONNX Runtime - 05082026 - #1244
Open
ai-fw-intg wants to merge 12 commits into
Open
Conversation
…icrosoft#31350) ### Description The uint8 → half/float dequantization in the 8-bit `MatMulNBits` GEMV kernels is **issue-bound, not bandwidth-bound**. Replacing the per-element integer-to-float converts with the "magic half" byte-permute decode makes the M=1 kernel **18% faster**, with **bit-identical** output. `ncu` on an H200 (SM90), 8-bit `MatMulNBits` with K=2048, N=248320 (an int8 `lm_head`): | metric | value | |---|---| | Compute (SM) Throughput | **86.46 %** | | DRAM Throughput | 35.02 % | | L1/TEX Cache Throughput | 66.00 % | | Achieved Occupancy | 94.51 % | | Executed Ipc Active | **3.49** inst/cycle (of 4) | Occupancy is already ~95% and DRAM is only a third utilized, so the kernel is limited by the ALU/convert instructions in the dequantization rather than by weight traffic. The fix is to issue fewer instructions per unpacked weight. ### Approach The half bit pattern `0x6400 | q` encodes exactly `1024 + q` for any `q ∈ [0, 255]`: the `2^10` exponent makes the low 10 mantissa bits integer-valued with `ulp == 1`. A single `__byte_perm` against the constant `0x64646464` therefore materializes **two halves at once**, so 8 `u16→half` converts plus 4 pack operations collapse into 4 permutes. Subtracting the matching biased zero point `0x6400 | zp` (i.e. `1024 + zp`) cancels the `1024` offset exactly. This is the same technique already vendored in this repository — see `start_byte_for_fp16 = 0x64646464` in [`onnxruntime/contrib_ops/cuda/llm/cutlass_extensions/interleaved_numeric_conversion.h`](onnxruntime/contrib_ops/cuda/llm/cutlass_extensions/interleaved_numeric_conversion.h#L54) (CUTLASS's `FastInterleavedAndBiasedNumericArrayConverter`) — applied here to the non-interleaved MatMulNBits layout. **This is bit-identical to the previous code, not an approximation.** Both operands are exact halves and the result `q - zp` lies in `[-255, 255]`, which half represents exactly, so no rounding is introduced anywhere. ### Key Changes | Function | Used by | Change | |---|---|---| | `AccumulateEightElements8b` (half overload) | `MatMulFloat8bKernelM1` | 8 converts + 4 packs → 4 `__byte_perm` | | `DequantizeEight8b<T>` | `MatMulFloat8bKernelBatched` | 8 int→float converts + 8 subtracts → 4 permutes + 4 `__hsub2` | The pre-existing scalar loop is retained under `#else` for `__CUDA_ARCH__ < 530`, matching the guard already used by the half accumulate path. No API, kernel-launch, or dispatch changes. ### Results Measured on H200 (SM90), nsys median over ~60 launches, N=248320, K=2048: | kernel | before | after | | |---|---:|---:|---:| | `MatMulFloat8bKernelM1` | 248.6 µs | **203.0 µs** | **−18%** | | `MatMulFloat8bKernelBatched` | 301.6 µs | 299.6 µs | −1% | `ncu` after the change shows Compute SM 87.2% / DRAM 42.9% / L1-TEX 82.5% — the kernel has moved off the instruction-issue limit and toward the L1 limit. The batched kernel gains little because it is **not** issue-bound: L1/TEX throughput is 98.5% and occupancy is capped at 34.9% (3 blocks, 80 registers/thread). Its bottleneck is activation re-reads — the `[M,K]` activation tile is re-read by every one of the N/16 blocks — which needs shared-memory staging and a register reduction. That is left to a separate change. ### Testing Notes Existing coverage exercises both modified paths; because the transform is bit-identical, no test changes are required. ```bash ninja -j32 onnxruntime_provider_test ./onnxruntime_provider_test --gtest_filter='*MatMul8Bits*:*MatMulNBits*:*MatMul8bits*' ``` Result: **74 tests, 71 PASSED, 0 FAILED**, 3 skipped (`DynamicZeroPoints_AsymmetricCompInt8`, `SharedPrepackedWeights_DynamicZeroPoints_AsymmetricCompInt8`, `Float16_Comprehensive` — all skipped on `main` as well). Reviewers may want to sanity-check the `__byte_perm` selector nibbles: bytes 0-3 come from `x`, bytes 4-7 from `y`, and nibble 0 (LSB) selects result byte 0. So `__byte_perm(lo32, 0x64646464, 0x4140)` yields `{q0, 0x64, q1, 0x64}`, i.e. the two halves `1024+q0` and `1024+q1` (`half2` stores `.x` in the low 16 bits).
To avoid build error like ``` C:\Users\cloudtest\AppData\Local\Temp\tmpxft_00000bec_00000000-7_matmul_block_scaled_fp4_sm120.compute_120.cudafe1.stub.c(93): error C2719: 'unnamed-parameter': formal parameter with requested alignment of 128 won't be aligned ```
### Description Add `ORT_API_CALL` to WebGPU data-transfer callbacks so their calling convention matches OrtDataTransferImpl. ### Motivation and Context On Windows x86, ORT_API_CALL expands to __stdcall. Without it, assigning CanCopyImpl, CopyTensorsImpl, and ReleaseImpl to the C API callback table fails due to incompatible function-pointer types. This is ABI-neutral on Windows x64/ARM64 and non-Windows platforms. Validated by building ONNX Runtime with WebGPU for Windows x86 and running DLL loading, environment creation, WebGPU registration, and Win32k sandbox tests.
) ## Description CUDA 13 generates host stubs with 128-byte aligned by-value CUTLASS parameters for the native SM120 TMA kernels. MSVC rejects those stubs with C2719. Skip the SM120 TMA object libraries only for MSVC builds using CUDA 13 or newer. Non-MSVC builds and MSVC builds with older CUDA versions retain native SM120 support. The MatMulBlockQuantizedFp4Weight native calls are guarded by ORT_ENABLE_BLOCKQUANT_SM120, which is defined only when the object library is created. Windows SM120 therefore uses the existing fused GEMV or dequantize-plus-cuBLAS fallback rather than referencing missing kernels. SM120 grouped MoE dispatch separately checks whether its TMA implementation was compiled. ## Validation Built the CUDA plugin target on Windows with MSVC 14.44 and CUDA 13.0: cmake --build build_plugin/Release --config Release --target onnxruntime_providers_cuda_plugin --parallel 4 The build completed and produced onnxruntime_providers_cuda.dll without compiling the SM120 TMA object target.
The Windows ARM64 CUDA plugin packaging job passes `--skip_tests`, but that option skips test execution only; it does not disable generation and compilation of unit-test targets. Because `onnxruntime_BUILD_UNIT_TESTS` defaults to ON, the job still builds `onnxruntime_mlas_test`. In the failing CI run, the MLAS library itself compiled and linked successfully, including `sqnbitgemm_kernel_avx512_2bit.cpp`. The later failure was an ARM64 MSVC compiler crash while compiling the unused `onnxruntime_mlas_test` target: `CL.exe` exited with code `57005` (`0xDEAD`) during `test_sqnbitgemm_2bit.cpp`. This change sets `onnxruntime_BUILD_UNIT_TESTS=OFF` only for the Windows ARM64 CUDA plugin packaging job. The CUDA plugin and MLAS library are still built, and the existing plugin binary verification remains enabled; only unit-test targets that are not run by this packaging job are excluded. Note: A more complete fix is to split sqnbitgemm_kernel_avx512_2bit.cpp into multiple files. Validation: - YAML parsed successfully. - Confirmed the define is present only in the ARM64 CUDA plugin build block. - `git diff --check` passed. - Windows ARM64 CI build remains to be verified by this PR.
Fix build errors in aarch64:
```
/onnxruntime_src/include/onnxruntime/core/framework/float4.h: In member function ‘std::pair<float, float> onnxruntime::Float4E2M1x2::ToFloat2() const’:
/onnxruntime_src/include/onnxruntime/core/framework/float4.h:173:67: note: parameter passing for argument of type ‘std::pair<float, float>’ when C++17 is enabled changed to match C++14 in GCC 10.1
173 | inline ORT_HOST_DEVICE std::pair<float, float> ToFloat2() const {
| ^
```
There is another build error in sm120 that data type shall be void*
instead of Stream* for cuda plugin.
Change Stream to auto, also add sm120 to CI pipeline to avoid similar
error in the future.
### Description - Adds int64 data type support to the Tile operator in the WebGPU execution provider. The op is used by [yolo26n-pose](https://huggingface.co/webnn/yolo26n-pose-ONNX/blob/main/onnx/model_fp16.onnx) and other models that tile int64 tensors. - Adds int64 data type support to the Concat operator in the WebGPU execution provider. The op is used by [whisper-base-decoder iobinding](https://huggingface.co/webnn/whisper-base-webnn/blob/main/whisper_base_decoder_static_kvcache_128_lm_fp16_layernorm_gelu_4dmask_iobinding.onnx) model that concat int64 tensors. ### Motivation and Context The WebGPU Tile and Concat kernels were registered with WebGpuSupportedNumberTypes() (float/fp16/int32/uint32 only), causing int64 tensors to fall back to CPU. Beyond the missing type constraint, the WGSL getter for Int64 variables returns only the low 32 bits (i32(buf[offset].x)), and the default setter sign-extends from that i32 — silently corrupting any value with non-zero high bits. Since Tile and Concat are pure data-movement with no arithmetic on element values, int64 is safe to support by treating each element as an opaque vec2<u32> copy. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Ningxin Hu <ningxin.hu@intel.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…31622) microsoft#31616 adds more cuda arch, but did not reduce nvcc_threads. That might cause out of memory.
### Description <!-- Describe your changes. --> Throw error on negative split axis ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…crosoft#31151) ### Description Extend the Blob-backed on-demand external data loading (microsoft#29477) from CPU / WebGPU EP to the WebNN EP. Route WebNN external initializers through `LoadWebAssemblyExternalData` instead of the custom `webnnRegisterMLConstant` path: `RegisterConstant` streams each initializer's byte range into a scratch buffer and builds the constant via the existing in-memory path. ### Motivation and Context - Load-time memory (JSPI): the range is read from the mounted Blob on demand and released right after copy, so JS heap peak drops from full model size to the largest single initializer (same win as microsoft#29477). - Fixes int32 offset/size truncation (loader uses `double/SafeInt<size_t>`), so big models split across multiple `.data` files load correctly. - Removes the WebNN constant glue (`webnnRegisterMLConstant`, `registerMLConstant`, etc.).
…soft#29716) ### Description <!-- Describe your changes. --> Add plans for deprecating and removing JSEP and WebGL in onnxruntime-web. The native WebGPU EP should be the preferred path going forward. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Share plans and get feedback before implementation. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
ai-fw-intg
requested review from
Jaswanth51,
ankitm3k,
jatinwadhwa921 and
vthaniel
August 4, 2026 20:35
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated daily backmerge from ORT main to ovep-develop. No conflicts detected. Do NOT squash or rebase - use merge commit only.