Skip to content

Fix ONNX export of channels_last_3d pre-decoder models; fail fast on requested export failures#92

Merged
ivanbasov merged 1 commit into
mainfrom
fix/onnx-export-channels-last-guard
Jul 22, 2026
Merged

Fix ONNX export of channels_last_3d pre-decoder models; fail fast on requested export failures#92
ivanbasov merged 1 commit into
mainfrom
fix/onnx-export-channels-last-guard

Conversation

@ivanbasov

Copy link
Copy Markdown
Member

Problem

Since #89, exporting the surface pre-decoder to ONNX (ONNX_WORKFLOW=1/2) fails for every SafeTensors model (Fast and Accurate alike):

ONNX export failed: onnx memory_format support is not implemented

The eval-input layout match (match_input_to_model_memory_format) runs inside PreDecoderMemoryEvalModule.forward, and the inference path unconditionally converts the model to channels_last_3d before export — so tracing bakes aten::contiguous(memory_format=channels_last_3d) into the graph, which the legacy (dynamo=False) exporter cannot lower. Worse, the failure was swallowed: the run fell back to PyTorch and exited 0, so automation believed the requested artifact had been produced. (Color-code workflows are unaffected: no channels_last_3d conversion and no export on that path.)

Fix

  • training/precision.py: match_input_to_model_memory_format() is now a no-op during ONNX export (torch.onnx.is_in_onnx_export()), covering all call sites at once. The op is layout-only and ONNX has no memory-format concept, so the exported graph is value-identical — verified in the new test by comparing the ONNX reference-evaluator output against eager.
  • evaluation/logical_error_rate.py, evaluation/failure_analysis.py: a failed explicitly-requested export now raises (nonzero exit) instead of silently falling back to PyTorch. Rank 0 broadcasts the failure so multi-GPU runs exit promptly on every rank instead of blocking in the next collective. INT8 quantization failure in the ablation path now falls back to the FP32 ONNX (mirroring the LER path and the README); FP8 stays fail-fast.
  • requirements_public_inference.txt: add onnx — the legacy exporter requires it at serialization time, so ONNX_WORKFLOW=1 could not produce an artifact without it. This also lets CI run the new regression test.
  • Tests: regression test for the export guard (unittest-style so unittest discover collects it in CI); the ablation export-failure test now asserts the fail-fast contract; unit tests for the shared fail-fast helper.

Testing

  • New guard test fails on main with the exact error above and passes with the fix.
  • test_precision.py, test_failure_analysis.py, test_tensorrt_fallback.py: 95 passed, 1 skipped (tensorrt absent), including CI-style unittest discover; yapf --diff clean.
  • Standalone repro on torch 2.11 confirmed: unguarded export of a channels_last_3d Conv3d wrapper fails, guarded export succeeds, eager outputs bit-identical with and without the layout conversion.

🤖 Generated with Claude Code

…port failure

The eval-input layout match introduced with the AMP training work calls
contiguous(memory_format=channels_last_3d) inside
PreDecoderMemoryEvalModule.forward. The surface inference path converts
every model to channels_last_3d before export, so the traced graph
contains a memory-format op the legacy exporter cannot lower and
torch.onnx.export(..., dynamo=False) fails with "onnx memory_format
support is not implemented". Skip the layout match during export: ONNX
has no memory-format concept, so the artifact is value-identical.

Also stop silently downgrading to PyTorch when ONNX_WORKFLOW=1/2 was
explicitly requested and the export fails: broadcast the failure to all
ranks (so multi-GPU runs exit promptly instead of blocking in the next
collective) and raise, making local_run.sh exit nonzero. INT8
quantization failure in the ablation path now falls back to the FP32
ONNX like the LER path instead of aborting the run; FP8 stays
fail-fast.

Add onnx to the public inference requirements (the legacy exporter
needs it to serialize the requested artifact) and add a CI-collected
regression test that exports a channels_last_3d wrapper and checks the
ONNX output against eager.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ivanbasov
ivanbasov merged commit b0a32fe into main Jul 22, 2026
17 checks passed
@ivanbasov
ivanbasov deleted the fix/onnx-export-channels-last-guard branch July 22, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants