Conversation
No CI run has passed since 2026-05-11, and the five failures had five unrelated causes -- each verified against the runs for cb296c8, not inferred from the last green build. **Tests and Benchmarks were compiling the tree in one unit.** One pytest selection is one compilation unit, so `pytest -v` over the whole tree was a single 2,187-case build: 1,802 s on macOS (the `--mojo-timeout` default) and OOM-killed on Linux; the benchmark suite, at -O3, died the same way. Both are split by directory now. Almost all of a unit is elaborating marrow rather than the test bodies, so a directory costs about what one file costs -- `marrow/tests` measures 121 s and 5.5 GB peak, well inside a free runner -- and splitting finer would only pay for another elaboration. The six test suites run concurrently; `golden/` and `python/` are the only two that build `libmarrow.so`, which every job used to pay 13 minutes for. Benchmark results from several selections now merge into one per-commit envelope instead of the last one overwriting the rest. **The binary-size gate could not pass anywhere but the machine that recorded it.** The same source builds 0.5-1.6% larger on a macOS runner than on a developer's Mac, which trips a 0.5% threshold on its own: the job reported six REGRESSIONs against a baseline re-recorded that same day. It now measures the commit under test and the commit it descends from on one runner and compares those, caching each measurement by SHA so a run normally builds once. `baseline.json` keeps the gate list and threshold, and its numbers stay the local reference. **The docs job asked a CPU-only runner to compile a GPU listing.** `DeviceContext()` is instantiated against the host's accelerator, so `guide/gpu.qmd`'s two blocks failed with `function instantiation failed` on ubuntu-latest no matter how correct they were. They are `{.mojo .gpu}` now, compiled where an accelerator exists and skipped where none does. **Integration hit Boost.Process.** Arrow C++ needs explicit per-release support for it (apache/arrow#43746, #45121) and apache-arrow-24.0.0 predates conda-forge's 1.91, so `libarrow_testing.so` failed to link. Boost is pinned below 1.88; the lock change is confined to the integration environment. Also: `wheels.yml` has not run since 2026-05-11 and is manual-only until a leg passes on a real run, with what is known to be wrong recorded in it; the mojo pin in `python/pyproject.toml` was four months stale; actions and setup-pixi are on current majors; `MODULAR_HOME` was pointing at a directory nothing reads; and every workflow now has a concurrency group and a timeout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Splitting the suites let CI reach code it had not compiled since 2026-05-11, and four real faults were waiting there. None of them is a CI-config problem. **`marrow/parquet/tests/test_writer.mojo` aborted the whole test binary.** Four columns were built as `pa.array(numpy.arange(n), ...)`; PyArrow wraps a numpy buffer zero-copy at 16-byte alignment and `Buffer.from_foreign` asserts 64, so `_to_marrow` hit `Assert Error: Buffer pointer must be 64-byte aligned` and took the shared driver with it -- reporting all 54 cases as failed with no indication which one did it, which is why the selection's own output could not name it and a per-file sweep had to. This file already used the list-backed idiom elsewhere and documents the hazard in a comment; now it uses it throughout. macOS survived on alignment luck. The underlying gap is marrow's -- the C Data Interface calls alignment "recommended, but not required" -- and is recorded in the backlog rather than fixed here. **`_ldd_deps` ran `ldd` with `check=True`.** One `test_compile.py` case builds a 4-byte stub library; `otool` reports "not an object file" and exits 0, so macOS never noticed, while `ldd` exits 1 and raised. An inspector that cannot read a file now warns and reports no dependencies, which is the judgement `stage_codec_libs` already makes about a codec that is not installed. **`marrow/expr/tests` does not fit a free runner**, and measurement says why: 17.5 GB peak, of which `test_optimizer.mojo` alone is 15.4 GB. Peak follows the heaviest instantiations, not the file count -- 6 of the 13 files peak the same 17.3 GB -- so the directory is split by lane, `test_optimizer.mojo` is its own job, and the Linux runners get a 24 GB swapfile on /mnt, because one file is the floor and that floor is above 16 GB. **The Benchmarks job was benchmarking by naming directories**, and `--benchmark` adds the bench files rather than replacing the tests -- so it also compiled and ran every test unit, including `marrow/expr`. It names `bench_*` files now, and gets the same swap and a 3,600 s compile budget; its own -O3 unit had been timing out at the 1,800 s default. Also: the Linux asan job is gone rather than silently `if: false` -- the same selection is 5m15s on macOS and had not finished after 60 minutes on Linux, measured twice, and the cases are identical on both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`DictionaryArray.__eq__` compared decoded values as `self[i].value() != other[i].value()` -- two `DynScalar`s. Comparing *elements* of an erased container is exactly what `ArrayData.__eq__`'s docstring warns about two thousand lines up: `DynArray.__eq__` and the nested arrays' `__eq__` become mutually recursive at instantiation and the elaborator never resolves it, parking at 0% CPU with no diagnostic. It did not fire from `marrow/tests/test_arrays.mojo`, which is how it survived. Reached from `test_ipc.mojo` it deadlocked `mojo build`: 6m40s of wall clock for 11.9s of user CPU, on Linux on both architectures, while macOS compiled it -- so CI could not see it either, because no Linux suite has run since 2026-05-11. `test_ipc.mojo` was killed at the 1,800s timeout with all 37 cases reported failed and nothing to say which two were responsible; a per-file then per-case sweep named them as the two dictionary round-trips, and deleting just the `got == expected` assertion made the case compile in 22s. Resolving the values dtype to a concrete array type first -- bool directly, binary-like and primitive through their `dispatch_*` -- means no typed `__eq__` is ever named through an erased one. Decoded-value semantics are unchanged, so `eq_permuted_dictionary` (logical equality, arrow-rs) and `eq_respects_offset` still hold. A dictionary over a *nested* value type has no typed leaf to compare and answers on the representation instead, which reads a permuted one as unequal rather than hanging; Arrow allows that shape and nothing in marrow builds it. `test_ipc.mojo` now compiles and passes in 46s on Linux, and `marrow/tests` is 575 passed there. macOS: 734 passed across `marrow/tests` and the kernel suites that use dictionaries. Found with a native arm64 Linux container, added here alongside: `Dockerfile.x86` becomes `Dockerfile.linux` with the hardcoded platform dropped -- compose.yaml sets it per service -- and `pixi.toml` declares `linux-aarch64`, for which the pinned mojo and MAX both publish builds. On an Apple Silicon machine `arm64` is native and `x86` needs Rosetta: Docker's default qemu CPU stops at SSE4.2, and the linux-64 mojo binary dies with `Illegal instruction` before printing its version. The two are not interchangeable -- this hang reproduces on arm64, `test_scalars.mojo`'s crash does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
**Grouping bench files is worse, not better.** Five kernel bench files as one -O3 unit did not finish compiling in 3,600 s; the same five as five units compile in 165 s in total (20 + 10 + 86 + 12 + 37, measured on Linux). The combined unit blows up superlinearly, so paying for one elaboration per file is the cheap option -- the opposite of the tests, where one directory per unit is what fits. Benchmarks now run one file at a time. **`pytest-benchmark` was pinned `>=5.2.3,<6` and the harness cannot take 5.3.** `conftest.py` constructs a `BenchmarkFixture` directly with pytest-benchmark's own keyword API, and 5.3.0 added required arguments to that constructor: every benchmark fails with `BenchmarkFixture.__init__() missing 2 required positional arguments` before anything is compiled. Both versions satisfied `<6`, so which one you got depended on the solve -- adding `linux-aarch64` picked 5.3.0 there and 5.2.3 on the two platforms CI runs, which is the only reason CI never saw it. Pinned `<5.3`; all sixteen entries in the lock now agree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ci: make every workflow run again
No CI run has passed since 2026-05-11, and the five failures had five
unrelated causes -- each verified against the runs for cb296c8, not
inferred from the last green build.
Tests and Benchmarks were compiling the tree in one unit. One pytest
selection is one compilation unit, so
pytest -vover the whole tree was asingle 2,187-case build: 1,802 s on macOS (the
--mojo-timeoutdefault) andOOM-killed on Linux; the benchmark suite, at -O3, died the same way. Both are
split by directory now. Almost all of a unit is elaborating marrow rather than
the test bodies, so a directory costs about what one file costs --
marrow/testsmeasures 121 s and 5.5 GB peak, well inside a free runner --and splitting finer would only pay for another elaboration. The six test
suites run concurrently;
golden/andpython/are the only two that buildlibmarrow.so, which every job used to pay 13 minutes for. Benchmark resultsfrom several selections now merge into one per-commit envelope instead of the
last one overwriting the rest.
The binary-size gate could not pass anywhere but the machine that recorded
it. The same source builds 0.5-1.6% larger on a macOS runner than on a
developer's Mac, which trips a 0.5% threshold on its own: the job reported six
REGRESSIONs against a baseline re-recorded that same day. It now measures the
commit under test and the commit it descends from on one runner and compares
those, caching each measurement by SHA so a run normally builds once.
baseline.jsonkeeps the gate list and threshold, and its numbers stay thelocal reference.
The docs job asked a CPU-only runner to compile a GPU listing.
DeviceContext()is instantiated against the host's accelerator, soguide/gpu.qmd's two blocks failed withfunction instantiation failedonubuntu-latest no matter how correct they were. They are
{.mojo .gpu}now,compiled where an accelerator exists and skipped where none does.
Integration hit Boost.Process. Arrow C++ needs explicit per-release
support for it (apache/arrow#43746, #45121) and apache-arrow-24.0.0 predates
conda-forge's 1.91, so
libarrow_testing.sofailed to link. Boost is pinnedbelow 1.88; the lock change is confined to the integration environment.
Also:
wheels.ymlhas not run since 2026-05-11 and is manual-only until a legpasses on a real run, with what is known to be wrong recorded in it; the mojo
pin in
python/pyproject.tomlwas four months stale; actions and setup-pixiare on current majors;
MODULAR_HOMEwas pointing at a directory nothingreads; and every workflow now has a concurrency group and a timeout.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
🤖 Generated with Claude Code