update 57.2.0 to 57.3.0 - #15
Open
mandrush wants to merge 909 commits into
Open
Conversation
Documentation follow-up for #9755. Comments only — no code changes. Documents the `InProgressArray` copy methods and clarifies which operate on the source set via `set_source` versus the one that takes the source directly. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Which issue does this PR close? closes #10471 # Rationale for this change The Miri workflow currently covers only a subset of packages. This adds more miri coverage to detect unsafe code, Parquet crate remains excluded because it is tracked separately in #614. # What changes are included in this PR? - Adds Miri testing for arrow, arrow-avro, arrow-cast, arrow-csv, arrow-ipc, arrow-json, arrow-row, arrow-select, and arrow-string. - Updates the gh action workflow path filters in `miri.yaml`, some crates were missing before - adds ignore to tests that takes too long to run - adds ignore to tests that were failing due to inline assembly - Sets `INSTA_WORKSPACE_ROOT` to prevent snapshot tests from invoking unsupported subprocess operations under Miri note: it takes about 50mins to run miri on my M3 laptop. when this pr gets merge i think it's expected to prolong the CI time in the miri workflow, but we can always add more parallel workers # Are these changes tested? i ran the following to test locally. - `rustup run nightly bash .github/workflows/miri.sh` (runs the whole miri suite) - `cargo test` - `cargo clippy --workspace --all-targets --all-features -- -D warnings` - `git diff --check` # Are there any user-facing changes? no, only tests # AI disclosure I used codex to draft this pr after i got the project and miri tests running on my laptop. I have reviewed all the code and have built, tested these changes. I am happy to own follow-ups too. --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
# Which issue does this PR close? - Closes #10491. # Rationale for this change PyO3's `experimental-inspect` feature records the Python type of every conversion into the built binary, so that tools like `maturin generate-stubs` can emit a real `.pyi`. The type comes from the `INPUT_TYPE` / `OUTPUT_TYPE` associated constants on `FromPyObject` / `IntoPyObject`. `PyArrowType` implements both traits but leaves those constants at their default, `_typeshed.Incomplete`. Because `PyArrowType` is how bindings exchange *every* Arrow value with Python, that one default erases the entire public API of any such binding. A function like ```rust #[pyfunction] fn cast_record_batch( record_batch: PyArrowType<RecordBatch>, schema: PyArrowType<Schema>, ) -> PyResult<PyArrowType<RecordBatch>> { ... } ``` generates ```python def cast_record_batch(record_batch: Incomplete, schema: Incomplete) -> Incomplete: ... ``` # What changes are included in this PR? - A new `experimental-inspect` feature on `arrow-pyarrow`, which just turns on PyO3's. `arrow` gets a matching feature that implies `pyarrow`. - `FromPyArrow::INPUT_TYPE`, `ToPyArrow::OUTPUT_TYPE` and `IntoPyArrow::OUTPUT_TYPE`, defaulting to `_typeshed.Incomplete` so that out-of-tree implementors are unaffected. - Implementations for every type the crate converts: `DataType`, `Field`, `Schema`, `ArrayData` (→ `pyarrow.Array`), `RecordBatch`, `Vec<T>`, `ArrowArrayStreamReader`, `Box<dyn RecordBatchReader + Send>` and `Table`. - `PyArrowType`'s `FromPyObject` / `IntoPyObject` impls forward them. - Unit tests pinning the rendered hint for each type, and a note in the crate docs. Input and output are separate constants because they genuinely differ: `Vec<T>` is built from anything iterable but handed back as a `list`, so it is `collections.abc.Iterable[pyarrow.RecordBatch]` in and `list[pyarrow.RecordBatch]` out. # Are these changes tested? Yes. `arrow-pyarrow` gains unit tests asserting the rendered hint for every conversion, run with `cargo test -p arrow-pyarrow --features experimental-inspect`. Beyond that, the change was verified end to end against a real binding: with `arrow` patched to this branch and the binding's own code left on plain `PyArrowType`, `maturin generate-stubs` produces the signatures and the `from pyarrow import ...` line shown above. # Are there any user-facing changes? No behaviour changes, and nothing is compiled unless `experimental-inspect` is enabled. The traits gain associated constants, but they are defaulted and feature-gated, so existing implementations keep compiling untouched. ---------
…length does not match its range (#10564) # Which issue does this PR close? Closes #10563. # Rationale for this change A short read pushed into the parquet push decoder panics at `PushBuffers::push_range` ("Range length must match buffer length") although every public entry point that feeds it returns `Result`. Through the async reader this turns a transient store fault into a reader-thread panic the caller cannot classify or retry; the json and avro readers surface the equivalent fault as a decode error. # What changes are included in this PR? `PushBuffers::push_range` and `push_ranges` return `Result<(), ParquetError>` instead of asserting (buffer/range length mismatch, and ranges/buffers count mismatch in `push_ranges`). The error propagates through the crate-internal chain: `ParquetMetaDataPushDecoder::push_range`/`push_ranges` (already `Result`, now use `?`), and `ParquetDecoderState::push_data` -> `RemainingRowGroups::push_data` -> `RowGroupReaderBuilder::push_data`, the last two becoming fallible; all their callers were already in `Result` contexts. # Are these changes tested? New unit tests in `push_buffers.rs` cover the accepted case, the short-buffer error, and the count-mismatch error. The existing parquet test suite passes. # Are there any user-facing changes? No stable API change: the public decoder entry points keep their `Result` signatures. `PushBuffers` (exported only under the `experimental` feature, which carries no stability guarantees) changes `push_range`/`push_ranges` from `fn(...)` to `fn(...) -> Result<(), ParquetError>`. Code that previously panicked on mismatched pushes now receives an `Err`.
# Which issue does this PR close? N/A # Rationale for this change Remove deprecated functions from the public API. # What changes are included in this PR? Removed functions and tests that used them. # Are these changes tested? Covered by existing tests # Are there any user-facing changes? Yes, removes functions from the public API
Bumps the codeql-actions group with 2 updates: [github/codeql-action/init](https://github.com/github/codeql-action) and [github/codeql-action/analyze](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.37.4 to 4.37.5 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/releases">github/codeql-action/init's releases</a>.</em></p> <blockquote> <h2>v4.37.5</h2> <ul> <li>Fixed a bug where a network error while streaming the download of the CodeQL bundle could terminate the <code>init</code> Action instead of falling back to downloading the bundle before extracting it. <a href="https://redirect.github.com/github/codeql-action/pull/4061">#4061</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action/init's changelog</a>.</em></p> <blockquote> <h1>CodeQL Action Changelog</h1> <p>See the <a href="https://github.com/github/codeql-action/releases">releases page</a> for the relevant changes to the CodeQL CLI and language packs.</p> <h2>[UNRELEASED]</h2> <p>No user facing changes.</p> <h2>4.37.6 - 04 Aug 2026</h2> <ul> <li>Changed the default filepath for the new remote file address format that was introduced in CodeQL Action 4.37.0 / 3.37.0 to <code>.github/codeql-config.yml</code> to align it with the suggested path that is used elsewhere. <a href="https://redirect.github.com/github/codeql-action/pull/4070">#4070</a></li> </ul> <h2>4.37.5 - 03 Aug 2026</h2> <ul> <li>Fixed a bug where a network error while streaming the download of the CodeQL bundle could terminate the <code>init</code> Action instead of falling back to downloading the bundle before extracting it. <a href="https://redirect.github.com/github/codeql-action/pull/4061">#4061</a></li> </ul> <h2>4.37.4 - 29 Jul 2026</h2> <ul> <li>This version of the CodeQL Action adds support for the <code>tools</code> input for the <code>codeql-action/init</code> step to be specified using a <code>github-codeql-tools</code> <a href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">repository property</a>. This feature will gradually be rolled out following the release of this version. Once rolled out, this allows for the CodeQL CLI version that is used in GitHub-managed workflows, such as Default Setup, to be set to a custom value. For example, customers who run into issues with rate limits when a new CodeQL CLI version is released can set the value to <code>toolcache</code> to always use the CodeQL CLI version that is available in the runner toolcache. For Advanced Setup workflows, the value provided for <code>tools</code> in the workflow definition always takes precedence unless the value of the repository property starts with <code>!</code>. <a href="https://redirect.github.com/github/codeql-action/pull/4037">#4037</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.2">2.26.2</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4051">#4051</a></li> </ul> <h2>4.37.3 - 22 Jul 2026</h2> <p>No user facing changes.</p> <h2>4.37.2 - 21 Jul 2026</h2> <ul> <li>The new address format for the <code>config-file</code> input that was introduced in CodeQL Action 4.37.0 is now enabled by default. In addition to the format described there, the <code>remote=</code> prefix can now be used to explicitly indicate that the input refers to a remote file. All previous input formats continue to be accepted as well. <a href="https://redirect.github.com/github/codeql-action/pull/4023">#4023</a></li> <li>The CodeQL Action can now make use of <a href="https://docs.github.com/en/code-security/how-tos/secure-at-scale/configure-organization-security/manage-usage-and-access/giving-org-access-private-registries">configured private registries</a> in Default Setup to retrieve CodeQL configuration files from remote repositories that require authentication. This will allow customers to store their CodeQL configuration in a single repository that can then be referenced by Default Setup workflows in other repositories. We expect to roll this and other, related changes out to everyone in July. <a href="https://redirect.github.com/github/codeql-action/pull/4007">#4007</a></li> </ul> <h2>4.37.1 - 16 Jul 2026</h2> <ul> <li><em>Upcoming breaking change</em>: Add a deprecation warning for customers using CodeQL version 2.20.6 and earlier. These versions of CodeQL were discontinued on 1 July 2026 alongside GitHub Enterprise Server 3.16, and will be unsupported by the next minor release of the CodeQL Action. <a href="https://redirect.github.com/github/codeql-action/pull/3956">#3956</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.1">2.26.1</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4019">#4019</a></li> </ul> <h2>4.37.0 - 08 Jul 2026</h2> <ul> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.0">2.26.0</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3995">#3995</a></li> <li>In addition to the existing input format, the <code>config-file</code> input for the <code>codeql-action/init</code> step will soon support a new <code>[owner/]repo[@ref][:path]</code> format. All components except the repository name are optional. If omitted, <code>owner</code> defaults to the same owner as the repository the analysis is running for, <code>ref</code> to <code>main</code>, and <code>path</code> to <code>.github/codeql-action.yaml</code>. Support for this format ships in this version of the CodeQL Action, but will only be enabled over the coming weeks. <a href="https://redirect.github.com/github/codeql-action/pull/3973">#3973</a></li> </ul> <h2>4.36.3 - 01 Jul 2026</h2> <p>No user facing changes.</p> <h2>4.36.2 - 04 Jun 2026</h2> <ul> <li>Cache CodeQL CLI version information across Actions steps. <a href="https://redirect.github.com/github/codeql-action/pull/3943">#3943</a></li> <li>Reduce requests while waiting for analysis processing by using exponential backoff when polling SARIF processing status. <a href="https://redirect.github.com/github/codeql-action/pull/3937">#3937</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.6">2.25.6</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3948">#3948</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/github/codeql-action/commit/d1ba80a13dd99fba24a470575428917156a28b43"><code>d1ba80a</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4067">#4067</a> from github/update-v4.37.5-1cd4d01d5</li> <li><a href="https://github.com/github/codeql-action/commit/e74600b0d945db9734eb044f95cd43f34b773451"><code>e74600b</code></a> Update changelog for v4.37.5</li> <li><a href="https://github.com/github/codeql-action/commit/1cd4d01d58f636bbdbffbfd3c121b3446d07e9f3"><code>1cd4d01</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4061">#4061</a> from github/henrymercer/turbo-system</li> <li><a href="https://github.com/github/codeql-action/commit/d2bfc30bc3373a3ec945dc3f93bc8b8f07fa16b9"><code>d2bfc30</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4050">#4050</a> from github/mbg/status/registries</li> <li><a href="https://github.com/github/codeql-action/commit/68028fcb1605f3cdd37e4e1845c3f78af017a3ea"><code>68028fc</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4062">#4062</a> from github/sam-robson/migrate-enterprise-release-pat</li> <li><a href="https://github.com/github/codeql-action/commit/c29563eeaafbc75499c7bb0d74bf77b3506c1cbd"><code>c29563e</code></a> ci: use federated enterprise release PAT</li> <li><a href="https://github.com/github/codeql-action/commit/155e5229973b426bd1ae2f83bb1bf42417fa2a8f"><code>155e522</code></a> Link the PR from the changelog entry</li> <li><a href="https://github.com/github/codeql-action/commit/2d3b351ea6452a9b21346f8d64567e5b833924de"><code>2d3b351</code></a> Handle network errors when streaming the CodeQL bundle download</li> <li><a href="https://github.com/github/codeql-action/commit/5d3eb98e4a780bfe1a53f57fbdc278ea5da1274b"><code>5d3eb98</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4055">#4055</a> from github/dependabot/npm_and_yarn/npm-minor-203262...</li> <li><a href="https://github.com/github/codeql-action/commit/c5f739bd6460e096aaf40962517e46a846b8b6b1"><code>c5f739b</code></a> Merge branch 'main' into dependabot/npm_and_yarn/npm-minor-2032624187</li> <li>Additional commits viewable in <a href="https://github.com/github/codeql-action/compare/f205ea1c3313d32999d8d6a48b4f6530d4437b38...d1ba80a13dd99fba24a470575428917156a28b43">compare view</a></li> </ul> </details> <br /> Updates `github/codeql-action/analyze` from 4.37.4 to 4.37.5 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/releases">github/codeql-action/analyze's releases</a>.</em></p> <blockquote> <h2>v4.37.5</h2> <ul> <li>Fixed a bug where a network error while streaming the download of the CodeQL bundle could terminate the <code>init</code> Action instead of falling back to downloading the bundle before extracting it. <a href="https://redirect.github.com/github/codeql-action/pull/4061">#4061</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action/analyze's changelog</a>.</em></p> <blockquote> <h1>CodeQL Action Changelog</h1> <p>See the <a href="https://github.com/github/codeql-action/releases">releases page</a> for the relevant changes to the CodeQL CLI and language packs.</p> <h2>[UNRELEASED]</h2> <p>No user facing changes.</p> <h2>4.37.6 - 04 Aug 2026</h2> <ul> <li>Changed the default filepath for the new remote file address format that was introduced in CodeQL Action 4.37.0 / 3.37.0 to <code>.github/codeql-config.yml</code> to align it with the suggested path that is used elsewhere. <a href="https://redirect.github.com/github/codeql-action/pull/4070">#4070</a></li> </ul> <h2>4.37.5 - 03 Aug 2026</h2> <ul> <li>Fixed a bug where a network error while streaming the download of the CodeQL bundle could terminate the <code>init</code> Action instead of falling back to downloading the bundle before extracting it. <a href="https://redirect.github.com/github/codeql-action/pull/4061">#4061</a></li> </ul> <h2>4.37.4 - 29 Jul 2026</h2> <ul> <li>This version of the CodeQL Action adds support for the <code>tools</code> input for the <code>codeql-action/init</code> step to be specified using a <code>github-codeql-tools</code> <a href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">repository property</a>. This feature will gradually be rolled out following the release of this version. Once rolled out, this allows for the CodeQL CLI version that is used in GitHub-managed workflows, such as Default Setup, to be set to a custom value. For example, customers who run into issues with rate limits when a new CodeQL CLI version is released can set the value to <code>toolcache</code> to always use the CodeQL CLI version that is available in the runner toolcache. For Advanced Setup workflows, the value provided for <code>tools</code> in the workflow definition always takes precedence unless the value of the repository property starts with <code>!</code>. <a href="https://redirect.github.com/github/codeql-action/pull/4037">#4037</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.2">2.26.2</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4051">#4051</a></li> </ul> <h2>4.37.3 - 22 Jul 2026</h2> <p>No user facing changes.</p> <h2>4.37.2 - 21 Jul 2026</h2> <ul> <li>The new address format for the <code>config-file</code> input that was introduced in CodeQL Action 4.37.0 is now enabled by default. In addition to the format described there, the <code>remote=</code> prefix can now be used to explicitly indicate that the input refers to a remote file. All previous input formats continue to be accepted as well. <a href="https://redirect.github.com/github/codeql-action/pull/4023">#4023</a></li> <li>The CodeQL Action can now make use of <a href="https://docs.github.com/en/code-security/how-tos/secure-at-scale/configure-organization-security/manage-usage-and-access/giving-org-access-private-registries">configured private registries</a> in Default Setup to retrieve CodeQL configuration files from remote repositories that require authentication. This will allow customers to store their CodeQL configuration in a single repository that can then be referenced by Default Setup workflows in other repositories. We expect to roll this and other, related changes out to everyone in July. <a href="https://redirect.github.com/github/codeql-action/pull/4007">#4007</a></li> </ul> <h2>4.37.1 - 16 Jul 2026</h2> <ul> <li><em>Upcoming breaking change</em>: Add a deprecation warning for customers using CodeQL version 2.20.6 and earlier. These versions of CodeQL were discontinued on 1 July 2026 alongside GitHub Enterprise Server 3.16, and will be unsupported by the next minor release of the CodeQL Action. <a href="https://redirect.github.com/github/codeql-action/pull/3956">#3956</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.1">2.26.1</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4019">#4019</a></li> </ul> <h2>4.37.0 - 08 Jul 2026</h2> <ul> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.0">2.26.0</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3995">#3995</a></li> <li>In addition to the existing input format, the <code>config-file</code> input for the <code>codeql-action/init</code> step will soon support a new <code>[owner/]repo[@ref][:path]</code> format. All components except the repository name are optional. If omitted, <code>owner</code> defaults to the same owner as the repository the analysis is running for, <code>ref</code> to <code>main</code>, and <code>path</code> to <code>.github/codeql-action.yaml</code>. Support for this format ships in this version of the CodeQL Action, but will only be enabled over the coming weeks. <a href="https://redirect.github.com/github/codeql-action/pull/3973">#3973</a></li> </ul> <h2>4.36.3 - 01 Jul 2026</h2> <p>No user facing changes.</p> <h2>4.36.2 - 04 Jun 2026</h2> <ul> <li>Cache CodeQL CLI version information across Actions steps. <a href="https://redirect.github.com/github/codeql-action/pull/3943">#3943</a></li> <li>Reduce requests while waiting for analysis processing by using exponential backoff when polling SARIF processing status. <a href="https://redirect.github.com/github/codeql-action/pull/3937">#3937</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.6">2.25.6</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3948">#3948</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/github/codeql-action/commit/d1ba80a13dd99fba24a470575428917156a28b43"><code>d1ba80a</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4067">#4067</a> from github/update-v4.37.5-1cd4d01d5</li> <li><a href="https://github.com/github/codeql-action/commit/e74600b0d945db9734eb044f95cd43f34b773451"><code>e74600b</code></a> Update changelog for v4.37.5</li> <li><a href="https://github.com/github/codeql-action/commit/1cd4d01d58f636bbdbffbfd3c121b3446d07e9f3"><code>1cd4d01</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4061">#4061</a> from github/henrymercer/turbo-system</li> <li><a href="https://github.com/github/codeql-action/commit/d2bfc30bc3373a3ec945dc3f93bc8b8f07fa16b9"><code>d2bfc30</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4050">#4050</a> from github/mbg/status/registries</li> <li><a href="https://github.com/github/codeql-action/commit/68028fcb1605f3cdd37e4e1845c3f78af017a3ea"><code>68028fc</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4062">#4062</a> from github/sam-robson/migrate-enterprise-release-pat</li> <li><a href="https://github.com/github/codeql-action/commit/c29563eeaafbc75499c7bb0d74bf77b3506c1cbd"><code>c29563e</code></a> ci: use federated enterprise release PAT</li> <li><a href="https://github.com/github/codeql-action/commit/155e5229973b426bd1ae2f83bb1bf42417fa2a8f"><code>155e522</code></a> Link the PR from the changelog entry</li> <li><a href="https://github.com/github/codeql-action/commit/2d3b351ea6452a9b21346f8d64567e5b833924de"><code>2d3b351</code></a> Handle network errors when streaming the CodeQL bundle download</li> <li><a href="https://github.com/github/codeql-action/commit/5d3eb98e4a780bfe1a53f57fbdc278ea5da1274b"><code>5d3eb98</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4055">#4055</a> from github/dependabot/npm_and_yarn/npm-minor-203262...</li> <li><a href="https://github.com/github/codeql-action/commit/c5f739bd6460e096aaf40962517e46a846b8b6b1"><code>c5f739b</code></a> Merge branch 'main' into dependabot/npm_and_yarn/npm-minor-2032624187</li> <li>Additional commits viewable in <a href="https://github.com/github/codeql-action/compare/f205ea1c3313d32999d8d6a48b4f6530d4437b38...d1ba80a13dd99fba24a470575428917156a28b43">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
# Which issue does this PR close? - Part of #10540 # Rationale for this change I am trying to understand the depth of our round trip tests (there are over 4000 lines of tests in the arrow writer module). After some study it appears there are tests to round trip both single columns and record batches, which share some non trivial amount of logic. It also makes it hard to evaluate coverage because there are several similar but not quite the same free functions and it is unclear hwo they are related to each other and what some of the parameters mean (like the argument to `roundtrip_one_column`) I think making it easier to find and evaluate test coverage will make it easier to maintain and extend this crate in the future. # What changes are included in this PR? 1. Move round trip logic into methods on `RoundTripTest` rather than free functions # Are these changes tested? Only tests # Are there any user-facing changes? No --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
# Which issue does this PR close? - Closes #10381. # Rationale for this change `RowGroup.ordinal` is **optional** in the parquet-format Thrift spec with no uniformity requirement, but since #8715 the reader hard-errors on files whose row groups disagree on whether it is populated (`Inconsistent ordinal assignment: ...`). Such files are produced in the wild (e.g. Go parquet writers flushing row groups incrementally) and were readable before 57.1. This implements the plan agreed on the issue with @alamb, @etseidl and @vustef: - **All ordinals present** → honor them as written. - **None present** → sequential-fill at decode time, *unconditionally* — not just when row numbers are enabled — so downstream consumers behave identically whether the metadata was decoded fresh or reused from a prior read (the metadata-reuse hazard @vustef pointed out). - **Mixed** → leave the metadata untouched. Positional backfill could disagree with the ordinals that are present, and a partial backfill would make row-number results depend on which row groups a query happens to select. Instead, consumers that require complete ordinals fail deterministically; plain reads succeed. # What changes are included in this PR? - `thrift/mod.rs`: replace the per-row-group `OrdinalAssigner` (which errored on the first inconsistency) with a post-decode `ensure_row_group_ordinals` implementing the three cases above. - `array_reader/row_number.rs`: `RowNumberReader::try_new` rejects mixed-ordinal files up front — **even when every selected row group carries an ordinal** — so row numbering for a given file either always works or always fails, regardless of row-group pruning (@vustef's determinism concern). - `thrift/encryption.rs`: the encrypted column-metadata path used `rg.ordinal.unwrap()` for the AAD; with mixed metadata now reaching this code, return a proper error instead of panicking (@etseidl's encryption concern — files with fully-populated or fully-missing ordinals are unaffected). # Are these changes tested? - Decode round-trip tests for all three shapes (honored / sequential-filled / left untouched), including the exact Go-writer shape from the issue (first row group missing the ordinal). The mixed cases fail on main with the `Inconsistent ordinal assignment` error. - `RowNumberReader` unit tests: mixed metadata errors even for an ordinal-only selection; all-missing metadata errors for any selection. - End-to-end test: a real 4-row-group file with one ordinal stripped reads fine without virtual columns and fails deterministically with the row-number column, even when selecting only row groups that carry ordinals. - Full `parquet` test suite passes with `--all-features`. # Are there any user-facing changes? Files with mixed or absent row-group ordinal metadata are readable again (as before 57.1). Row-number virtual columns keep their strict guarantee: they now fail deterministically per file instead of depending on row-group selection. No API changes.
) The AsRef impl was limited to references to Buffer instead of Buffer itself which is unnecessary since the signature of as_ref already provides the necessary indirection and it also is not the implementation required for wider ecosystem integration. # Which issue does this PR close? - Closes #10570. # Rationale for this change I am trying to use the newly available `StreamEncoder` from the `arrow_ipc` crate together with `Body::from_stream` from `axum`, but calling `bytes::Bytes::from_owner` fails due to an unsatisfied trait bound which I have to work around by local newtype wrapping. # What changes are included in this PR? A fix for the overly restrictive `AsRef` impl and also writing `Bytes::from_owner` through as generic and canonical way to wrap a given `Buffer` as a `bytes::Bytes` object (so it can be used with `Body::from_stream` directly. # Are these changes tested? The more general impl is covered by the same tests as the existing one and the new `From` impl is exercised by the doctest. # Are there any user-facing changes? ~~This is strictly speaking a breaking change via downstream trait coherence, but I would argue that the existing `AsRef` is erroneously restricted and would therefore classify this as a typing bug fix.~~ This is not breaking as [the standard library's blanket impl](https://doc.rust-lang.org/std/convert/trait.AsRef.html#impl-AsRef%3CU%3E-for-%26T) will continue to provide the previous impl so this appears strictly additive with `cargo semver-checks` v0.50.0 agreeing. Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes #10562 # Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Keeping dependencies up to date. # What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Bump `rand` to `0.10` in the arrow crate. Since we expose functions that rely on the rand API, this is a breaking change. Also because `half` doesn't support `rand` `0.10` (see VoidStarKat/half-rs#144) we have to remove dependency on its `rand_distr` feature and handroll it ourselves. # Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> Existing tests # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. --> Yes, API change. `create_primitive_array()` from `bench_util` no longer supports `Float16Array`s; instead you must use the new `create_nullable_f16_array()` function.
# Which issue does this PR close? No issue in particular - Follow-up to #10533 - Follow-up to #10552 - Related to #10458 # Rationale for this change There are a lot of lints we can enable to, which different abilities to: * Simplify the code * Write more efficient code * Catch bugs * etc ## What changes are included in this PR? One commit per new lint (maybe easiest to review commit by commit!). Let me know if you disagree with any of them. I hope to add even more lints in later PRs. ## Are these changes tested? Covered by existing tests plus the clippy CI job. ## Are there any user-facing changes? No. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
# Which issue does this PR close? - Closes #10521. # Rationale for this change `MessageReader::maybe_next` reserves both the metadata length and the message body length before reading any of the bytes they describe. Both come out of the stream itself, so a corrupted or truncated stream is handed straight to the allocator: `MutableBuffer::from_len_zeroed(message.bodyLength() as usize)` on an implausible length either aborts the process (`memory allocation of N bytes failed`, which is not catchable and takes the host process with it) or panics on `LayoutError`. A negative `bodyLength` is accepted too, since `as usize` wraps it to a large positive length. I hit this fuzzing real IPC blocks rather than crafted ones: single-bit flips over the framing region of genuine streams produced `memory allocation of 1125899907497992 bytes failed` and `SIGABRT`. Where those blocks cross disk or a network, one flipped bit ends the process instead of failing a read the caller could retry. # What changes are included in this PR? `bodyLength` now goes through `usize::try_from`, so a negative length is a parse error rather than a huge positive one. Neither length reserves more than `MAX_PREALLOC_BYTES` (64 MiB) before the bytes behind it have arrived. Bodies up to that size are allocated in one go exactly as before; larger ones grow as the data arrives, which costs the reallocations that `MutableBuffer::reserve` doubling implies. That constant is the one judgement call here, trading the size of the bounded allocation a malformed stream can still ask for against how large a body keeps the single-allocation path, so it is worth a second opinion. The metadata read switches from `resize(meta_len, 0)` plus `read_exact` to `take(meta_len).read_to_end(&mut self.buf)`. That reuses the retained capacity across messages and drops the zeroing entirely, so it should be slightly cheaper than what it replaces rather than a cost, and `Take` returns `Ok(0)` at its limit so there is no extra read. Only the streaming path is touched. `read_block` on the file side has the same shape at `arrow-ipc/src/reader.rs:875` and two `unwrap()`s on block metadata besides; I left it alone to keep this reviewable, and noted it in the issue. This overlaps #9777, which is after the same zeroing for performance reasons. The two want the same thing here, and I am happy to rebase onto whatever lands first. # Are these changes tested? Yes, two tests in `arrow-ipc/src/reader.rs`. `test_stream_reader_rejects_implausible_body_length` covers `i64::MAX`, `1 << 50` and `-1`; `test_stream_reader_rejects_unbacked_metadata_length` covers a metadata length of `i32::MAX` with nineteen bytes behind it. Both fail without the change: the first panics inside `MutableBuffer::from_len_zeroed`, and the second spends 7.7s zeroing 2 GiB before reporting the wrong error. The existing `arrow-ipc` suite passes (139 tests), along with `cargo fmt --all --check` and `cargo clippy -p arrow-ipc --all-targets --all-features -- -D warnings`. # Are there any user-facing changes? No API changes. A stream that previously aborted or panicked now returns an `ArrowError`. No breaking changes. --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
…rder` (#10573) # Which issue does this PR close? - Closes #10548. # Rationale for this change See issue # What changes are included in this PR? Document the relationship between the `ColumnOrder` and `SortOrder` enums used in the parquet crate. # Are these changes tested? N/A documentation only # Are there any user-facing changes? No
# Which issue does this PR close? - part of #10540 # Rationale for this change The arrow_reader tests are over 4000 lines in one module. I would like to try and break it up a bit so the tests are easier to understand and navigate I think tests that read example fixtures from parquet_testing is a natural thing to consolidate (so among other things we can be sure we are testing all the parquet_testing files) # What changes are included in this PR? 1. Add parquet_testing.rs to the `arrow_reader` test 2. Move some tests # Are these changes tested? Only tests # Are there any user-facing changes? No
since we're using git cliff now, cleaning up this unused file reference: - #10451
…10574) # Which issue does this PR close? Split out of #10446 so that CI can compare both sides — the benchmark is new there, so the merge-base has nothing to compare against and the `main` column comes out empty. # Rationale for this change `row_selector.rs` already benchmarks `intersection`/`union`, but it builds the operands with `from_filters`, which is selector-backed. Those take the `RowSelector` merge path and never reach the bitwise one used when both operands are mask-backed. # What changes are included in this PR? Adds `mask_intersection`/`mask_union`, varying the two dimensions that drive the bitwise path: - the ratio between operand lengths, since unequal lengths pass the longer side's tail through unchanged - the bit offsets the operands carry, since masks come from `BooleanBuffer::slice` and whether the two share a sub-64-bit alignment decides which path the underlying helpers take Benchmark only, no library changes. # Are these changes tested? N/A — this is a benchmark. It builds and runs on `main` as-is. # Are there any user-facing changes? No.
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes #10145. # What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Seperate the logic for `variant_shred` and `variant_get` by introduce `shred` flag for row builder - Move some logic in `Variant::as_xxx` to `type_conversion`, and these moved code will be used when `variant_get`, `variant_shred` will use `Variant::as_xxx` - After the change, when shredding a variant, all `Variant::as_xx()` is identity function now, `Variant::Int8` can only be treated as `int8`, but not `int16`/`int32`/`int64`, etc. - Removed the `Variant::as_f16` - Add a test to cover that shred can/can't be shredded to some datatype in `test_variant_type_shredded_correctly` # Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> Yes, added some tests to cover the logic # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. --> Yes, some `Variant::as_xx` logic have been changed. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Kosta Tarasov <33369833+sdf-jkl@users.noreply.github.com>
# Which issue does this PR close? - Related to #10157 # Rationale for this change It seems #10157 has a logical conflict with newly added clippy lints After merging #10157 it is failing with For example: https://github.com/apache/arrow-rs/actions/runs/31189404230/job/92902082102 ``` warning: unnested or-patterns --> parquet-variant-compute/src/shred_variant.rs:3134:17 | 3134 | / (Variant::Int8(_), DataType::Int8) 3135 | | | (Variant::Int8(_), DataType::Int16) 3136 | | | (Variant::Int8(_), DataType::Int32) 3137 | | | (Variant::Int8(_), DataType::Int64) ... | 3228 | | | (Variant::String(_), DataType::LargeUtf8) 3229 | | | (Variant::Time(_), DataType::Time64(_)) | |_____________________________________________________________^ | = help: for further information visit https://rust-lang.github.io/rust-clippy/rust-1.97.0/index.html#unnested_or_patterns = note: requested on the command line with `-W clippy::unnested-or-patterns` help: nest the patterns | 3134 ~ (Variant::Int8(_) | Variant::Int16(_) | Variant::Int32(_) | Variant::Int64(_) 3135 + | Variant::Decimal4(_) | Variant::Decimal8(_) | Variant::Decimal16(_), 3136 + DataType::Int8) | 3137 + (Variant::Int8(_) | Variant::Int16(_) | Variant::Int32(_) | Variant::Int64(_) 3138 + | Variant::Decimal4(_) | Variant::Decimal8(_) | Variant::Decimal16(_), 3139 + DataType::Int16) | ``` # What changes are included in this PR? Do what clippy says (just accept its suggestion) # Are these changes tested? By CI # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -->
Bumps the codeql-actions group with 2 updates: [github/codeql-action/init](https://github.com/github/codeql-action) and [github/codeql-action/analyze](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.37.5 to 4.37.6 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/releases">github/codeql-action/init's releases</a>.</em></p> <blockquote> <h2>v4.37.6</h2> <ul> <li>Changed the default filepath for the new remote file address format that was introduced in CodeQL Action 4.37.0 / 3.37.0 to <code>.github/codeql-config.yml</code> to align it with the suggested path that is used elsewhere. <a href="https://redirect.github.com/github/codeql-action/pull/4070">#4070</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action/init's changelog</a>.</em></p> <blockquote> <h1>CodeQL Action Changelog</h1> <p>See the <a href="https://github.com/github/codeql-action/releases">releases page</a> for the relevant changes to the CodeQL CLI and language packs.</p> <h2>[UNRELEASED]</h2> <p>No user facing changes.</p> <h2>4.37.6 - 04 Aug 2026</h2> <ul> <li>Changed the default filepath for the new remote file address format that was introduced in CodeQL Action 4.37.0 / 3.37.0 to <code>.github/codeql-config.yml</code> to align it with the suggested path that is used elsewhere. <a href="https://redirect.github.com/github/codeql-action/pull/4070">#4070</a></li> </ul> <h2>4.37.5 - 03 Aug 2026</h2> <ul> <li>Fixed a bug where a network error while streaming the download of the CodeQL bundle could terminate the <code>init</code> Action instead of falling back to downloading the bundle before extracting it. <a href="https://redirect.github.com/github/codeql-action/pull/4061">#4061</a></li> </ul> <h2>4.37.4 - 29 Jul 2026</h2> <ul> <li>This version of the CodeQL Action adds support for the <code>tools</code> input for the <code>codeql-action/init</code> step to be specified using a <code>github-codeql-tools</code> <a href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">repository property</a>. This feature will gradually be rolled out following the release of this version. Once rolled out, this allows for the CodeQL CLI version that is used in GitHub-managed workflows, such as Default Setup, to be set to a custom value. For example, customers who run into issues with rate limits when a new CodeQL CLI version is released can set the value to <code>toolcache</code> to always use the CodeQL CLI version that is available in the runner toolcache. For Advanced Setup workflows, the value provided for <code>tools</code> in the workflow definition always takes precedence unless the value of the repository property starts with <code>!</code>. <a href="https://redirect.github.com/github/codeql-action/pull/4037">#4037</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.2">2.26.2</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4051">#4051</a></li> </ul> <h2>4.37.3 - 22 Jul 2026</h2> <p>No user facing changes.</p> <h2>4.37.2 - 21 Jul 2026</h2> <ul> <li>The new address format for the <code>config-file</code> input that was introduced in CodeQL Action 4.37.0 is now enabled by default. In addition to the format described there, the <code>remote=</code> prefix can now be used to explicitly indicate that the input refers to a remote file. All previous input formats continue to be accepted as well. <a href="https://redirect.github.com/github/codeql-action/pull/4023">#4023</a></li> <li>The CodeQL Action can now make use of <a href="https://docs.github.com/en/code-security/how-tos/secure-at-scale/configure-organization-security/manage-usage-and-access/giving-org-access-private-registries">configured private registries</a> in Default Setup to retrieve CodeQL configuration files from remote repositories that require authentication. This will allow customers to store their CodeQL configuration in a single repository that can then be referenced by Default Setup workflows in other repositories. We expect to roll this and other, related changes out to everyone in July. <a href="https://redirect.github.com/github/codeql-action/pull/4007">#4007</a></li> </ul> <h2>4.37.1 - 16 Jul 2026</h2> <ul> <li><em>Upcoming breaking change</em>: Add a deprecation warning for customers using CodeQL version 2.20.6 and earlier. These versions of CodeQL were discontinued on 1 July 2026 alongside GitHub Enterprise Server 3.16, and will be unsupported by the next minor release of the CodeQL Action. <a href="https://redirect.github.com/github/codeql-action/pull/3956">#3956</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.1">2.26.1</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4019">#4019</a></li> </ul> <h2>4.37.0 - 08 Jul 2026</h2> <ul> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.0">2.26.0</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3995">#3995</a></li> <li>In addition to the existing input format, the <code>config-file</code> input for the <code>codeql-action/init</code> step will soon support a new <code>[owner/]repo[@ref][:path]</code> format. All components except the repository name are optional. If omitted, <code>owner</code> defaults to the same owner as the repository the analysis is running for, <code>ref</code> to <code>main</code>, and <code>path</code> to <code>.github/codeql-action.yaml</code>. Support for this format ships in this version of the CodeQL Action, but will only be enabled over the coming weeks. <a href="https://redirect.github.com/github/codeql-action/pull/3973">#3973</a></li> </ul> <h2>4.36.3 - 01 Jul 2026</h2> <p>No user facing changes.</p> <h2>4.36.2 - 04 Jun 2026</h2> <ul> <li>Cache CodeQL CLI version information across Actions steps. <a href="https://redirect.github.com/github/codeql-action/pull/3943">#3943</a></li> <li>Reduce requests while waiting for analysis processing by using exponential backoff when polling SARIF processing status. <a href="https://redirect.github.com/github/codeql-action/pull/3937">#3937</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.6">2.25.6</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3948">#3948</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/github/codeql-action/commit/5595ccaf912efad79be6eef63a5619ff05969be3"><code>5595cca</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4071">#4071</a> from github/update-v4.37.6-6a9359a1b</li> <li><a href="https://github.com/github/codeql-action/commit/ec9c75796a7f2cee5af0c5ffa0b81dc3bb58754b"><code>ec9c757</code></a> Add change note for PR 4070</li> <li><a href="https://github.com/github/codeql-action/commit/45c8742e17cbd668814137f95e605d925b8722a2"><code>45c8742</code></a> Update changelog for v4.37.6</li> <li><a href="https://github.com/github/codeql-action/commit/6a9359a1bd054c53cae7bb737bd8d796cfbf3014"><code>6a9359a</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4070">#4070</a> from github/mbg/remote-address/change-file-default</li> <li><a href="https://github.com/github/codeql-action/commit/065cdc0394d424981db720df63ebc570e41b775f"><code>065cdc0</code></a> Change <code>DEFAULT_CONFIG_FILE_NAME</code></li> <li><a href="https://github.com/github/codeql-action/commit/f99dd5aeee9cf92e92d0c700cb0aa7afd7bbf431"><code>f99dd5a</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4066">#4066</a> from github/dependabot/npm_and_yarn/js-yaml-5.2.2</li> <li><a href="https://github.com/github/codeql-action/commit/1804b211a343d69a6584d26fb3a68a8fe6ca39d4"><code>1804b21</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4068">#4068</a> from github/mergeback/v4.37.5-to-main-d1ba80a1</li> <li><a href="https://github.com/github/codeql-action/commit/3020a2f46286abb1704269b22ada83bd0e81c64f"><code>3020a2f</code></a> Rebuild</li> <li><a href="https://github.com/github/codeql-action/commit/93c3a5a40b7affbf8ea6a480767ed0db8e8d3c5c"><code>93c3a5a</code></a> Update changelog and version after v4.37.5</li> <li><a href="https://github.com/github/codeql-action/commit/266c7bdbd2ad8151d42fd682e28c126c5da068da"><code>266c7bd</code></a> Rebuild</li> <li>Additional commits viewable in <a href="https://github.com/github/codeql-action/compare/d1ba80a13dd99fba24a470575428917156a28b43...5595ccaf912efad79be6eef63a5619ff05969be3">compare view</a></li> </ul> </details> <br /> Updates `github/codeql-action/analyze` from 4.37.5 to 4.37.6 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/releases">github/codeql-action/analyze's releases</a>.</em></p> <blockquote> <h2>v4.37.6</h2> <ul> <li>Changed the default filepath for the new remote file address format that was introduced in CodeQL Action 4.37.0 / 3.37.0 to <code>.github/codeql-config.yml</code> to align it with the suggested path that is used elsewhere. <a href="https://redirect.github.com/github/codeql-action/pull/4070">#4070</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action/analyze's changelog</a>.</em></p> <blockquote> <h1>CodeQL Action Changelog</h1> <p>See the <a href="https://github.com/github/codeql-action/releases">releases page</a> for the relevant changes to the CodeQL CLI and language packs.</p> <h2>[UNRELEASED]</h2> <p>No user facing changes.</p> <h2>4.37.6 - 04 Aug 2026</h2> <ul> <li>Changed the default filepath for the new remote file address format that was introduced in CodeQL Action 4.37.0 / 3.37.0 to <code>.github/codeql-config.yml</code> to align it with the suggested path that is used elsewhere. <a href="https://redirect.github.com/github/codeql-action/pull/4070">#4070</a></li> </ul> <h2>4.37.5 - 03 Aug 2026</h2> <ul> <li>Fixed a bug where a network error while streaming the download of the CodeQL bundle could terminate the <code>init</code> Action instead of falling back to downloading the bundle before extracting it. <a href="https://redirect.github.com/github/codeql-action/pull/4061">#4061</a></li> </ul> <h2>4.37.4 - 29 Jul 2026</h2> <ul> <li>This version of the CodeQL Action adds support for the <code>tools</code> input for the <code>codeql-action/init</code> step to be specified using a <code>github-codeql-tools</code> <a href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">repository property</a>. This feature will gradually be rolled out following the release of this version. Once rolled out, this allows for the CodeQL CLI version that is used in GitHub-managed workflows, such as Default Setup, to be set to a custom value. For example, customers who run into issues with rate limits when a new CodeQL CLI version is released can set the value to <code>toolcache</code> to always use the CodeQL CLI version that is available in the runner toolcache. For Advanced Setup workflows, the value provided for <code>tools</code> in the workflow definition always takes precedence unless the value of the repository property starts with <code>!</code>. <a href="https://redirect.github.com/github/codeql-action/pull/4037">#4037</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.2">2.26.2</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4051">#4051</a></li> </ul> <h2>4.37.3 - 22 Jul 2026</h2> <p>No user facing changes.</p> <h2>4.37.2 - 21 Jul 2026</h2> <ul> <li>The new address format for the <code>config-file</code> input that was introduced in CodeQL Action 4.37.0 is now enabled by default. In addition to the format described there, the <code>remote=</code> prefix can now be used to explicitly indicate that the input refers to a remote file. All previous input formats continue to be accepted as well. <a href="https://redirect.github.com/github/codeql-action/pull/4023">#4023</a></li> <li>The CodeQL Action can now make use of <a href="https://docs.github.com/en/code-security/how-tos/secure-at-scale/configure-organization-security/manage-usage-and-access/giving-org-access-private-registries">configured private registries</a> in Default Setup to retrieve CodeQL configuration files from remote repositories that require authentication. This will allow customers to store their CodeQL configuration in a single repository that can then be referenced by Default Setup workflows in other repositories. We expect to roll this and other, related changes out to everyone in July. <a href="https://redirect.github.com/github/codeql-action/pull/4007">#4007</a></li> </ul> <h2>4.37.1 - 16 Jul 2026</h2> <ul> <li><em>Upcoming breaking change</em>: Add a deprecation warning for customers using CodeQL version 2.20.6 and earlier. These versions of CodeQL were discontinued on 1 July 2026 alongside GitHub Enterprise Server 3.16, and will be unsupported by the next minor release of the CodeQL Action. <a href="https://redirect.github.com/github/codeql-action/pull/3956">#3956</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.1">2.26.1</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4019">#4019</a></li> </ul> <h2>4.37.0 - 08 Jul 2026</h2> <ul> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.0">2.26.0</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3995">#3995</a></li> <li>In addition to the existing input format, the <code>config-file</code> input for the <code>codeql-action/init</code> step will soon support a new <code>[owner/]repo[@ref][:path]</code> format. All components except the repository name are optional. If omitted, <code>owner</code> defaults to the same owner as the repository the analysis is running for, <code>ref</code> to <code>main</code>, and <code>path</code> to <code>.github/codeql-action.yaml</code>. Support for this format ships in this version of the CodeQL Action, but will only be enabled over the coming weeks. <a href="https://redirect.github.com/github/codeql-action/pull/3973">#3973</a></li> </ul> <h2>4.36.3 - 01 Jul 2026</h2> <p>No user facing changes.</p> <h2>4.36.2 - 04 Jun 2026</h2> <ul> <li>Cache CodeQL CLI version information across Actions steps. <a href="https://redirect.github.com/github/codeql-action/pull/3943">#3943</a></li> <li>Reduce requests while waiting for analysis processing by using exponential backoff when polling SARIF processing status. <a href="https://redirect.github.com/github/codeql-action/pull/3937">#3937</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.6">2.25.6</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3948">#3948</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/github/codeql-action/commit/5595ccaf912efad79be6eef63a5619ff05969be3"><code>5595cca</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4071">#4071</a> from github/update-v4.37.6-6a9359a1b</li> <li><a href="https://github.com/github/codeql-action/commit/ec9c75796a7f2cee5af0c5ffa0b81dc3bb58754b"><code>ec9c757</code></a> Add change note for PR 4070</li> <li><a href="https://github.com/github/codeql-action/commit/45c8742e17cbd668814137f95e605d925b8722a2"><code>45c8742</code></a> Update changelog for v4.37.6</li> <li><a href="https://github.com/github/codeql-action/commit/6a9359a1bd054c53cae7bb737bd8d796cfbf3014"><code>6a9359a</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4070">#4070</a> from github/mbg/remote-address/change-file-default</li> <li><a href="https://github.com/github/codeql-action/commit/065cdc0394d424981db720df63ebc570e41b775f"><code>065cdc0</code></a> Change <code>DEFAULT_CONFIG_FILE_NAME</code></li> <li><a href="https://github.com/github/codeql-action/commit/f99dd5aeee9cf92e92d0c700cb0aa7afd7bbf431"><code>f99dd5a</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4066">#4066</a> from github/dependabot/npm_and_yarn/js-yaml-5.2.2</li> <li><a href="https://github.com/github/codeql-action/commit/1804b211a343d69a6584d26fb3a68a8fe6ca39d4"><code>1804b21</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4068">#4068</a> from github/mergeback/v4.37.5-to-main-d1ba80a1</li> <li><a href="https://github.com/github/codeql-action/commit/3020a2f46286abb1704269b22ada83bd0e81c64f"><code>3020a2f</code></a> Rebuild</li> <li><a href="https://github.com/github/codeql-action/commit/93c3a5a40b7affbf8ea6a480767ed0db8e8d3c5c"><code>93c3a5a</code></a> Update changelog and version after v4.37.5</li> <li><a href="https://github.com/github/codeql-action/commit/266c7bdbd2ad8151d42fd682e28c126c5da068da"><code>266c7bd</code></a> Rebuild</li> <li>Additional commits viewable in <a href="https://github.com/github/codeql-action/compare/d1ba80a13dd99fba24a470575428917156a28b43...5595ccaf912efad79be6eef63a5619ff05969be3">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…eld as nullable (spec requires `required`) (#10320) # Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes #10315. - let's wait till #10318 ships # Rationale for this change - Parquet spec required unshredded `VariantArray` to be non-nullable. Other readers might reject our impl because `unshred_variant` and `variant_get as_type None` return nullable unshredded Variant. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? - builds on #10318 - Add changes to `unshred_variant` and `variant_get` to return non-nullable unshredded Variant <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? - yes, added new unit tests and fixed some old ones <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? - Yes, the nullability now conforms the spec <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. --> --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
Closes #10583. Replace source builds of `cargo-audit` and `cargo-msrv` in CI with prebuilt binary installation via `taiki-e/install-action`. This keeps the audit and MSRV verification commands unchanged while avoiding compilation from source during CI setup. ## Testing - `actionlint .github/workflows/audit.yml .github/workflows/rust.yml` - YAML parsing - `git diff --check` Co-authored-by: yang3.xie <yang3.xie@ly.com> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes #NNN. # Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -->
# Which issue does this PR close? Part of #10458. # Rationale for this change Enable clippy::allow_attributes for arrow-arith so future lint suppressions use expectations and stale suppressions are detectable. The existing allow(deprecated) on test_unary_f64_slice no longer suppresses a warning, so it can be removed. # What changes are included in this PR? - Deny clippy::allow_attributes in the arrow-arith crate. - Remove the stale deprecated lint suppression from test_unary_f64_slice. # Are these changes tested? - cargo fmt --all -- --check - cargo clippy -p arrow-arith --all-targets --all-features --no-deps -- -D warnings - cargo test -p arrow-arith --all-features (231 unit tests and 16 doc tests passed) - cargo check -p arrow-arith --all-targets --all-features # Are there any user-facing changes? No. # AI assistance Codex was used to identify the lint occurrence, draft the minimal edit, and run the validation commands. I reviewed the full diff and verified the change with formatting, Clippy, tests, and compilation checks. Co-authored-by: ZhangStudyLife <174326754+ZhangStudyLife@users.noreply.github.com>
before #10507 (running miri on more crates), miri took about 15-20 minutes on average. with the new PR it increased to a bit over 30 minutes because we're running more tests. increase the partitions count to try get average runtime down, also use taiki-e to install nextest via binary to shave off 3 minutes spent on compiling nextest from source. also include arrow-cmp (new crate) in the miri execution
…yData` roundtrip (#10807) # Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Part of #9298. # Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Eliminate the `make_array` roundtrip in `RunArray::try_new` to improve performance # Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> Correctness is ensured by existing tests Bench is extracted #10809. Local benchmark gave the results below: | Bench | Before | After | Change | |---|---|---|---| | i32, 256 | 497 ns | 222 ns | −59% | | utf8, 256 | 585 ns | 246 ns | −56% | | i32, 1024 | 999 ns | 664 ns | −37% | | utf8, 1024 | 988 ns | 518 ns | −48% | | i32, 4096 | 1.98 µs | 1.80 µs | −9% | | utf8, 4096 | 2.01 µs | 1.79 µs | −11% | | i32, 8192 | 3.64 µs | 3.50 µs | −4% | | utf8, 8192 | 3.66 µs | 3.49 µs | −5% | # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. --> No --------- Co-authored-by: Jefffrey <jeffrey.vo.australia@gmail.com>
…idate (#10798) # Which issue does this PR close? - Closes #10034. # Rationale for this change `from_ffi` realigned under-aligned C Data Interface buffers (e.g. an 8-byte aligned `Decimal128` from a JVM producer) *after* `consume()`. under `force_validate`, `consume()`'s `build()` validates first and rejects the buffer before the realign runs, so spec-legal input errors. reachable via the `arrow` crate with `features = ["force_validate", "ffi"]` calling `arrow::ffi::from_ffi`. # What changes are included in this PR? - `ImportedArrowArray::consume` builds through `ArrayDataBuilder` with `align_buffers(true)` before validation, matching `arrow-ipc`'s `create_array_from_builder` - dropped the now-redundant outer `align_buffers()` calls in `from_ffi` / `from_ffi_and_data_type`. # Are these changes tested? covered by `test_decimal128_under_aligned_round_trip`. the issue suggested ungating it under `force_validate`, but that isn't possible as its fixture is a misaligned `ArrayData` built with `build_unchecked`, which validates under `force_validate` and so rejects the input at construction, before `from_ffi` runs. the gate stays with a comment explaining why # Are there any user-facing changes? no public API change. behavior only changes under `force_validate`, where spec-legal under-aligned input is realigned instead of erroring.
…ate loops (#10974) # Which issue does this PR close? - Closes #10961. # Rationale for this change We recently consolidated the decimal parsing code to use a single, more correct implementation (#10850); however, this resulted in regression performance somewhat. `parse_decimal` used a single loop to scan each digit of the input. Digits before and after the decimal point are treated very differently; profiling the code revealed that doing a data-dependent branch for each digit resulted in worse codegen. Instead, we can split the parsing logic to use two loops, in sequence: first look for digits that precede the decimal point, then those that follow. This avoids the data-dependent branch and seems to significantly improve codegen. On M4 Max, this improves the end-to-end CSV parsing benchmark by 5-15%, and the `parse_decimal` microbenchmark by 5-33%. # What changes are included in this PR? * Split decimal parsing logic into two sequential loops * Refactor digit accumulation logic to avoid duplication * Mark `fold_decimal_chunk` as `inline(always)` -- the refactor resulted in LLVM deciding not to inline this call inside the hot loop, which regressed performance significantly. # Are these changes tested? Yes, covered by existing tests. # Are there any user-facing changes? No. # Tool usage Explored optimization ideas and developed this optimization with Claude Code (Fable 5.1). I revised and understand the resulting code.
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes #10883. # Rationale for this change see #10883. `NullBuffer::expand` repeats each validity bit count times, used whenever a parent null must be propagated to a fixed-size group of child elements. The previous implementation set output bits one at a time regardless of count, which is unnecessarily slow for common sizes. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? Three tiered fast paths are added to try_expand, checked in order: - count % 8 == 0 : each expanded run is byte-aligned, so contiguous runs of valid bits are filled with memset(0xFF) via BitSliceIterator rather than setting bits individually. - count % 4 == 0 : each bit's range starts and ends on a nibble boundary. Full interior bytes are filled with 0xFF; the one partial boundary byte is set with |= 0x0F or |= 0xF0 depending on alignment. No inner loop. - General case : unchanged bit-by-bit path. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? yes, existing test cover this as well as 1 new test <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? no <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -->
… i32 (#10981) # Which issue does this PR close? - Closes #10972. # Rationale for this change `concat_elements_fixed_size_binary` adds the two input widths together as a usize, then casts the sum to i32 to size the builder. two arrays of width 0x70000000 come out at 3758096384, which wraps to -536870912, and the builder asserts the value length is not negative. that width is accepted at array construction and can come from a user controlled schema, so nothing unusual on the caller's side is needed to reach it it comes through `concat_elements_dyn` as well, since that dispatches here for fixed size binary inputs # What changes are included in this PR? use i32::try_from on the combined width and return an invalid argument error when it does not fit, rather than casting the byte view builder in this same file already guards exactly this, checking data_size against i32::MAX before it builds, so this is that guard applied to the fixed size binary path instead of a new mechanism. widths that already fit behave the same as before i went through the rest of the file for the same shape while i was in there. that cast was the only unchecked one, so this is a single site rather than a family the way #10437 and #10575 were one thing i left out on purpose. the next line still reserves the combined width through `MutableBuffer::with_capacity`, so a sum just under i32::MAX asks for roughly 2 GB before a single row is written. that looked like #10973 rather than this one, but say the word and i will fold it in # Are these changes tested? yes. `test_fixed_size_binary_concat_width_overflow` uses the widths from the issue and checks the call comes back as an error instead of panicking. with only the test applied to current main it fails inside `fixed_size_binary_builder.rs` at line 64, which is the panic site in the report. fmt and clippy with -D warnings are both clean on arrow-string # Are there any user-facing changes? concatenating two fixed size binary arrays whose widths sum past i32::MAX returns an error now instead of panicking. no API changes
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Part of #10541 - Closes #8748 . - Related spec PR: apache/parquet-format#557 # Rationale for this change check issue <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ounts (#10554) - closes #10538 > [!IMPORTANT] > **Stacked on #10505 and #10745 — do not merge first.** The commits below the top one belong to those PRs. > Review only this PR's own commit: [`cff2ca07fe...bd66b9e`](pydantic/arrow-rs@cff2ca0...bd66b9e). > Once #10505 and #10745 merge I'll rebase and this PR's diff becomes clean on its own. ## The problem `byte_budget_sub_batch_size` asks the encoder how many *values* fit in a page byte budget, then converts that to a *level* count using the chunk-wide level:value ratio, rounded up: ```rust (values_per_subbatch * chunk_size).div_ceil(vals_in_chunk).max(1) ``` For a chunk with no nulls this is exact. With one null in 17 levels it gives `ceil(17/16) == 2`, and `write_granular_chunk` slices the chunk into uniform two-level windows — most of which carry **two** values, i.e. twice what the budget allowed. The mechanism predates #10505; it dates to #9972. Where the encoding compresses a value against its predecessor, that round-up costs whole values of output. 128 values of 2 MiB at one null in 16: | | file size | | --- | --- | | ratio-scaled windows | 16.78 MB | | value-exact windows | **2.10 MB** | At 8 MiB values it is 64 MiB against 8 MiB, and the acceptance case in #10538 — 16 identical 64 KiB values with one null — goes from five pages storing ~5 values in full to one page storing ~1. ## The fix Have the chunker return the value count it already computed, and let `write_granular_chunk` end a window by walking definition levels until it has covered that many values. No ratio, no rounding. Then apply it only where it changes the bytes written, because it is not free — value-exact windows roughly double the mini-batch count on a nullable column. Two conditions, both required: **The budget must be the data page budget.** That one is a constant `data_page_size_limit`, so a one-value budget means the value itself overflows a page. The dictionary page budget is the limit *minus what the dictionary already holds*, so it shrinks toward zero as the dictionary fills and reaches a one-value budget on perfectly ordinary values; cutting exactly there measured +13.0% on `string/default` and +8.3% on `string/parquet_2`. **The encoding must compress against the previous value.** `PLAIN` and `DELTA_LENGTH_BYTE_ARRAY` store a value identically wherever it lands, so value-exact windows leave their output byte for byte the same while doubling the page count — measured +27.6% on a nullable column for no reduction in output at all. The `compresses_against_previous_value` flag #10505 added already marks exactly the right set. ## What the other paths give up A ratio-scaled window spans `ceil(values × levels / values_in_chunk)` levels. Where one value already fills the budget that covers **at most two values**, whatever the null density, and exactly one wherever the ratio is a whole number. Measured against a 1 MiB limit: | values | nulls | encoding | max page, ratio | max page, value-exact | floor | | --- | --- | --- | --- | --- | --- | | 2 MiB | 1-in-16 | `PLAIN` | 4.00× | 2.00× | 2.00× | | 2 MiB | 1-in-4 | `PLAIN` | 4.00× | 2.00× | 2.00× | | 2 MiB | 1-in-2 | `PLAIN` | 2.00× | 2.00× | 2.00× | | 8 MiB | 1-in-16 | `PLAIN` | 16.00× | 8.00× | 8.00× | The floor is what a page must hold: one value. So the concession is a factor of two above an unavoidable minimum, it does not vary with null density, and — the property #9972 exists for — it does not scale with `write_batch_size`. Before #9972 a page took a whole mini-batch: 1024 × 2 MiB, roughly 2000× the limit. ## Measurements Base is #10505's head (measured at `fd806be5d3`; both branches have since been rebased onto `main`, with the trees verified identical across the rebase), so these isolate this PR. Local, run base → branch → base on an idle machine, with the two base passes as a per-benchmark noise floor. Benchmarks are the ones added in #10561. | benchmark | before the encoding gate | after | noise | | --- | --- | --- | --- | | `..._nullable/plain` | +27.6% | **−0.8%** | 0.4% | | `..._nullable_trailing/delta_byte_array` | −27.8% | **−27.5%** | 0.8% | | `..._nullable/delta_byte_array` | +7.6% | +7.4% | 0.0% | | `..._nullable_dense/delta_byte_array` | −0.2% | +0.6% | 1.0% | | `large_string_distinct_nullable/delta_byte_array` | +3.1% | +11.3% | 2.0% | | `medium_string_shared_prefix_nullable/delta_byte_array` | +1.3% | +1.7% | 1.2% | Two costs remain, both on `DELTA_BYTE_ARRAY` where the byte win does not materialise: - **+7.4%** where output was already close to deduplicated (16.78 MB → 2.10 MB is still a 8× reduction, so this one pays for itself). - **+11.3%** where the values share no prefix at all, and the file is 251.670105 MB against 251.670637 MB — no reduction. The writer cannot know in advance whether values will share prefixes, so this is the premium for the 8× win when they do. Verified byte-identical output — same length, same hash — between base and this PR for dictionary-encoded nullable columns (four shapes) and for repeated columns (both encodings), confirming those paths are untouched rather than merely unchanged in aggregate. ## Scope Repeated columns are unchanged: records cannot span pages, so a record holding several over-limit values still exceeds the budget. That is inherent to the format. ## Tests - `test_column_writer_delta_byte_array_nullable_shared_prefix_dedup` — re-pinned from `[2, 2, 2, 2, 9]` to `[17]` and renamed, the layout #10505 left a marker for. - `test_column_writer_caps_page_size_with_sparse_nulls` — pins two values per page under `PLAIN`, so it fails both if the encoding gate is dropped (pages would hold one) and if the bound is lost (they would hold many). Full `parquet` suite green (1312 lib + integration), `fmt` and `clippy -D warnings` clean. ## Note on #10505 Its `bool_to_int_with_if` trips `cargo clippy -- -D warnings`, which CI runs — worth fixing on that branch too. Corrected here as part of editing that test. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
…10979) # Which issue does this PR close? - Closes #10920. # Rationale for this change `read_thrift_vec` reserved `Vec` capacity from the compact list header's declared element count before reading any element. A 23-byte footer whose `schema` list claimed ~109 million `SchemaElement`s asked for ~10 GB. Allocation failure aborts the process; `ParquetMetaDataReader` never returns an error. Each list element occupies at least one byte on the wire, so the declared size can be bounded by the bytes remaining in the metadata slice. That slice is already length-limited by the footer; this check is the level below. # What changes are included in this PR? - `ThriftCompactInputProtocol::remaining_bytes` reports unread length for slice-backed input (`ThriftSliceInputProtocol`). Stream-backed input (`ThriftReadInputProtocol`) leaves it unset. - `read_thrift_vec` returns `ParquetError::General` when the declared size exceeds remaining input, and only then calls `Vec::with_capacity`. No public API change. # Are these changes tested? Unit tests in `parquet_thrift`: - well-formed 2-element `i32` list still decodes - 14-element header with no payload returns an error - declared size 109_002_364 with two leftover bytes returns an error (the schema-list case from the issue) `cargo test -p parquet --lib parquet_thrift` and `cargo clippy -p parquet --lib --tests --all-features -- -D warnings`. # Are there any user-facing changes? Malformed footers that previously aborted now return `ParquetError`. Valid files are unchanged. # AI Disclosure Assisted draft of the remaining-bytes check and regression tests. Reviewed against the compact protocol list encoding and verified with the tests above. Co-authored-by: Marcelo Tesla <9055877+M-Tesla@users.noreply.github.com>
) # Which issue does this PR close? - Closes #10951. # Rationale for this change For a struct with a non-zero parent offset, visible parent row `i` maps to child row `offset + i`. Nullability validation compared the visible parent null buffer against the unsliced child null buffer, so it could reject masked child nulls and accept visible child nulls. # What changes are included in this PR? - Slice each struct child to the parent-visible window before validating non-nullable fields. - Add a regression test covering both the correctly masked and incorrectly unmasked cases. # Are these changes tested? Yes. TDD evidence on `main` at `c134baf8f`: - Before the production change, the focused regression failed because the correctly masked child null was rejected. - After the change, the focused regression passes. - `cargo test -p arrow-data --lib`: 56 passed. - `cargo fmt --all -- --check`: passed. - `cargo clippy -p arrow-data --all-targets -- -D warnings`: passed. - `git diff --check`: passed. # Are there any user-facing changes? Yes. Validation now correctly accepts child nulls masked by a sliced struct parent and rejects child nulls visible through a non-null parent row. There are no API changes. # AI assistance I used AI assistance to investigate the validation path, implement the focused test and fix, and prepare this description. I reviewed the complete diff and verified the behavior with the commands above.
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes #10642 . - stacks on #9372 # Rationale for this change We need a benchmark to measure performance and prove future optimizations <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? - Added a criterion benchmark <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? - N/A <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -->
…fers on wasm32 (#10989) # Which issue does this PR close? * Closes #10986. # Rationale for this change On `wasm32`, `usize` is 32-bit, causing the compressed IPC buffer length prefix to be written as 4 bytes instead of the required 8-byte `i64` value. # What changes are included in this PR? Serialize the uncompressed buffer length explicitly as `i64` to ensure the IPC compression prefix is always 8 bytes across platforms. # Are these changes tested? Yes. Verified the generated compressed IPC buffer and confirmed the 8-byte length prefix is correctly written on `wasm32`. # Are there any user-facing changes? No. This is a bug fix for IPC compression compatibility on `wasm32`; there are no API or breaking changes. Co-authored-by: Narendran <narendranKT@lumel.com>
# Which issue does this PR close? - N/A # Rationale for this change Add `format_decimal.rs`, which formats arrays of 8192 random values through ArrayFormatter like `format_temporal.rs` does, for Decimal32, Decimal64, Decimal128 and Decimal256 at representative precisions and scales, with digit counts that cover values below the scale, values that fit an i128 and the widest Decimal256 values. There was previously no benchmark that covered formatting decimals. # What changes are included in this PR? See above. # Are these changes tested? Yes. # Are there any user-facing changes? No.
# Which issue does this PR close? - N/A # Rationale for this change Display for i256 converted the value to a num-bigint value and formatted that, which allocates the BigInt's digit vector and runs a generic arbitrary-precision conversion. By formatting the value ourselves, we can do better because we don't need to support arbitrary precision, and we can also avoid the heap allocation and type conversion overhead. format_decimal benchmark (#10997), Apple M4 Max: case before after change decimal256 (76, 10) 38 digits 1730.13 958.33 -44.6% decimal256 (76, 10) 76 digits 2088.60 1760.90 -15.7% The win is bigger for small Decimal256 values because we can do those entirely in i128; in principle num-bigint could implement a similar optimization for small values, but it currently does not. Decimal256 values larger than i128 are split into three chunks and formatted as i128; this is still faster than going through num-bigint. In practice, this improves the performance of writing out decimal values as CSV and JSON, as well as pretty-printing them. # What changes are included in this PR? See above. # Are these changes tested? Yes; new test added, existing tests pass. # Are there any user-facing changes? No; decimal format is unchanged. # AI usage Developed with Claude Code Fable 5.1; reviewed with Codex Astra 6. I reviewed, revised, and understand the resulting code. Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
) # Which issue does this PR close? - N/A # Rationale for this change `parse_decimal` validated the parsed value against the precision with `is_valid_decimal_precision`, an out-of-line call that loads the bounds for the precision and compares the value against both. The parser already knows how many digits it kept. We can modify the mantissa scan to return the digit count and then use this count to skip the precision check for the common case that the number of parsed digits implies the parsed value is well within the allowed precision. `parse_decimal` microbenchmark, Apple M4 Max, µs per 1024 inputs: ``` case before after change 123.123 7.93 7.85 -1.0% 123.1234 11.39 10.96 -3.7% 123.1 11.22 10.27 -8.5% 123 10.49 9.47 -9.7% -123.123 8.16 8.07 -1.0% -123.1234 11.47 11.09 -3.3% -123.1 11.45 10.53 -8.0% -123 10.71 9.84 -8.1% 0.0000123 8.37 7.89 -5.7% 12. 10.21 9.54 -6.5% -12. 10.44 9.76 -6.5% 00.1 10.69 9.90 -7.4% -00.1 10.93 10.00 -8.5% 12345678912345678.1234 24.49 24.53 +0.1% -12345678912345678.1234 24.67 24.77 +0.4% 99999999999999999.999 20.31 19.90 -2.0% -99999999999999999.999 20.25 20.40 +0.7% .123 6.34 6.18 -2.6% -.123 6.59 6.41 -2.7% 123. 10.67 10.12 -5.1% -123. 10.94 10.30 -5.8% string decimal128 short 8.20 7.85 -4.2% string decimal128 integer 11.19 10.89 -2.7% string decimal128 exact scale 11.43 11.05 -3.3% string decimal128 padded scale 10.50 10.42 -0.8% string decimal128 rounded scale 15.23 14.63 -3.9% string decimal128 signed 15.28 14.78 -3.3% string decimal128 38 digits 24.19 23.65 -2.2% string decimal128 exponent 20.28 20.18 -0.5% string decimal128 negative exponent 19.52 18.79 -3.7% string decimal128 negative scale 18.32 17.97 -1.9% string decimal128 long fraction 22.39 22.14 -1.1% string decimal256 76 digits 59.03 57.76 -2.1% string decimal256 rounded scale 55.72 54.71 -1.8% string decimal32 short 8.51 7.74 -9.1% string decimal32 9 digits 10.45 9.76 -6.6% string decimal64 short 7.29 6.84 -6.2% string decimal64 18 digits 12.04 11.66 -3.2% ``` # What changes are included in this PR? See above. # Are these changes tested? Yes; new test added, existing tests pass. # Are there any user-facing changes? No. # AI usage Developed with Claude Code Fable 5.1; reviewed with Codex Astra 6. I reviewed, revised, and understand the resulting code.
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes none. # Rationale for this change `filter_fixed_size_binary` used `MutableBuffer::new` which zero-initializes the buffer before immediately overwriting every byte. This skips that. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? Adds `copy_fsb_indices` helper used by the `IndexIterator` and `Indices` paths in `filter_fixed_size_binary`. Swaps `MutableBuffer::new` + `extend_from_slice` for `with_capacity` + `ptr::copy_nonoverlapping`. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? yes, existing test. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? no <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -->
## note I did use ai to help me with the bit maniluplation logic! # Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - follow up to #10976 # Rationale for this change `NullBuffer::expand` now has a fast path for count % 8 == 0 (byte-aligned) and count % 4 == 0 (nibble-aligned) counts. For all other values of count the fallback iterated every bit individually, calling set_bit per valid index. This is O(n × count) bit-level writes even when the validity buffer is mostly non-null. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? Replaces the bit-by-bit fallback in `NullBuffer::try_expand` with a BitSliceIterator based approach that works over contiguous runs of valid bits rather than individual bits. For each run [start, end) it computes the output byte range [start*count/8, end*count/8] and sets it with byte-level OR masks: - A partial leading byte is ORed with 0xFF << start_offset. - Full interior bytes are filled with 0xFF. - A partial trailing byte is ORed with (1 << end_offset) - 1. - When the entire run fits in a single byte, both masks are ANDed together. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? yes, test was introduced in #10976 <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? no <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -->
…10245) (#11005) ## What does this PR do? Replaces `BufferBuilder::<R::Native>` with `Vec::<R::Native>` when re-encoding sliced run-ends in `into_zero_offset_run_array` (`arrow-ipc/src/writer.rs`). This is part of #10245 — Rust's `Vec` has a highly optimized implementation, and switching from `BufferBuilder`/`OffsetBufferBuilder` to `Vec` typically yields a speedup. No behavior change: the re-encoded run-ends buffer is byte-identical. Existing run-array roundtrip tests (`test_run_array_unslice`, `test_roundtrip_stream_run_array_sliced`) cover the re-encoding path for all slice lengths and both slice offsets. ## Checklist - [x] Ran `cargo test -p arrow-ipc` (131 passed) - [x] Ran `cargo clippy -p arrow-ipc --all-targets` - [x] Ran `cargo fmt -p arrow-ipc -- --check`
…yData::validate` (#10709) # Which issue does this PR close? - Closes #7379. # Rationale for this change `ArrayData::validate` sizes the null bitmap length check with `len_plus_offset`, which folds in `ArrayData::offset`: ```rust let actual_len = nulls.validity().len(); let needed_len = bit_util::ceil(len_plus_offset, 8); ``` That offset does not apply to the null buffer. `ArrayData::nulls` says so directly: *"Note: `ArrayData::offset` does NOT apply to the returned `NullBuffer`"*. The `NullBuffer` carries its own offset. So a null buffer at offset 0 backing an array sliced to offset 50 is rejected, using the reporter's test: ``` InvalidArgumentError("null_bit_buffer size too small. got 7 needed 13") ``` Decoupled offsets are a supported state rather than something the validator was guarding against. `arrow-data/src/ffi.rs` `align_nulls` exists precisely because `data.offset() != nulls.offset()` is legal: it fast-paths when they match and re-aligns the bits otherwise. As the issue notes, only the false-rejection direction is reachable today, since there is currently no way to build an invalid `BooleanBuffer`. This is a correctness fix to the check, not a soundness fix. # What changes are included in this PR? One line in `ArrayData::validate`, sizing the check from the null buffer's own offset and length. The near-identical check in `ArrayData::try_new` is deliberately left alone. It takes a raw `null_bit_buffer: Option<Buffer>` for which the data offset genuinely does apply, and it is pinned by `arrow/tests/array_validation.rs` `test_bitmap_too_small`. I confirmed that test still passes. # Are these changes tested? Yes. Added `null_buffer_offset_is_independent_of_data_offset` in `arrow-data/src/data.rs`, covering the reporter's scenario: 100 values sliced to the last 50, then the same 50 nulls supplied both sliced (offset 50) and unsliced (offset 0). Both must validate. I checked it is not vacuous: reverting only the one-line fix while keeping the test makes it fail with the exact error from the issue, `got 7 needed 13`. Ran locally on `5ce0ebe`: * `cargo test -p arrow-data`: 44 passed, plus 13 doctests * `cargo test -p arrow-data -p arrow-array -p arrow-buffer -p arrow-select -p arrow-cast`: all green * `cargo test -p arrow --test array_validation`: 58 passed, including `test_bitmap_too_small` * `cargo fmt -p arrow-data -- --check` and `cargo clippy -p arrow-data --all-targets -- -D warnings`: clean One pre-existing failure unrelated to this change: `util::test_util::tests::test_happy` needs the `testing/data` submodule, which my clone did not initialize. It fails the same way on a pristine checkout. # Are there any user-facing changes? `ArrayData::validate` no longer rejects a valid null buffer whose offset differs from the array's. No API change. No behavior change for null buffers whose offset already matched the data offset. # AI disclosure Per `CONTRIBUTING.md` "AI Generated Submissions". I used an AI assistant to help draft the fix, the test and this description. I reproduced the failure with the reporter's verbatim test before changing anything, confirmed the `try_new` site is a separate case that must not change and that its pinning test still passes, and verified the new test fails with the fix reverted. I ran every command listed above myself. --------- Signed-off-by: Ali <alliasgher123@gmail.com>
# Which issue does this PR close? - Closes #10973. # Rationale for this change `collect_bool` reserves ceil(len / 64) u64 words before it calls the closure once, so a `len` that comes from a row or bit count in user controlled data aborts the process rather than failing. the reservation went through `Vec::with_capacity`, which has no way to report that. this file already has the shape for it. `with_capacity` is `try_with_capacity(..).unwrap_or_else(|e| panic!("{e}"))` at line 154, and there are seven other fallible twins beside it: `try_from_len_zeroed`, `try_reserve`, `try_repeat_slice_n_times`, `try_resize`, `try_shrink_to_fit`, `try_extend_from_slice` and `try_extend_zeros`. `collect_bool` is the allocating entry point that does not have one. # What changes are included in this PR? - add `try_collect_bool` returning `Result<Self, MutableBufferError>` - `collect_bool` calls it and unwraps, the same way `with_capacity` calls `try_with_capacity` - the reservation goes through `Vec::try_reserve`, and a failure maps onto the existing `AllocationError` variant carrying the layout it tried to take no behaviour change for any `len` that already worked. `collect_bool` still panics, one level down, and its doc comment now says so. i went with a fallible entry point rather than capping the reservation, since a cap only moves where the abort happens rather than letting a caller handle it. happy to do the cap instead if you would rather not grow the api surface. # Are these changes tested? yes, two tests. `try_collect_bool_reports_a_len_it_cannot_reserve` uses the 2^60 from the issue and asserts it comes back as `AllocationError` and that the closure ran zero times. the reproducer in the issue aborts on that same input with `memory allocation of 144115188075855872 bytes failed`. `try_collect_bool_matches_collect_bool_for_sizes_that_fit` checks the two agree byte for byte at 0, 1, 63, 64, 65 and 1000 bits, since 64 is the word boundary and 8 is the truncation boundary. `arrow-buffer` is 351 passed, `arrow-array` is 721 passed, and fmt and clippy with `-D warnings` are clean. i ran arrow-array as well because `collect_bool` has around twenty call sites in it and this changes how the function is built. # Are there any user-facing changes? `try_collect_bool` is new and additive. `collect_bool` keeps its signature and still panics on a `len` it cannot reserve. --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
…1009) # Which issue does this PR close? - Closes #11008. # Rationale for this change The Arrow spec allows sparse unions to contain arbitrary values in "unselected" children: > Only the slot in the array corresponding to the type index is considered. All "unselected" values are ignored and could be any semantically correct array value. Array equality for sparse unions (`ArrayData::eq`) compares every child array over the full slot range, regardless of which child each slot's type id selects. This is inconsistent with the spec, as well as the Arrow C++ implementation. # What changes are included in this PR? * Fix equality semantics for sparse unions * Add unit tests # Are these changes tested? Yes, new tests added. # Are there any user-facing changes? Yes, equality semantics for sparse unions have changed. However, it is pretty unlikely that user code would depend on the previous behavior (if anything, it would probably result in bugs in user code). # AI usage Developed with Claude Code Fable 5.1. I reviewed and revised the resulting code.
Bumps [brotli](https://github.com/dropbox/rust-brotli) from 8.0.4 to 9.0.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/dropbox/rust-brotli/commit/a51b65e75f68e3485be12bc816118e523e95647d"><code>a51b65e</code></a> 9.0.0 release</li> <li><a href="https://github.com/dropbox/rust-brotli/commit/fb3104d1188106ad819d8bfbfd87ea47d0087423"><code>fb3104d</code></a> Add non-default portable-float feature for target-independent encoder output</li> <li><a href="https://github.com/dropbox/rust-brotli/commit/c9af3defc656269e90abca762b3adf05a50ee5bc"><code>c9af3de</code></a> Fix <a href="https://redirect.github.com/dropbox/rust-brotli/issues/257">#257</a></li> <li><a href="https://github.com/dropbox/rust-brotli/commit/93ea851349e9b44e63436c5f530c0d85e5d66472"><code>93ea851</code></a> fix: handle short empty streams in BroCatli</li> <li><a href="https://github.com/dropbox/rust-brotli/commit/5bf2ed82a362879dd99fbea8106bdade3998bd9a"><code>5bf2ed8</code></a> fixed simd (.to_int() renamed to .to_simd())</li> <li><a href="https://github.com/dropbox/rust-brotli/commit/ac02943519bd829ff2e025deca40388248f19170"><code>ac02943</code></a> Revert "fixed simd (.to_int() renamed to .to_simd())"</li> <li><a href="https://github.com/dropbox/rust-brotli/commit/b4ed6ac16fa1d43efe02284b94743947bbfcdea4"><code>b4ed6ac</code></a> fixed simd (.to_int() renamed to .to_simd())</li> <li><a href="https://github.com/dropbox/rust-brotli/commit/3e7adf2fb56590d2378df08e2b37fd660cca3379"><code>3e7adf2</code></a> fixed simd (mask.to_int() renamed to mask.to_simd())</li> <li>See full diff in <a href="https://github.com/dropbox/rust-brotli/compare/8.0.4...9.0.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [rust-toolchain](https://github.com/rust-lang/rust) from 1.97.1 to 1.98.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/rust-lang/rust/releases">rust-toolchain's releases</a>.</em></p> <blockquote> <h2>Rust 1.98.0</h2> <p><!-- raw HTML omitted --><!-- raw HTML omitted --></p> <h2>Language</h2> <ul> <li><a href="https://redirect.github.com/rust-lang/rust/pull/149219">Allow shortening lifetime of <code>&mut</code> when unsize-coercing, even in an invariant position.</a> For example, you can now coerce a <code>Cell<&'long mut i32></code> to a <code>Cell<&'short mut dyn Send></code>. Such shortenings were already previously allowed when coercing a <code>&mut</code> to a <code>&</code>, or coercing a <code>&</code> to a <code>&</code>.</li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155521">Add deny-by-default <code>invalid_runtime_symbol_definitions</code> lint and warn-by-default <code>suspicious_runtime_symbol_definitions</code> lint</a> <ul> <li>The lints currently specifically targets <code>core</code> runtime symbols like <code>memcmp</code>, <code>memset</code>, <code>strlen</code>, ... and is planned to be expanded in the next few releases.</li> </ul> </li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/156379">Add warn-by-default <code>c_void_returns</code> lint to check <code>core::ffi::c_void</code> as a return type</a></li> </ul> <p><!-- raw HTML omitted --><!-- raw HTML omitted --></p> <h2>Platform Support</h2> <ul> <li><a href="https://redirect.github.com/rust-lang/rust/pull/144220">Add <code>powerpc64-unknown-linux-gnuelfv2</code> as Tier 3</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155722">Add <code>aarch64-unknown-linux-pauthtest</code> as Tier 3 target</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155763">Promote <code>thumbv7a-none-eabi</code> to Tier 2</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155763">Promote <code>thumbv7a-none-eabihf</code> to Tier 2</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155763">Promote <code>thumbv7r-none-eabi</code> to Tier 2</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155763">Promote <code>thumbv7r-none-eabihf</code> to Tier 2</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155763">Promote <code>thumbv8r-none-eabihf</code> to Tier 2</a></li> </ul> <p>Refer to Rust's <a href="https://doc.rust-lang.org/rustc/platform-support.html">platform support page</a> for more information on Rust's tiered platform support.</p> <p><!-- raw HTML omitted --><!-- raw HTML omitted --></p> <h2>Libraries</h2> <ul> <li><a href="https://redirect.github.com/rust-lang/rust/pull/146561">Change <code>Location<'_></code> lifetime to <code>'static</code> in <code>Panic[Hook]Info</code></a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155421">Document panic in <code>RangeInclusive::from(legacy::RangeInclusive)</code></a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155750">Document that <code>ManuallyDrop</code>'s <code>Box</code> interaction has been fixed</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/issues/156908">Stabilize LoongArch CRC Intrinsics</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/issues/154645">The <code>derive</code> macro is available at <code>{core,std}::derive</code>.</a> This was previously <a href="https://redirect.github.com/rust-lang/rust/issues/159856">unintentionally stabilized in 1.96</a>, but is now <a href="https://redirect.github.com/rust-lang/rust/issues/154645">explicitly accepted</a> as a stabilized API. <ul> <li>Please note that the MSRV for <code>{core,std}::derive</code> will be 1.96, and not 1.98.</li> </ul> </li> </ul> <p><!-- raw HTML omitted --><!-- raw HTML omitted --></p> <h2>Stabilized APIs</h2> <ul> <li><a href="https://doc.rust-lang.org/stable/std/primitive.str.html#method.substr_range"><code>str::substr_range</code></a></li> <li><a href="https://doc.rust-lang.org/stable/std/primitive.slice.html#method.subslice_range"><code>[T]::subslice_range</code></a></li> <li><a href="https://doc.rust-lang.org/stable/core/fmt/struct.NumBuffer.html"><code>core::fmt::NumBuffer</code></a></li> <li><a href="https://doc.rust-lang.org/stable/core/primitive.usize.html#method.format_into"><code><{integer}>::format_into</code></a></li> <li><a href="https://doc.rust-lang.org/stable/std/process/struct.CommandArgs.html#impl-Send-for-CommandArgs%3C'a%3E"><code>Send/Sync for std::process::CommandArgs</code></a></li> <li><a href="https://doc.rust-lang.org/stable/core/primitive.f32.html#method.algebraic_add"><code>{fN}::algebraic_add</code></a></li> <li><a href="https://doc.rust-lang.org/stable/core/primitive.f32.html#method.algebraic_sub"><code>{fN}::algebraic_sub</code></a></li> <li><a href="https://doc.rust-lang.org/stable/core/primitive.f32.html#method.algebraic_mul"><code>{fN}::algebraic_mul</code></a></li> <li><a href="https://doc.rust-lang.org/stable/core/primitive.f32.html#method.algebraic_div"><code>{fN}::algebraic_div</code></a></li> <li><a href="https://doc.rust-lang.org/stable/core/primitive.f32.html#method.algebraic_rem"><code>{fN}::algebraic_rem</code></a></li> <li><a href="https://doc.rust-lang.org/stable/core/num/struct.NonZero.html#method.from_str_radix-4"><code>NonZero<{integer}>::from_str_radix</code></a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/rust-lang/rust/blob/main/RELEASES.md">rust-toolchain's changelog</a>.</em></p> <blockquote> <h1>Version 1.98.0 (2026-08-20)</h1> <p><!-- raw HTML omitted --><!-- raw HTML omitted --></p> <h2>Language</h2> <ul> <li><a href="https://redirect.github.com/rust-lang/rust/pull/149219">Allow shortening lifetime of <code>&mut</code> when unsize-coercing, even in an invariant position.</a> For example, you can now coerce a <code>Cell<&'long mut i32></code> to a <code>Cell<&'short mut dyn Send></code>. Such shortenings were already previously allowed when coercing a <code>&mut</code> to a <code>&</code>, or coercing a <code>&</code> to a <code>&</code>.</li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155521">Add deny-by-default <code>invalid_runtime_symbol_definitions</code> lint and warn-by-default <code>suspicious_runtime_symbol_definitions</code> lint</a> <ul> <li>The lints currently specifically targets <code>core</code> runtime symbols like <code>memcmp</code>, <code>memset</code>, <code>strlen</code>, ... and is planned to be expanded in the next few releases.</li> </ul> </li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/156379">Add warn-by-default <code>c_void_returns</code> lint to check <code>core::ffi::c_void</code> as a return type</a></li> </ul> <p><!-- raw HTML omitted --><!-- raw HTML omitted --></p> <h2>Platform Support</h2> <ul> <li><a href="https://redirect.github.com/rust-lang/rust/pull/144220">Add <code>powerpc64-unknown-linux-gnuelfv2</code> as Tier 3</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155722">Add <code>aarch64-unknown-linux-pauthtest</code> as Tier 3 target</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155763">Promote <code>thumbv7a-none-eabi</code> to Tier 2</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155763">Promote <code>thumbv7a-none-eabihf</code> to Tier 2</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155763">Promote <code>thumbv7r-none-eabi</code> to Tier 2</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155763">Promote <code>thumbv7r-none-eabihf</code> to Tier 2</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155763">Promote <code>thumbv8r-none-eabihf</code> to Tier 2</a></li> </ul> <p>Refer to Rust's <a href="https://doc.rust-lang.org/rustc/platform-support.html">platform support page</a> for more information on Rust's tiered platform support.</p> <p><!-- raw HTML omitted --><!-- raw HTML omitted --></p> <h2>Libraries</h2> <ul> <li><a href="https://redirect.github.com/rust-lang/rust/pull/146561">Change <code>Location<'_></code> lifetime to <code>'static</code> in <code>Panic[Hook]Info</code></a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155421">Document panic in <code>RangeInclusive::from(legacy::RangeInclusive)</code></a></li> <li><a href="https://redirect.github.com/rust-lang/rust/pull/155750">Document that <code>ManuallyDrop</code>'s <code>Box</code> interaction has been fixed</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/issues/156908">Stabilize LoongArch CRC Intrinsics</a></li> <li><a href="https://redirect.github.com/rust-lang/rust/issues/154645">The <code>derive</code> macro is available at <code>{core,std}::derive</code>.</a> This was previously <a href="https://redirect.github.com/rust-lang/rust/issues/159856">unintentionally stabilized in 1.96</a>, but is now <a href="https://redirect.github.com/rust-lang/rust/issues/154645">explicitly accepted</a> as a stabilized API. <ul> <li>Please note that the MSRV for <code>{core,std}::derive</code> will be 1.96, and not 1.98.</li> </ul> </li> </ul> <p><!-- raw HTML omitted --><!-- raw HTML omitted --></p> <h2>Stabilized APIs</h2> <ul> <li><a href="https://doc.rust-lang.org/stable/std/primitive.str.html#method.substr_range"><code>str::substr_range</code></a></li> <li><a href="https://doc.rust-lang.org/stable/std/primitive.slice.html#method.subslice_range"><code>[T]::subslice_range</code></a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/rust-lang/rust/commit/88d9e12ae178fab0fb5cc050a94da85685d449ea"><code>88d9e12</code></a> Auto merge of <a href="https://redirect.github.com/rust-lang/rust/issues/161313">#161313</a> - cuviper:stable-next, r=cuviper</li> <li><a href="https://github.com/rust-lang/rust/commit/8f6f43c06e1f0f0d80f2a6616a711ed99a987453"><code>8f6f43c</code></a> update relnotes</li> <li><a href="https://github.com/rust-lang/rust/commit/c6d0dd078ec3ac8828f3cd58125a1b1c89066e46"><code>c6d0dd0</code></a> Bump stable channel</li> <li><a href="https://github.com/rust-lang/rust/commit/824ef7e4e02b43a361849bf49c58df63bf378891"><code>824ef7e</code></a> Revert "riscv: promote d, e, and f target_features to CfgStableToggleUnstable"</li> <li><a href="https://github.com/rust-lang/rust/commit/04eb0d7194012623e82369b466d5084522957446"><code>04eb0d7</code></a> Auto merge of <a href="https://redirect.github.com/rust-lang/rust/issues/161039">#161039</a> - flip1995:clippy-beta-backport, r=Mark-Simulacrum</li> <li><a href="https://github.com/rust-lang/rust/commit/6cc28588ea61d68cc901e5c8257c0926a5832b27"><code>6cc2858</code></a> Fix ICE when resolving local in <code>unnecessary_unwrap_unchecked</code> (<a href="https://redirect.github.com/rust-lang/rust/issues/17353">#17353</a>)</li> <li><a href="https://github.com/rust-lang/rust/commit/d440739bfd7b68ca41f4b028ba3502901a35b97f"><code>d440739</code></a> <code>unnecessary_unwrap_unchecked</code>: don't trigger inside the <code>_unchecked</code> fn (<a href="https://redirect.github.com/rust-lang/rust/issues/17">#17</a>...</li> <li><a href="https://github.com/rust-lang/rust/commit/e1114a15d06a78347870c2291ed20480a8b1008b"><code>e1114a1</code></a> Auto merge of <a href="https://redirect.github.com/rust-lang/rust/issues/160714">#160714</a> - cuviper:beta-next, r=cuviper</li> <li><a href="https://github.com/rust-lang/rust/commit/ebcacaa7ae16f0766782ec8e9db3e1eda8bd7627"><code>ebcacaa</code></a> [beta-1.98] Update cargo submodule</li> <li><a href="https://github.com/rust-lang/rust/commit/33c0be17df8184eb540e2b387b1dc8a7feb04800"><code>33c0be1</code></a> Update attr name for <code>#[unroll]</code> in Unstable Book</li> <li>Additional commits viewable in <a href="https://github.com/rust-lang/rust/compare/1.97.1...1.98.0">compare view</a></li> </ul> </details> <br /> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jefffrey <jeffrey.vo.australia@gmail.com>
# Which issue does this PR close? - Closes #6897. # Rationale for this change rustfmt does not discover out-of-line modules declared inside macros (rust-lang/rustfmt#3253). The experimental module macro therefore causes cargo fmt to skip several Parquet module trees. This uses the same approach as delta-io/delta-kernel-rs#935. # What changes are included in this PR? - Replace experimental module macro declarations with explicit, mutually exclusive feature-gated declarations while preserving visibility and documentation behavior. - Remove the separate Parquet formatting workaround from CI. - Remove the workaround from the contributor documentation and use the standard workspace formatting command. # Are these changes tested? - cargo fmt --all -- --check - cargo check -p parquet - cargo check -p parquet --all-features - cargo test -p parquet --lib encodings::rle (15 passed) - Verbose cargo fmt output confirms that the formerly skipped module trees are traversed. A full cargo test -p parquet --lib run compiled successfully and passed 1,278 tests; 93 fixture-dependent tests could not run because the local checkout does not have the testing and parquet-testing data submodules initialized. # Are there any user-facing changes? Contributors can now format all Parquet sources with the standard cargo fmt --all command. There are no public API changes. # AI usage OpenAI Codex was used to investigate the rustfmt behavior, implement the feature-gated module declarations, update CI and contributor documentation, and run the validation commands reported above. The resulting code and PR description were AI-assisted.
## Why are the changes needed? ### Which issue does this PR close? Closes #10845. ### Rationale for this change A dense Union can have a child longer than `i32::MAX` while referencing only valid `i32` offsets. `UnionArray::try_new` truncates child lengths to `i32`, so the length can collide with its missing-type sentinel or fail the bounds check. A `NullArray` reproduces this without a large allocation. ## What changes were proposed in this PR? ### What changes are included in this PR? Keep child lengths as `Option<usize>`, separating a missing type ID from a valid length. Reject negative offsets before converting them to `usize` for comparison. Add coverage for lengths `i32::MAX + 1` and `i32::MAX + 2`, offsets zero and `i32::MAX`, and invalid negative offsets. ### Are there any user-facing changes? Previously rejected valid Unions are accepted. Invalid type IDs and out-of-bounds offsets still return errors. No public API changes. ## How was this PR tested? ### Are these changes tested? - The large-child test failed on the unchanged base with the type-ID error, then passed with the fix; both constructed arrays pass full Arrow validation. - `cargo test --offline -p arrow-array --lib`: 719 passed, 1 ignored. - `cargo clippy --offline -p arrow-array --all-targets --all-features -- -D warnings`: passed. - `cargo fmt --all -- --check`: passed. - Independent source review checked missing IDs, empty children, signed offsets, and the safety of the lookup after type-ID validation. AI assistance: Codex generated the implementation, regression tests, and PR text, and performed the stated local checks and source review. This does not claim a separate human review.
…es (#10842) # Which issue does this PR close? - Part of #7582 - Closes #10824. # Rationale for this change This grew out of a discussion in #10784 and relates to apache/datafusion#24288 (comment). This PR provides a new builder for creating page index structures, and also adds a new `PageIndexProvider` trait to allow more performant implementations. # What changes are included in this PR? Adds `PageIndexBuilder`, `PageIndexProvider`, implements `PageIndexProvider` for `PageIndex`, and adds `RowGroupPageIndex` as a helper to support fetching page indexes for a specific row group (replaces the `X_index_for_rowgroup()` functions on `PageIndex`). # Are these changes tested? Yes, should be covered by existing tests # Are there any user-facing changes? Yes, this changes the public API for accessing page index information Created with the aid of Claude Code, but I own the changes. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…ith 2 updates (#11023) Bumps the all-other-cargo-deps group with 2 updates in the / directory: [indexmap](https://github.com/indexmap-rs/indexmap) and [syn](https://github.com/dtolnay/syn). Updates `indexmap` from 2.14.1 to 2.14.2 <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/indexmap-rs/indexmap/blob/main/RELEASES.md">indexmap's changelog</a>.</em></p> <blockquote> <h2>2.14.2 (2026-09-04)</h2> <ul> <li>Fix item hygiene in map and set macros. Previously, an internal <code>const CAP</code> could shadow the same name in the caller's namespace.</li> <li>Allow <code>const</code> initialization of empty <code>indexmap_with_default!</code> and <code>indexset_with_default!</code>. The hasher may also be omitted if it's inferrable.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/indexmap-rs/indexmap/commit/41a870887c4c77adf665886e63df08f406bfe37a"><code>41a8708</code></a> Merge pull request <a href="https://redirect.github.com/indexmap-rs/indexmap/issues/450">#450</a> from cuviper/macros</li> <li><a href="https://github.com/indexmap-rs/indexmap/commit/0fb7b5c813c94dcb16f220d249e42a7b9e4ea44d"><code>0fb7b5c</code></a> Release 2.14.2</li> <li><a href="https://github.com/indexmap-rs/indexmap/commit/55e6b28278f242b4d4190a14b6e86c09580dd75a"><code>55e6b28</code></a> Improve the empty macro cases</li> <li><a href="https://github.com/indexmap-rs/indexmap/commit/c067355e1ca5ec39348f42dcee582d4f222c3295"><code>c067355</code></a> Fix item hygiene in user macros</li> <li>See full diff in <a href="https://github.com/indexmap-rs/indexmap/compare/2.14.1...2.14.2">compare view</a></li> </ul> </details> <br /> Updates `syn` from 3.0.4 to 3.0.5 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/dtolnay/syn/releases">syn's releases</a>.</em></p> <blockquote> <h2>3.0.5</h2> <ul> <li>Report correct span for lex errors from <code>LitStr::parse_with</code> (<a href="https://redirect.github.com/dtolnay/syn/issues/2080">#2080</a>, thanks <a href="https://github.com/sunshowers"><code>@sunshowers</code></a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/dtolnay/syn/commit/e0ad92d68b588c964e82bb74c3bd3e1e99f97ceb"><code>e0ad92d</code></a> Release 3.0.5</li> <li><a href="https://github.com/dtolnay/syn/commit/74e7d75f263942576a25d5ed4bf11d0d1e80e698"><code>74e7d75</code></a> Merge pull request <a href="https://redirect.github.com/dtolnay/syn/issues/2080">#2080</a> from sunshowers/lit-str-span</li> <li><a href="https://github.com/dtolnay/syn/commit/4c264f3db3f0d567fb6a6ddfe1e129bbe5c2acbe"><code>4c264f3</code></a> In LitStr::parse_with, report correct span for lex errors</li> <li><a href="https://github.com/dtolnay/syn/commit/7e2b27bc9331ba59c8e3a25a8eba2f5c647317bc"><code>7e2b27b</code></a> Update test suite to nightly-2026-08-26</li> <li>See full diff in <a href="https://github.com/dtolnay/syn/compare/3.0.4...3.0.5">compare view</a></li> </ul> </details> <br /> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.86.5 to 2.87.6. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/taiki-e/install-action/releases">taiki-e/install-action's releases</a>.</em></p> <blockquote> <h2>2.87.6</h2> <ul> <li> <p>Update <code>rafn@latest</code> to 0.1.6.</p> </li> <li> <p>Update <code>editorconfig-checker@latest</code> to 3.11.3.</p> </li> <li> <p>Update <code>dprint@latest</code> to 0.57.1.</p> </li> <li> <p>Update <code>convco@latest</code> to 0.7.2.</p> </li> </ul> <h2>2.87.5</h2> <ul> <li> <p>Update <code>vacuum@latest</code> to 0.30.3.</p> </li> <li> <p>Update <code>uv@latest</code> to 0.12.9.</p> </li> <li> <p>Update <code>typos@latest</code> to 1.50.1.</p> </li> <li> <p>Update <code>tombi@latest</code> to 1.5.1.</p> </li> <li> <p>Update <code>release-plz@latest</code> to 0.3.161.</p> </li> <li> <p>Update <code>prek@latest</code> to 0.5.2.</p> </li> <li> <p>Update <code>oxfmt@latest</code> to 1.81.0.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.9.1.</p> </li> <li> <p>Update <code>martin@latest</code> to 1.15.0.</p> </li> <li> <p>Update <code>git-cliff@latest</code> to 2.14.1.</p> </li> <li> <p>Update <code>cargo-zigbuild@latest</code> to 0.23.4.</p> </li> <li> <p>Update <code>cargo-deb@latest</code> to 3.8.0.</p> </li> <li> <p>Update <code>biome@latest</code> to 2.5.12.</p> </li> </ul> <h2>2.87.4</h2> <ul> <li> <p>Update <code>uv@latest</code> to 0.12.8.</p> </li> <li> <p>Update <code>protoc@latest</code> to 3.36.1.</p> </li> <li> <p>Update <code>coreutils@latest</code> to 0.11.0.</p> </li> </ul> <h2>2.87.3</h2> <ul> <li> <p>Update <code>zizmor@latest</code> to 1.30.0.</p> </li> <li> <p>Update <code>ubi@latest</code> to 0.12.0.</p> </li> <li> <p>Update <code>protoc-gen-connect-openapi@latest</code> to 0.25.8.</p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md">taiki-e/install-action's changelog</a>.</em></p> <blockquote> <h1>Changelog</h1> <p>All notable changes to this project will be documented in this file.</p> <p>This project adheres to <a href="https://semver.org">Semantic Versioning</a>.</p> <!-- raw HTML omitted --> <h2>[Unreleased]</h2> <h2>[2.87.8] - 2026-09-07</h2> <ul> <li> <p>Update <code>shfmt@latest</code> to 3.14.1.</p> </li> <li> <p>Update <code>release-plz@latest</code> to 0.3.162.</p> </li> <li> <p>Update <code>protoc-gen-connect-openapi@latest</code> to 0.26.0.</p> </li> <li> <p>Update <code>dprint@latest</code> to 0.57.4.</p> </li> <li> <p>Update <code>cargo-llvm-cov@latest</code> to 0.9.1.</p> </li> <li> <p>Update <code>cargo-crap@latest</code> to 0.5.0.</p> </li> <li> <p>Update <code>cargo-binstall@latest</code> to 1.23.0.</p> </li> </ul> <h2>[2.87.7] - 2026-09-06</h2> <ul> <li> <p>Update <code>wasm-bindgen@latest</code> to 0.2.128.</p> </li> <li> <p>Update <code>uv@latest</code> to 0.12.10.</p> </li> <li> <p>Update <code>tombi@latest</code> to 1.5.2.</p> </li> <li> <p>Update <code>rclone@latest</code> to 1.75.1.</p> </li> </ul> <h2>[2.87.6] - 2026-09-05</h2> <ul> <li> <p>Update <code>rafn@latest</code> to 0.1.6.</p> </li> <li> <p>Update <code>editorconfig-checker@latest</code> to 3.11.3.</p> </li> <li> <p>Update <code>dprint@latest</code> to 0.57.1.</p> </li> <li> <p>Update <code>convco@latest</code> to 0.7.2.</p> </li> </ul> <h2>[2.87.5] - 2026-09-04</h2> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/taiki-e/install-action/commit/7b8d4719ee4aaa279bdf55df38dacb9ebfe12a6c"><code>7b8d471</code></a> Release 2.87.6</li> <li><a href="https://github.com/taiki-e/install-action/commit/a67fcd18912da6d4796bca317d5cab21011e4100"><code>a67fcd1</code></a> Update wasm-bindgen manifest</li> <li><a href="https://github.com/taiki-e/install-action/commit/efe69d32714f20bb09c75259e13aff27b522a932"><code>efe69d3</code></a> Update uv manifest</li> <li><a href="https://github.com/taiki-e/install-action/commit/ce51454fc8877f04f22b2f56d89ea2651ed30e39"><code>ce51454</code></a> Update tombi manifest</li> <li><a href="https://github.com/taiki-e/install-action/commit/e3284bdb652397d2e383409c416379d54965a495"><code>e3284bd</code></a> Update <code>rafn@latest</code> to 0.1.6</li> <li><a href="https://github.com/taiki-e/install-action/commit/50bb7c8acf710975ea99f8a03c5c0908eb81a222"><code>50bb7c8</code></a> Update <code>editorconfig-checker@latest</code> to 3.11.3</li> <li><a href="https://github.com/taiki-e/install-action/commit/2e1a951416180a0bba89f6085bdc643badc7bcf6"><code>2e1a951</code></a> Update <code>dprint@latest</code> to 0.57.1</li> <li><a href="https://github.com/taiki-e/install-action/commit/2d664e31d201ebb33db21aa8f8ed354b56349426"><code>2d664e3</code></a> Update <code>convco@latest</code> to 0.7.2</li> <li><a href="https://github.com/taiki-e/install-action/commit/5f8ff1a44efd92cf3d48505f1adfee33393eb853"><code>5f8ff1a</code></a> ci: Disable debian 11</li> <li><a href="https://github.com/taiki-e/install-action/commit/5bf6ce016fd2e72eefc647cbca1e4213f65955b8"><code>5bf6ce0</code></a> Release 2.87.5</li> <li>Additional commits viewable in <a href="https://github.com/taiki-e/install-action/compare/v2.86.5...7b8d4719ee4aaa279bdf55df38dacb9ebfe12a6c">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [zstd](https://github.com/gyscos/zstd-rs) from 0.13.3 to 0.14.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/gyscos/zstd-rs/commit/648acb476da66b4bb856043c9b8ea1fdbfe69093"><code>648acb4</code></a> Avoid let...else in the seekable callbacks</li> <li><a href="https://github.com/gyscos/zstd-rs/commit/e1152c1161fa6704b0b74bd538a9c92d2fe784a4"><code>e1152c1</code></a> Bump versions for the next release</li> <li><a href="https://github.com/gyscos/zstd-rs/commit/7caed6e87ae89a31524018d2b3da023f3e4e724e"><code>7caed6e</code></a> Derive the usual traits on ResetDirective</li> <li><a href="https://github.com/gyscos/zstd-rs/commit/bf7b1f762e5593c91f8b276eed7a15716fc14308"><code>bf7b1f7</code></a> Refuse to reuse a context an error may have left undefined</li> <li><a href="https://github.com/gyscos/zstd-rs/commit/15656186f8b1e541cc86da7cade2187afca0015d"><code>1565618</code></a> Check the target, not the host, for MSVC</li> <li><a href="https://github.com/gyscos/zstd-rs/commit/8315a6265d93f757a6fd8cd5b392207423731f1f"><code>8315a62</code></a> Return Ok(0) from Read::read for an empty buffer</li> <li><a href="https://github.com/gyscos/zstd-rs/commit/a7cfa93a6e265ec7d89be5a6f65024dffab89f71"><code>a7cfa93</code></a> Keep the std gate on the Cursor WriteBuf impl</li> <li><a href="https://github.com/gyscos/zstd-rs/commit/9bf1692593efe098ec1cf08f48c1cba149b2c672"><code>9bf1692</code></a> Say why the Send and Sync impls hold</li> <li><a href="https://github.com/gyscos/zstd-rs/commit/681bcc36c49d568d439eccf6722c8a33e0f52cde"><code>681bcc3</code></a> Don't truncate Cursor positions on 32-bit targets</li> <li><a href="https://github.com/gyscos/zstd-rs/commit/d5a1fdd7724c24aab5c32984cce9b6c40574dab2"><code>d5a1fdd</code></a> Don't hand out a &mut Seekable from AdvancedSeekable</li> <li>Additional commits viewable in <a href="https://github.com/gyscos/zstd-rs/compare/v0.13.3...v0.14.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Broke in #10046. Same fix as #10915 # Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes #11025. # Rationale for this change (see issue). <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? Yes. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? Yes, this fixes interleave semantics to the semantics in v58.3, which I had before upgrading today. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -->
#11020) # Which issue does this PR close? Closes #11019. # Rationale for this change `DictionaryBatch.data` is optional in the flatbuffer grammar and required by the format. `get_dictionary_values` opened it with `unwrap`, so a file that omits it panicked instead of producing an `ArrowError`. This is the same class of field, in the same file, that the footer's custom metadata was just fixed for: ```rust let (Some(key), Some(value)) = (kv.key(), kv.value()) else { return Err(ArrowError::ParseError( "Custom metadata in the IPC footer is missing a key or a value".to_string(), )); }; ``` The patch uses that shape. The proposal is essentially "finish the round". A panic matters more than usual on this path: under `libfuzzer-sys` a caught panic still becomes `abort()` before unwinding, so a fuzz target reading untrusted IPC goes into quarantine rather than reporting a rejected input. # What changes are included in this PR? One `let ... else` in `get_dictionary_values`, plus a regression test. Measured before and after on `main`: the fixture panics without the change and returns `Parser error: Dictionary batch is missing its data` with it. `cargo test -p arrow-ipc` passes — 132, 7 and 11. # Are these changes tested? Yes. The fixture is the original 2169-byte input from the fuzzing corpus of [plenora-IO-tools](https://github.com/PlenoraETL/plenora-IO-tools), not a reduced one, added under `arrow-ipc/test/data/`. # Are there any user-facing changes? A file that used to panic now returns an `ArrowError::ParseError`. No API change. Co-authored-by: PlenoraETL <4678670+PlenoraETL@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?