Add page-level I/O and materialization in Hybrid Scan by mhaseeb123 · Pull Request #23375 · NVIDIA/cudf

mhaseeb123 · 2026-07-21T17:43:20Z

Description

Contributes to #23519

This PR adds new hybrid scan APIs to support page-level (sparse) I/O for payload columns. This includes:

API to compute (pruned) page-level I/O byte range for payload columns
API to only materialize these sparse pages into the output table
End-to-end test coverage and some negative tests. More extensive tests will land in Enable page-level payload I/O in hybrid scan #23362

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

Preserve page locations and variable-width offset state needed to safely reconstruct columns from a sparse subset of Parquet data pages.

Expose multifile page-range planning and consume the selected payload pages so hybrid scan avoids fetching pruned Parquet payload data.

copy-pr-bot · 2026-07-21T17:49:59Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Part of #23362 This PR includes bug fixes and supporting features needed to enable page pruning with page-level (sparse) I/O (for payload columns) in hybrid scan, that is upcoming in #23375 and includes end to end tests. Authors: - Muhammad Haseeb (https://github.com/mhaseeb123) Approvers: - Paul Mattione (https://github.com/pmattione-nvidia) - Vukasin Milovanovic (https://github.com/vuule) - Bradley Dice (https://github.com/bdice) - Nghia Truong (https://github.com/ttnghia) URL: #23374

…sparse-page-io-hybrid

coderabbitai · 2026-07-31T23:41:55Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Summary by CodeRabbit

New Features
- Added sparse, page-level payload scanning for hybrid multifile Parquet reads.
- Added row-mask-based page selection and targeted chunking for more efficient filtered reads.
- Added support for dictionary pages, pruned pages, and chunked multifile scans.
Bug Fixes
- Improved handling of empty buffers, sparse page metadata, dictionary data, and page masks.
- Added validation for missing offset indexes and inconsistent page data.

Walkthrough

The PR adds payload page-range APIs, sparse page-data chunking, page-level pass masks, sparse Parquet preprocessing, and multifile hybrid-scan coverage for pruning, dictionaries, missing offset indexes, and row-group ordering.

Changes

Sparse payload hybrid scan

Layer / File(s)	Summary
Payload range planning `cpp/include/cudf/io/experimental/hybrid_scan_multifile.hpp`, `cpp/src/io/parquet/experimental/hybrid_scan_helpers.hpp`, `cpp/src/io/parquet/experimental/hybrid_scan_impl.*`, `cpp/src/io/parquet/experimental/hybrid_scan_multifile.cpp`	Adds flattened payload page ranges, source indices, dictionary-page ranges, offset-index validation, pruned-page representation, and a page-data chunking overload.
Sparse page preprocessing `cpp/src/io/parquet/experimental/hybrid_scan_preprocess.cu`, `cpp/src/io/parquet/reader_impl_preprocess_utils.`, `cpp/src/io/parquet/reader_impl.`, `cpp/src/io/parquet/io_utils/parquet_io_utils.cpp`	Validates sparse page spans, decodes page headers, tracks sparse I/O state, handles sparse dictionary pointers, and safely represents empty device buffers.
Sparse pass chunking and masks `cpp/src/io/parquet/experimental/hybrid_scan_chunking.cu`, `cpp/src/io/parquet/experimental/hybrid_scan_impl.*`, `cpp/src/io/parquet/experimental/page_index_filter.cu`	Pass setup accepts current data-page masks. Sparse mode initializes page metadata, builds masks from page spans, and accounts for sparse input sizes.
Multifile integration and validation `cpp/tests/io/experimental/hybrid_scan_multifile_composer.`, `cpp/tests/io/experimental/hybrid_scan_multifile_test.cpp`	Adds sparse multifile composition and coverage for empty masks, missing offset indexes, dictionary pages, asymmetric row-group ordering, and fully pruned payload data.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

rapidsai/cudf#23013: Modifies the hybrid-scan chunking APIs used for sparse page-level payload I/O.
rapidsai/cudf#23317: Modifies related payload byte-range and page-span APIs.
rapidsai/cudf#23374: Modifies related page pruning and sparse page I/O handling.

Suggested labels: cuIO, improvement

Suggested reviewers: simoneves, mattgara, vuule

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 10.26% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: adding page-level I/O and materialization support to Hybrid Scan.
Description check	✅ Passed	The description directly explains the page-level sparse I/O APIs, materialization support, and test coverage in the changeset.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

mhaseeb123 · 2026-07-31T23:40:05Z

-
-    // Must be called as soon as we create the pass
-    set_pass_page_mask(data_page_mask);
+    setup_next_pass(column_chunk_data, data_page_mask);


set_pass_page_mask is now moved inside the setup_next_pass

mhaseeb123 · 2026-07-31T23:40:42Z


    // Setup page information for the chunk (which we can access without decompressing)
-    setup_compressed_data(column_chunk_data);
+    if (_sparse_page_io) {


Use either the dense or sparse overloads of the setup_compressed_data and set_pass_page_mask APIs.

The sparse vs dense functions seem on the edge of similarity where it could make sense to use a template parameter and if constexpr instead of having separate functions. I'll defer to your judgment, just a thought.

True for setup_xx_compressed_data but not really for the setup_xx_page_mask. Will defer this to a follow up if the duplicated portion seems to be growing

mhaseeb123 · 2026-07-31T23:41:16Z

+   * @param schema_indices Schema indices from the first source
+   * @return A pair indicating column-index and offset-index presence, respectively
+   */
+  [[nodiscard]] std::pair<bool, bool> page_index_presence(


Simply moved from private to public scope

Part 1 of 3. Contributes to #23519 This PR improves Parquet reader's pass construction by making it column chunk aware so that it is more accurate (less conservative) when reading select columns from the file. Subsequent PRs will propagate these improvements to hybrid scan as well as enable hybrid scan to calculate at page-level (needs #23375) Authors: - Muhammad Haseeb (https://github.com/mhaseeb123) Approvers: - Bradley Dice (https://github.com/bdice) - Qi Chen (https://github.com/qbacpey) URL: #23446

vyasr

Some minor thoughts but nothing blocking, LGTM!

vyasr · 2026-08-18T17:22:27Z


    // Setup page information for the chunk (which we can access without decompressing)
-    setup_compressed_data(column_chunk_data);
+    if (_sparse_page_io) {


The sparse vs dense functions seem on the edge of similarity where it could make sense to use a template parameter and if constexpr instead of having separate functions. I'll defer to your judgment, just a thought.

mhaseeb123 added 2 commits July 21, 2026 17:39

Add page topology support for sparse reads

cf9226a

Preserve page locations and variable-width offset state needed to safely reconstruct columns from a sparse subset of Parquet data pages.

Add sparse page I/O to hybrid scan

f356b0d

Expose multifile page-range planning and consume the selected payload pages so hybrid scan avoids fetching pruned Parquet payload data.

mhaseeb123 requested a review from a team as a code owner July 21, 2026 17:43

mhaseeb123 requested review from mattgara and simoneves and removed request for a team July 21, 2026 17:43

github-actions Bot assigned mhaseeb123 Jul 21, 2026

github-actions Bot added the libcudf Affects libcudf (C++/CUDA) code. label Jul 21, 2026

mhaseeb123 marked this pull request as draft July 21, 2026 17:49

mhaseeb123 mentioned this pull request Jul 22, 2026

Enable hybrid scan page pruning for page level IO #23374

Merged

3 tasks

mhaseeb123 mentioned this pull request Jul 27, 2026

Improve Parquet reader pass construction #23446

Merged

3 tasks

mhaseeb123 changed the base branch from feature/sparse-page-io-foundation to main July 28, 2026 00:40

Merge branch 'main' into feature/sparse-page-io-hybrid

20508fb

VibhuJawa mentioned this pull request Jul 30, 2026

[FEA] Support the Parquet FILE logical type for external/blob payloads VibhuJawa/cudf#8

Open

mhaseeb123 added 2 commits July 30, 2026 14:14

Merge branch 'main' into feature/sparse-page-io-hybrid

b64685b

Merge branch 'main' into feature/sparse-page-io-hybrid

382ba5c

VibhuJawa mentioned this pull request Jul 31, 2026

[FEA] Support the Parquet FILE logical type for external/blob payloads #23494

Open

mhaseeb123 added feature request New feature or request 3 - Ready for Review Ready for review by team non-breaking Non-breaking change labels Jul 31, 2026

mhaseeb123 added 2 commits July 31, 2026 23:26

Simplify and cleanup

af6c466

Merge commit '3b3cea4d9669b9db1e35769ea727514eac3cb3e7' into feature/…

1cf8e93

…sparse-page-io-hybrid

mhaseeb123 force-pushed the feature/sparse-page-io-hybrid branch from ce70f6f to 1cf8e93 Compare July 31, 2026 23:26

mhaseeb123 marked this pull request as ready for review July 31, 2026 23:33

mhaseeb123 changed the title ~~Add sparse page I/O to hybrid scan~~ Add page-level (sparse) materializers in Hybrid Scan Jul 31, 2026

mhaseeb123 changed the title ~~Add page-level (sparse) materializers in Hybrid Scan~~ Add page-level I/O and materialization in Hybrid Scan Jul 31, 2026

mhaseeb123 commented Jul 31, 2026

View reviewed changes

coderabbitai Bot mentioned this pull request Aug 3, 2026

Migrate Thrust zip iterators to CUDA equivalents #23336

Merged

3 tasks

mhaseeb123 mentioned this pull request Aug 3, 2026

[STORY] Support page-level I/O in hybrid scan #23519

Open

Simplify further and add comments

7db024f

mhaseeb123 requested a review from qbacpey August 3, 2026 19:19

Merge branch 'main' into feature/sparse-page-io-hybrid

dbbaee0

mhaseeb123 requested a review from pmattione-nvidia August 4, 2026 02:51

qbacpey reviewed Aug 5, 2026

View reviewed changes

Comment thread cpp/src/io/parquet/experimental/hybrid_scan_preprocess.cu

mhaseeb123 commented Aug 5, 2026

View reviewed changes

Comment thread cpp/src/io/parquet/experimental/hybrid_scan_chunking.cu Outdated

coderabbitai Bot mentioned this pull request Aug 7, 2026

Fix uninitialized page info for column chunks without a dictionary page #23579

Merged

3 tasks

Update cpp/src/io/parquet/experimental/hybrid_scan_chunking.cu

d6a6f4c

mhaseeb123 requested a review from qbacpey August 7, 2026 21:59

mhaseeb123 removed the request for review from qbacpey August 17, 2026 22:46

Merge branch 'main' into feature/sparse-page-io-hybrid

1dec084

mhaseeb123 requested a review from a team as a code owner August 17, 2026 22:47

mhaseeb123 added 5 commits August 17, 2026 16:19

Merge branch 'main' into feature/sparse-page-io-hybrid

c7c05ae

Use cuda::stream_ref

b563b74

Minor bug fix

d202c71

Merge branch 'main' into feature/sparse-page-io-hybrid

06e4085

style fix

9c1eeb3

mhaseeb123 requested a review from qbacpey August 18, 2026 02:38

mhaseeb123 added 4 - Needs Review Waiting for reviewer to review or respond and removed 3 - Ready for Review Ready for review by team labels Aug 18, 2026

vyasr approved these changes Aug 18, 2026

View reviewed changes

mhaseeb123 added 2 commits August 18, 2026 20:36

Remove unnecessary column selection

37ee404

Merge branch 'main' into feature/sparse-page-io-hybrid

411ebd0

qbacpey approved these changes Aug 19, 2026

View reviewed changes

Merge branch 'main' into feature/sparse-page-io-hybrid

acd5be3

mhaseeb123 added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 4 - Needs Review Waiting for reviewer to review or respond labels Aug 19, 2026

Uh oh!

Conversation

mhaseeb123 commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

copy-pr-bot Bot commented Jul 21, 2026

Uh oh!

coderabbitai Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Summary by CodeRabbit

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

mhaseeb123 Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

mhaseeb123 Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

vyasr Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

mhaseeb123 Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

mhaseeb123 Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vyasr left a comment

Choose a reason for hiding this comment

Uh oh!

vyasr Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mhaseeb123 commented Jul 21, 2026 •

edited

Loading

coderabbitai Bot commented Jul 31, 2026 •

edited

Loading