Skip to content

Bound dft_stac_cube() streaming to the AOI with opt-in tile_size (#38) - #40

Merged
NewGraphEnvironment merged 8 commits into
mainfrom
38-bound-dft-stac-cube-streaming-to-the-aoi
Jul 12, 2026
Merged

Bound dft_stac_cube() streaming to the AOI with opt-in tile_size (#38)#40
NewGraphEnvironment merged 8 commits into
mainfrom
38-bound-dft-stac-cube-streaming-to-the-aoi

Conversation

@NewGraphEnvironment

@NewGraphEnvironment NewGraphEnvironment commented Jul 12, 2026

Copy link
Copy Markdown
Owner

Summary

  • Adds an opt-in tile_size to dft_stac_cube() (Bound dft_stac_cube() streaming to the AOI (fetch-time read still bbox-bound after #32) #38) that bounds the STAC read to the AOI footprint — the continuous-path twin of Tile dft_stac_fetch to bound download over sparse-floodplain bounding boxes #36's dft_stac_fetch(tile_size=). By default one gdalcubes cube is streamed over the whole AOI bounding box, so for a thin, diagonal floodplain corridor (area/bbox ≈ 0.1) the COG streaming — the dominant cost (~10–30 min for a multi-year monthly Sentinel-2 fetch) — is ~10× larger than the polygon needs. When tile_size is set, the bbox is split into a res-aligned grid and only tiles intersecting the AOI polygon are streamed (each carrying the full SCL mask + spectral index + 2022 baseline-offset split), then mosaicked. This is the filter_geom-independent path (the in-cube polygon clip segfaults on the pinned gdalcubes build).
  • Default tile_size = NULL is byte-for-byte the previous behavior: a frozen golden-hash guardian proves the untiled cache key is unchanged, so existing cube_*.tif stay valid. The refactor into an assemble_index_stack(extent) closure routes the untiled path over the bbox extent unchanged; the tiled path maps it over the tiles and merges with a new mosaic_stacks().
  • Honest grid caveat (surfaced by running the network e2e): because the cube resamples with bilinear, a tiled cube faithfully reproduces the untiled cube (bilinear-aligned correlation ≈ 0.997, per-layer means within ~1e-3, no tile seams) but lands on a bbox-anchored grid that is sub-pixel-offset from — not pixel-identical to — the untiled cube (gdalcubes enlarges the untiled bbox extent symmetrically). The offset is immaterial to the per-pixel dft_rast_break() / dft_rast_trend() reducers. Documented in NEWS, the roxygen, and inst/notes/gdalcubes-pc-gotchas.md.

Verification

  • Offline (CI): devtools::test() green (365 pass / 6 skip); devtools::check() clean; lintr clean. Offline oracles cover the cache-key guardian, multi-layer mosaic_stacks merge, the cover-then-merge == merge-then-cover commutativity (offset split under tiling), and the clip = FALSE tile-union extent.
  • Network (opt-in, DRIFT_TEST_NETWORK=true): ran live against the Planetary Computer. It caught two real issues — a terra::nlyr() double/integer vapply template bug (fixed), and that tiled/untiled cubes are not co-lattice (which made pixel-identity the wrong assertion). Both resolved; the test now asserts grid-robust equivalence (bilinear-aligned correlation + per-layer means), with thresholds measured on real fetched cubes with headroom.

Related Issues

Test plan

  • devtools::test() — 365 pass / 6 skip / 0 fail
  • devtools::check() — 0 errors / 0 warnings / 0 notes
  • lintr::lint_package() — R/ source clean
  • Network e2e validated against live COGs (opt-in; skips in CI)

🤖 Generated with Claude Code

https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz

NewGraphEnvironment and others added 8 commits July 11, 2026 16:33
@NewGraphEnvironment @claude
Initialize PWF baseline for #38
8c29580 
Read-side tiling for dft_stac_cube() (tile_size opt-in), mirroring #36 on the
categorical dft_stac_fetch() path. Phases + Plan-agent-vetted findings scaffolded.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
@NewGraphEnvironment @claude
Add tile_size to cube cache key with frozen-hash guardian (#38)
90f9d93 
Phase 1 of read-side tiling for dft_stac_cube(). Migrate stac_cube_cache_key()
to the append-only shape from #36's stac_cache_key(): a new trailing
tile_size = NULL is appended to the hash ONLY when non-NULL, so an untiled cube
keeps the exact legacy 18-element hash and existing cube_<key>.tif stay valid.

A golden-hash guardian test (cube_key() == "638a2be11fdf") was authored BEFORE
the refactor and still passes, proving byte-for-byte preservation of the untiled
key. New tests cover tile_size distinctness and snap-before-key (504 -> 500 at
res 10 -> same key). The caller does not pass tile_size yet (defaults NULL); the
param + tiled read land in Phase 3, so there is no distinct-key-but-bbox-read
window on the branch.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
@NewGraphEnvironment @claude
Add mosaic_stacks() with offline merge/commutativity/extent oracles (#38
899ebe4 
)

Phase 2 of read-side tiling for dft_stac_cube(). mosaic_stacks() is the
reassembly step of the tiled read path: an in-memory, multi-layer
terra::merge(terra::sprc(stacks)) that reads per-tile monthly index stacks
(each possibly the terra::cover() of a pre/post offset split) back into one
raster. Distinct from the fetch sibling's file-based, single-layer
mosaic_tiles().

Three offline oracles run in CI (no network): a multi-layer merge oracle
(res-aligned tiles reassemble losslessly, nlyr + layer order preserved); a
cover-then-merge == merge-then-cover commutativity test (proves the 2022
offset-split commutes with spatial tiling without needing the network e2e); and
a tile-union extent test on the packaged diagonal AOI (n_kept 31 < n_full 49,
NA gaps where empty tiles are skipped) that pins the clip = FALSE + tile_size
narrowing semantics.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
@NewGraphEnvironment @claude
Tile the dft_stac_cube() read with opt-in tile_size (#38)
39ba02c 
Phase 3 of read-side tiling. dft_stac_cube() gains tile_size (default NULL): when
set, the AOI bbox is split into a res-aligned grid and only tiles intersecting
the AOI polygon are streamed, then mosaicked with mosaic_stacks() — so a sparse
corridor reads near its footprint instead of the full bounding box. This is the
filter_geom-independent, continuous-path twin of #36 (the polygon clip that would
do this in-cube segfaults on the pinned gdalcubes build).

The cube_view construction + the 2022 offset split + terra::cover are refactored
into a local closure assemble_index_stack(extent) (build_index_stack gains a v
param); the untiled path routes through it over the bbox extent, so its output is
unchanged (fresh-eyes review confirmed behavior-preservation). The tiled path
maps it over the tiles, guards uniform nlyr, and merges. tile_size is normalized
once via #36's tile_size_check() so the is.null() path gate and the cache-key
append share one scalar; the cube already caches .tif and already sets the GDAL
/vsicurl tuning, so no extension routing or config change is needed.

Roxygen documents tile_size and the clip = FALSE + tile_size contract (the
returned extent is the AOI-intersecting tile union, with NA where empty tiles
were skipped, not a gap-free bounding box).

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
@NewGraphEnvironment @claude
Add tiled-vs-untiled network e2e; fix nlyr guard the live run caught (#…
33e58aa 
…38)

Phase 4. Ran the opt-in network e2e against the Planetary Computer — it caught
two things the offline oracles structurally cannot (they never invoke the
network-bound dft_stac_cube):

1. terra::nlyr() returns a double, so the uniform-nlyr guard's vapply template
   integer(1) errored. Fixed to numeric(1). The tiled fetch itself worked — all
   12 tiles streamed, offset-split, and covered without error; only the strict
   guard tripped, after assembly.

2. The tiled and untiled cubes are not co-lattice: gdalcubes enlarges the untiled
   bbox extent symmetrically to align with dx/dy (~0.5px), while tiles anchor at
   the bbox lower-left, so they cannot be compared pixel-for-pixel. Confirmed
   offline on saved real cubes that this is a benign grid offset, not a bug:
   after bilinear alignment cor = 0.997, median |diff| = 3.4e-3, per-layer means
   agree to 6e-4, and mean |diff| at tile seams == interior (no seam artifacts —
   gdalcubes reads the source margin at tile edges). The tiled cube is a faithful
   resampling of the source; downstream reducers are per-pixel, so the offset is
   immaterial.

The network test is rewritten accordingly: a growing-season window for robust
valid-pixel coverage, and equivalence asserted via bilinear alignment
(cor > 0.98, median |diff| < 0.01) plus grid-independent per-layer means
(< 0.01) — thresholds measured on the real cubes with headroom — instead of the
flawed pixel-identity comparison. Offset-split-under-tiling stays covered by the
offline commutativity oracle. Tile anchoring stays at bbox-LL (re-anchoring to
the gdalcubes-enlarged origin would couple to gdalcubes internals and change the
shared #36 helper for no scientific gain).

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
@NewGraphEnvironment @claude
Document tile_size for dft_stac_cube: NEWS, gotchas note, roxygen (#38)
723c020 
Phase 5 docs. NEWS 0.7.0 section describes the opt-in read-bounding tile_size,
its filter_geom-independence, and the honest caveat that a tiled cube faithfully
reproduces the untiled cube (cor ~0.997, no seams) but lands on a bbox-anchored
grid sub-pixel-offset from it. The gotchas note flips the #38 read residual to
resolved and adds the bilinear-tiling lesson (not co-lattice with untiled; no
tile seams since gdalcubes reads the source margin at edges; compare via bilinear
alignment, never pixel-for-pixel; why re-anchoring tiles was rejected). Roxygen
@param tile_size gains the same grid-offset note.

Version bump held for the final commit after devtools::check.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
@NewGraphEnvironment @claude
Release v0.7.0
d7a5094 
Read-side tiling for dft_stac_cube() (opt-in tile_size, #38). devtools::check()
clean (0 errors / 0 warnings / 0 notes); 365 tests pass / 6 skip.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
@NewGraphEnvironment @claude
Archive planning files for issue #38
1a7c2c7 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
@NewGraphEnvironment
NewGraphEnvironment merged commit bb896ff into main Jul 12, 2026
1 check passed
@NewGraphEnvironment
NewGraphEnvironment deleted the 38-bound-dft-stac-cube-streaming-to-the-aoi branch July 12, 2026 01:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Bound dft_stac_cube() streaming to the AOI (fetch-time read still bbox-bound after #32)

1 participant

@NewGraphEnvironment