Add chunk_index to the Band schema - #1145
Conversation
A new nullable List<Int64> field on Band: one entry per dim_names entry, the zero-based block coordinate this band occupies along that axis within a larger logical array it's one chunk of -- a chunk number, not a pixel offset, so it stays meaningful across uneven/remainder chunk sizes. Nullable at the schema level, same as crs/transform on Raster today: unset for a band's ordinary role inside Raster.bands (addressed by index/name, not block coordinate), required and validated by whichever accessor or function actually depends on it for a standalone chunked-table (DataArray-style) use case. Threaded through RasterBuilder (StartBandArgs.chunk_index, validated against dim_names.len()), RasterStructArray/BandRef (a new chunk_index() accessor), and BandRef::copy_into (inherited by default via BandOverrides.chunk_index, like nodata/outdb_uri/outdb_format already are -- a chunk's position should survive being separated from the row it started in, e.g. through a derive operation, not just through UNNEST). All ~30 existing StartBandArgs construction sites are unaffected: every one already uses ..StartBandArgs::new(...) struct-update syntax, so the new field's None default requires no changes there. Verified directly by compiling the whole workspace, not assumed.
|
One issue with this is: how do we ensure that the chunk index is not lying? SedonaDB internals typically see one raster a time, but an application using other columns can make sure that it's only working with a table that is a raster. From SedonaDB's end, we can't typically know if somebody has unioned this with some other raster table. (Feel free to correct me if I'm missing something here). Broadly this gets at the higher level question: is a raster a row, is a raster a column, or is a raster a table? Or do rasters participate in relational algebra at all? (i.e., should we always just keep Raster objects as Raster objects and write our own ExecutionPlans for these ops if we need to play games like this to get acceptable performance?) |
|
Really good point Dewey and a big part of the reason I haven't flipped from draft; even in the closed ecosystem of SedonaDB I'm worried UDF implementations wont maintain the chunk indexes correctly. There are also the philosophical questions you pose and their pragmatic implications. Can this one type really effectively modelboth cases: where a row is a raster and where a column is a raster. |
2 participants
What
Adds
chunk_index: List<Int64> NULLABLEto the Band schema: one entry perdim_namesentry, the zero-based block coordinate this band occupies along that axis within a larger logical array it's one chunk of — a chunk number, not a pixel offset, so it stays meaningful across uneven/remainder chunk sizes.Nullable at the schema level, same as
crs/transformonRastertoday: unset for a band's ordinary role insideRaster.bands(addressed by index/name, not block coordinate), required and validated by whichever accessor or function actually depends on it for a standalone chunked-table (DataArray-style) use case — a chunked table with one row per block, where each chunk currently has nowhere to carry its own position except external sibling columns the caller has to build and maintain by hand.Why
A chunk keeps its own block coordinate wherever it goes — through a
SELECT, out of a function's return value, across a join, eventually throughUNNEST— instead of losing it the moment it's separated from the specific row it started in. Carrying this on the value itself, rather than requiring a consumer to invent and maintain separatedim_0_idx/dim_1_idx-style columns alongside it, means the block-coordinate alignment a chunked-array join needs can match directly onchunk_indexwith no dependency on external bookkeeping.What's in it
RasterSchema: newchunk_index_type(), the field appended toband_type(),column::CHUNK_INDEX/band_indices::CHUNK_INDEX,band_indices::FIELD_COUNTbumped 9→10.RasterBuilder:StartBandArgs.chunk_index(validated againstdim_names.len()), threaded throughstart_band/finishfollowing the same values+offsets+validity pattern already used forsource_shape/view.BandRef: newchunk_index()accessor (defaultNone), a real implementation in the Arrow-backedBandRefImpl.BandRef::copy_into:BandOverrides.chunk_index, inherited by default (overrides.chunk_index.or_else(|| self.chunk_index())) — same pattern asnodata/outdb_uri/outdb_format— so a chunk's position survives a derive operation, not just a raw copy.All ~30 existing
StartBandArgsconstruction sites are unaffected: every one already uses..StartBandArgs::new(...)struct-update syntax, so the new field'sNonedefault requires no changes there — verified directly by compiling the whole workspace, not assumed.Verification
copy_into's inherit-by-default + explicit-override paths.cargo testforsedona-schema,sedona-raster,sedona-raster-functions,sedona-raster-zarr,sedona-spatial-join-raster,sedona-testing— all pass, no regressions.cargo clippy --all-targets -- -D warningsandcargo fmt --all -- --check— clean.cargo test(excluding the environment-onlygdal-sys/sedona-gdalbindgen issue on this machine, unrelated to this change) — clean.