Problem
An interrupted dft_stac_fetch() leaves a truncated .nc cache file at the canonical cache
path, and the next run treats it as a valid cache hit ("<year>: cached") rather than
re-fetching. The partial file then fails downstream — it is not a readable raster.
Observed (drift 0.6.0) during a large-floodplain run that was killed mid-download, then resumed:
2017: cached
Error: [mask] rasterization failed
In addition: Warning messages:
2: [rast] cannot open this file with the multidim API: ~/Library/Caches/drift/io-lulc/2017_3425b70e8413.nc
3: Cannot invert geotransform (GDAL error 1)
The 2017 .nc was ~3 MB and half-written (the fetch was killed during that year's download). On
resume, dft_stac_fetch() saw the file present, reported cached, and returned it; the corrupt
grid then broke terra::mask() / classification.
Root cause
Two gaps in the cache path:
- Non-atomic write — the cube is written directly to
<cache_dir>/<source>/<year>_<hash>.nc,
so a process killed mid-write leaves a partial file at the canonical name. The cache-hit
check is presence-only (file.exists), so a partial file is indistinguishable from a complete
one.
- No integrity check on read — a cached file is returned without verifying it opens as a valid
raster.
This is latent on fast/small AOIs (fetch rarely interrupted) and surfaces on long large-floodplain
fetches that get killed (idle sleep, job limits, caffeinate gaps). Recovery today is manual:
delete the specific <year>_<hash>.nc and re-run.
Proposed solution
- Atomic write: fetch to a temp path (
<name>.nc.tmp or tempfile() in the same dir) and
file.rename() onto the canonical name only after a successful, complete write. A killed fetch
then leaves at most a .tmp orphan that is never treated as a cache hit.
- (optional) Validate on read: before returning a cache hit, open it (e.g.
terra::rast() +
a cheap metadata/geotransform check); on failure, treat as a miss and re-fetch. Belt-and-braces
for files corrupted by other means.
- A
.tmp-sweep in dft_cache_clear() would tidy orphans.
Atomic write is the primary fix (prevents the bad state); validate-on-read is defense in depth.
Surfaced from the floodplains driver (repeated long bt-floodplain runs killed mid-fetch).
Problem
An interrupted
dft_stac_fetch()leaves a truncated.nccache file at the canonical cachepath, and the next run treats it as a valid cache hit (
"<year>: cached") rather thanre-fetching. The partial file then fails downstream — it is not a readable raster.
Observed (drift 0.6.0) during a large-floodplain run that was killed mid-download, then resumed:
The 2017
.ncwas ~3 MB and half-written (the fetch was killed during that year's download). Onresume,
dft_stac_fetch()saw the file present, reportedcached, and returned it; the corruptgrid then broke
terra::mask()/ classification.Root cause
Two gaps in the cache path:
<cache_dir>/<source>/<year>_<hash>.nc,so a process killed mid-write leaves a partial file at the canonical name. The cache-hit
check is presence-only (
file.exists), so a partial file is indistinguishable from a completeone.
raster.
This is latent on fast/small AOIs (fetch rarely interrupted) and surfaces on long large-floodplain
fetches that get killed (idle sleep, job limits,
caffeinategaps). Recovery today is manual:delete the specific
<year>_<hash>.ncand re-run.Proposed solution
<name>.nc.tmportempfile()in the same dir) andfile.rename()onto the canonical name only after a successful, complete write. A killed fetchthen leaves at most a
.tmporphan that is never treated as a cache hit.terra::rast()+a cheap metadata/geotransform check); on failure, treat as a miss and re-fetch. Belt-and-braces
for files corrupted by other means.
.tmp-sweep indft_cache_clear()would tidy orphans.Atomic write is the primary fix (prevents the bad state); validate-on-read is defense in depth.
Surfaced from the
floodplainsdriver (repeated long bt-floodplain runs killed mid-fetch).