Skip to content

dft_stac_fetch: interrupted fetch leaves a corrupt .nc cache that is trusted as a hit on resume #41

Description

@NewGraphEnvironment

Problem

An interrupted dft_stac_fetch() leaves a truncated .nc cache file at the canonical cache
path
, and the next run treats it as a valid cache hit ("<year>: cached") rather than
re-fetching. The partial file then fails downstream — it is not a readable raster.

Observed (drift 0.6.0) during a large-floodplain run that was killed mid-download, then resumed:

  2017: cached
Error: [mask] rasterization failed
In addition: Warning messages:
2: [rast] cannot open this file with the multidim API: ~/Library/Caches/drift/io-lulc/2017_3425b70e8413.nc
3: Cannot invert geotransform (GDAL error 1)

The 2017 .nc was ~3 MB and half-written (the fetch was killed during that year's download). On
resume, dft_stac_fetch() saw the file present, reported cached, and returned it; the corrupt
grid then broke terra::mask() / classification.

Root cause

Two gaps in the cache path:

  1. Non-atomic write — the cube is written directly to <cache_dir>/<source>/<year>_<hash>.nc,
    so a process killed mid-write leaves a partial file at the canonical name. The cache-hit
    check is presence-only (file.exists), so a partial file is indistinguishable from a complete
    one.
  2. No integrity check on read — a cached file is returned without verifying it opens as a valid
    raster.

This is latent on fast/small AOIs (fetch rarely interrupted) and surfaces on long large-floodplain
fetches that get killed (idle sleep, job limits, caffeinate gaps). Recovery today is manual:
delete the specific <year>_<hash>.nc and re-run.

Proposed solution

  • Atomic write: fetch to a temp path (<name>.nc.tmp or tempfile() in the same dir) and
    file.rename() onto the canonical name only after a successful, complete write. A killed fetch
    then leaves at most a .tmp orphan that is never treated as a cache hit.
  • (optional) Validate on read: before returning a cache hit, open it (e.g. terra::rast() +
    a cheap metadata/geotransform check); on failure, treat as a miss and re-fetch. Belt-and-braces
    for files corrupted by other means.
  • A .tmp-sweep in dft_cache_clear() would tidy orphans.

Atomic write is the primary fix (prevents the bad state); validate-on-read is defense in depth.

Surfaced from the floodplains driver (repeated long bt-floodplain runs killed mid-fetch).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions