feat: integrate earthaccess for cloud-native NASA Earthdata streaming

[drnimbusrain]

Motivation

CECE's current data pipeline requires every input stream (LAI, meteorology, soil properties, PAR, etc.) to be pre-staged as local NetCDF files before a run can start. This creates three compounding operational burdens:

1. Users must independently identify the correct NASA dataset identifiers, versions, and temporal/spatial extents for each physics scheme.
2. Files must be downloaded, renamed into the expected directory layout, and kept in sync with the run window.
3. On HPC systems with limited scratch quotas this is frequently the bottleneck that delays or blocks science runs.

earthaccess is a NASA-supported Python library that authenticates against NASA Earthdata Login (EDL) once and returns fsspec file-like objects backed by S3 (in-region) or HTTPS (anywhere). netCDF4/h5netcdf and xarray can read these objects identically to local files — zero bytes hit the local drive unless the user explicitly asks for a download.

This PR adds a thin Python-layer bridge that lets any cece_data stream declare source: earthaccess, offloading granule discovery and credentialed streaming to earthaccess while leaving the C++ core and AMIO path completely unchanged.

---

Architecture

YAML config
  └─ source: earthaccess stream entry
       │  (short_name, temporal, bounding_box, variables map)
       ▼
src/python/config.py  ─ _from_dict()
       │  routes earthaccess entries to _cece_data["earthaccess_streams"]
       │  (AMIO file-path streams continue unchanged)
       ▼
src/python/earthaccess_resolver.py
  EarthAccessStreamResolver
       │  earthaccess.login()       → NASA EDL auth (env / .netrc / interactive)
       │  earthaccess.search_data() → CMR granule list
       │  earthaccess.open()        → list of fsspec file objects (no download)
       │  xr.open_mfdataset(engine="h5netcdf") → lazy xr.Dataset
       ▼
src/python/stream_bridge.py
  EarthAccessStreamBridge.inject_at_time(import_state, t)
       │  ds.sel(time=t, method="nearest") → per-timestep slice
       │  np.asfortranarray(..., dtype=float64)
       │  import_state.set_field(cece_field, arr)
       ▼
existing pybind11 CeceImportState  →  Kokkos views  →  physics routines

The pybind11 set_field injection path already exists; this PR adds nothing to the C++ layer.

---

Files Changed

New files

File	Purpose
src/python/earthaccess_resolver.py	EarthAccessStreamConfig dataclass (CMR search params) and EarthAccessStreamResolver (auth → search → open → xr.Dataset)
src/python/stream_bridge.py	EarthAccessStreamBridge — per-timestep nearest-time slice and field injection into CeceImportState
examples/cece_config_earthaccess.yaml	Worked example running MEGAN3 + BDSNP + Fengsha from six live NASA streams

Modified files

File	Change
src/python/config.py	_from_dict routes source: earthaccess streams to _cece_data["earthaccess_streams"]; adds earthaccess_streams property and parse_earthaccess_streams() module-level helper
src/python/__init__.py	Exports EarthAccessStreamConfig, EarthAccessStreamResolver, EarthAccessStreamBridge, parse_earthaccess_streams
pyproject.toml	Adds [cloud] optional-dependency group (earthaccess, xarray, h5netcdf, fsspec, s3fs)

---

Example: cece_config_earthaccess.yaml

Demonstrates running MEGAN3, BDSNP, and Fengsha driven entirely by live NASA Earthdata Cloud streams — no local NetCDF files required.

One-time credential setup

# Option A: environment variables (recommended for CI / containers / batch)
export EARTHDATA_USERNAME=<your-EDL-username>
export EARTHDATA_TOKEN=<your-EDL-token>

# Option B: store in ~/.netrc (earthaccess "netrc" strategy)
# machine urs.earthdata.nasa.gov login <user> password <pass>

Install cloud extras

pip install 'cece-tools[cloud]'

Stream-to-physics-scheme mapping

Stream	NASA dataset	DAAC	CECE fields	Physics scheme(s)
modis_lai	MCD15A2H v061	LPDAAC_ECS	leaf_area_index	MEGAN3
smap_soil	SPL4SMGP	NSIDC_CPRD	soil_moisture_root, soil_moisture	MEGAN3, BDSNP
smap_soiltemp	SPL4SMGP	NSIDC_CPRD	soil_temperature	BDSNP
ceres_par	CER_SYN1deg-Day	LARC_ASDC	par_direct, par_diffuse, solar_cosine	MEGAN3
era5_wind	ERA5_SURFACE_1HR	GES_DISC	u_wind, v_wind	Fengsha
soil_texture	MCD12Q1 v061	LPDAAC_ECS	land_use_type	Fengsha, BDSNP

Key config section

cece_data:
  streams:
    - name: modis_lai
      source: earthaccess          # ← triggers cloud path; no file: key needed
      short_name: MCD15A2H
      version: "061"
      daac: LPDAAC_ECS
      cloud_hosted: true
      temporal_start: "2022-06-25"   # extra window for prev-LAI interpolation
      temporal_end:   "2022-07-04"
      variables:
        Lai_500m: leaf_area_index

    - name: smap_soil
      source: earthaccess
      short_name: SPL4SMGP
      daac: NSIDC_CPRD
      cloud_hosted: true
      temporal_start: "2022-07-01"
      temporal_end:   "2022-07-04"
      variables:
        sm_rootzone: soil_moisture_root
        sm_surface:  soil_moisture

    - name: ceres_par
      source: earthaccess
      short_name: CER_SYN1deg-Day_Terra-Aqua-MODIS_Edition4A
      daac: LARC_ASDC
      cloud_hosted: true
      temporal_start: "2022-07-01"
      temporal_end:   "2022-07-04"
      variables:
        sfc_sw_down_dir_all_1h: par_direct
        sfc_sw_down_dif_all_1h: par_diffuse
        solar_zenith_angle:     solar_cosine

Python usage (pybind11 path)

import cece
from datetime import datetime, timedelta

config = cece.load_config("examples/cece_config_earthaccess.yaml")

# Opens all remote streams once — no data downloaded yet
bridge = cece.EarthAccessStreamBridge(config.earthaccess_streams)

cece.initialize(config)
state = cece.CeceState(nx=1440, ny=720, nz=1)

t = datetime(2022, 7, 1)
while t <= datetime(2022, 7, 3, 23):
    bridge.inject_at_time(state.import_state, t)   # streams slice on demand
    cece.compute(state, config)
    t += timedelta(hours=1)

---

Design decisions

Decision	Rationale
Python-only bridge; C++ core untouched	Avoids any risk to production builds; set_field pybind11 injection path already exists
engine="h5netcdf"	Only xarray engine that accepts file-like objects for HDF5/NetCDF4; scipy requires seekable local paths
earthaccess.open() not download()	Bytes fetched on-demand per slice; nothing written to disk
source: earthaccess opt-in key	Existing AMIO/local-file configs work with zero changes
[cloud] optional-dep group	Base install stays lean; HPC users with pre-staged files need nothing new
np.asfortranarray before set_field	Matches Kokkos LayoutLeft (column-major) convention used by all CECE import state fields

---

Open items / follow-up

• Validate short_name against CMR at config-parse time (warn before run starts rather than at first timestep)
• Auto-derive bounding_box from driver grid extents when not set explicitly
• Evaluate earthaccess.virtualize() (VirtualiZarr / DMR++) as an alternative to open_mfdataset for large granule counts
• Add block_size / cache_type fsspec tuning knobs to EarthAccessStreamConfig
• Unit tests: mock earthaccess.open() with local fixture files to allow CI without live EDL credentials

---

Tests added (tests/test_earthaccess_stream_bdsnp_megan3.py)

Result: 37 passed, 5 skipped, 0 failures (run: pytest tests/test_earthaccess_stream_bdsnp_megan3.py -v)

All tests run without NASA EDL credentials and without a CECE build. The 5 skipped tests are annotated: 2 require the pybind11 build (requires_cece_core), 3 require live EDL credentials (live_earthdata).

Test classes

Class	Tests	Grid	Scope
TestEarthAccessStreamConfig	4	—	Dataclass construction, defaults, variable map, bounding box
TestParseEarthAccessStreams	4	—	parse_earthaccess_streams() — empty, AMIO-only, mixed, earthaccess-only
TestCeceConfigEarthAccessRouting	3	—	_from_dict sends source:earthaccess to earthaccess_streams; AMIO streams untouched
TestEarthAccessStreamResolverMocked	2	—	open_as_xarray() with mocked earthaccess; no-granule RuntimeError
TestEarthAccessStreamBridgeMocked	4	—	inject_at_time() populates state; Fortran-contiguous; float64; multi-stream
TestBDSNPFieldInjectionOnHemcoGrid	5 + 1 skipped	72×46 HEMCO 4°×5°	Soil temperature and moisture fields present, shape (46, 72), T∈[200–340 K], SM∈[0–1], 24-h timestep advance. Mirrors PR #85 parity grid.
TestMEGAN3FieldInjectionOnHemcoGrid	8 + 1 skipped	72×46 HEMCO 4°×5°	All 5 MEGAN3 fields present; 5 parametrized physical-range checks (LAI, soil moisture, PAR direct/diffuse, solar cosine); shape (46, 72); Fortran-contiguous; 5-timestep advance. Mirrors PR #90 parity grid.
TestCombinedBdsnpMegan3Injection	1	72×46 HEMCO 4°×5°	All 7 BDSNP+MEGAN3 fields simultaneously from 3 streams in one inject_at_time call
TestEarthAccessImportError	1	—	Helpful ImportError + pip install 'cece-tools[cloud]' hint when cloud extras absent
TestExampleConfigParsing	4	—	tests/cece_config_earthaccess_4x5_test.yaml round-trip: file exists, parses, yields earthaccess streams, has both bdsnp and megan3 schemes
TestLiveEarthDataIntegration	3 skipped	—	CMR granule search + fsspec open smoke test (SMAP + MODIS LAI). Run with: pytest -m live_earthdata

New test config: tests/cece_config_earthaccess_4x5_test.yaml

Matches the exact HEMCO 4°×5° reference grid from PR #85 (SoilNOx parity) and PR #90 (MEGAN isoprene parity):

• Grid: 72 lon × 46 lat, 5° longitude spacing, ±89° polar centres
• Run window: 2022-07-01 12:00–13:00 UTC (one 3600-s step)
• Streams: MODIS MCD15A2H (LAI) + SMAP SPL4SMGP (soil temperature + moisture) + CERES SYN1deg (PAR)
• Schemes: megan3 + bdsnp (soil_no_method: hemco_3_12_1)

Cross-over with PRs #85 and #90

The BDSNP grid tests assert input field shape (46, 72) and physical ranges matching the HEMCO 3.12.1 MERRA-2 soil inputs used in PR #85. The MEGAN3 grid tests assert all six import fields used by the PR #90 parity simulation are present, correctly shaped, and within physical range before reaching the C++ kernel. The combined injection test verifies all schemes can be fed simultaneously from a single inject_at_time call — the same execution pattern as a live CECE driver timestep.

Source fixes required for standalone test loading

• src/python/stream_bridge.py: try/except on relative from .earthaccess_resolver import so the module loads without package context in the test runner
• src/python/config.py: same try/except on from .earthaccess_resolver import EarthAccessStreamConfig

Running

# Install test deps (no NASA credentials needed)
pip install 'cece-tools[cloud]' pytest

# All CI-safe tests
pytest tests/test_earthaccess_stream_bdsnp_megan3.py -v

# Include live EDL smoke tests
export EARTHDATA_USERNAME=<user>
export EARTHDATA_TOKEN=<token>
pytest tests/test_earthaccess_stream_bdsnp_megan3.py -v -m live_earthdata

[bbakernoaa]

I like the idea of this. I think we should also build in methods utilizing grib2io (followup PR) to directly stream met data from NOAA forecasts.