Release uncertainty-aware evaluation as 3.0.0b12 - #140
Closed
zhongwangwei wants to merge 14 commits into
Closed
Conversation
Add opt-in aggregate bootstrap intervals, separate model/reference spread products, and conservative pairwise verdicts while preserving deterministic outputs. Constraint: Preserve existing deterministic output contracts and add no dependencies Rejected: Pooling simulation-reference pairs into one ensemble | conflates model spread with reference sensitivity Confidence: high Scope-risk: moderate Directive: Keep model and reference uncertainty axes separate in future extensions Tested: pytest -q (1612 passed, 5 skipped); ruff check src tests; python -m compileall -q src/openbench; git diff --check Not-tested: Production-scale HPC bootstrap runtime
Preserve paired time coordinates, sample only within contiguous valid segments, and share station timestamp normalization between evaluation and uncertainty processing. Constraint: Keep model/reference resampling paired and preserve existing deterministic outputs Rejected: Dropping missing timestamps before block sampling | falsely makes separated periods adjacent Confidence: high Scope-risk: moderate Directive: Treat valid_pair_count as data coverage, not an autocorrelation-adjusted effective sample size Tested: pytest -q (1613 passed, 5 skipped); ruff check src tests; python -m compileall -q src/openbench; GRDC 6-station RMSE smoke tests at block lengths 6, 12, and 24 Not-tested: Full production-scale GRDC/HPC runtime
Batch resamples with NumPy and preserve each contiguous segment's observed contribution while avoiding blocks that cross gaps. Constraint: Bound memory without nested parallelism or new dependencies Rejected: Process-level bootstrap parallelism | risks nested execution overhead and oversubscription Confidence: high Scope-risk: moderate Directive: Preserve stratified segment contribution and bounded batch allocation in future samplers Tested: pytest -q (1615 passed, 5 skipped); ruff check .; python -m compileall -q src/openbench/core/uncertainty.py; git diff --check; 2706-station RMSE benchmark with 1000 resamples at block lengths 6, 12, and 24 (1.882s, 1.520s, 1.313s) Not-tested: Multi-node HPC execution
Compute grid intervals from weighted per-cell metrics on common support, classify verdicts from interval evidence, reject non-resampleable series, and bind station/cache outputs to the current run. Constraint: Preserve deterministic evaluation outputs, add no dependencies, and bound grid-bootstrap working memory Rejected: Metric-after-spatial-mean aggregation | compensating grid errors can appear perfect Rejected: Directory-wide station scans | persistent output folders can contain stale runs Confidence: high Scope-risk: broad Directive: Keep paired grid verdicts on identical reference/support and never encode one-member spread as zero Tested: pytest -q (1624 passed, 5 skipped); ruff check .; python -m compileall -q src/openbench; git diff --check; 2706-station GRDC RMSE with 1000 resamples at blocks 6/12/24 (2.956s/3.344s/3.618s); synthetic 2000-cell x 120-step grid with 1000 resamples (4.693s) Not-tested: Full production-resolution grid evaluation or multi-node HPC execution
Constraint: Reference scan roots are UI state; explicit unified data_root overrides must still round-trip. Rejected: Infer Grid/MidRes in runtime adapter | guessing registry layout would preserve the GUI semantic bug. Confidence: high Scope-risk: moderate Directive: Keep _scan_root separate from explicit reference.data_root. Tested: ruff, compileall, targeted GUI/config/check tests, full pytest (1630 passed, 5 skipped) Not-tested: Manual Windows GUI launch
Constraint: CI enforces Ruff formatting across source and tests. Confidence: high Scope-risk: narrow Tested: Ruff check, Ruff format check, 69 GUI tests Not-tested: Full matrix; logic is unchanged.
Use the shared scanner for nested and direct-case roots, keep per-case model and file-pattern choices, and retain unchecked scan rows only as GUI state. Constraint: Unchecked discoveries must remain editable without entering exported runtime sources. Rejected: Store every discovery in source_configs | preview and remote staging treat source_configs as executable data. Confidence: high Scope-risk: moderate Directive: Keep local GUI discovery aligned with scan_simulation_roots and leave unresolved models explicit. Tested: Ruff check; Ruff format check on changed files; GUI compileall; pytest (1643 passed, 5 skipped). Not-tested: Live remote SSH scan and manual Windows GUI interaction.
Preserve per-case scanner metadata, save forward-navigation edits, and prepare station fulllists before local preview/export. Constraint: Mixed grid/station cases must retain detected metadata without shared defaults overwriting them. Rejected: Copy model-profile defaults into scanned cases | scanned NC metadata is more authoritative and station lists require materialization. Confidence: high Scope-risk: moderate Directive: Keep GUI scan/export behavior aligned with scan_simulation_roots and materialize_station_cases. Tested: Ruff check; changed-file format check; GUI compileall; pytest (1652 passed, 5 skipped); real station NetCDF export. Not-tested: Live remote station materialization and manual Windows GUI interaction.
Remove tuple-returning SMPI from selectable metrics, handle tied Mann-Kendall samples, align SMPI bootstrap filtering, validate categorical kappa input, and expose method assumptions. Constraint: Main must receive only fixes independent of the feature-only uncertainty workflow. Rejected: Expose SMPI as an ordinary metric | its tuple result belongs to the dedicated comparison contract. Confidence: high Scope-risk: moderate Directive: Keep SMPI comparison output separate from single-DataArray evaluation metrics. Tested: Targeted regression suites; full pytest (1658 passed, 5 skipped); Ruff; compileall; diff check. Not-tested: Manual GUI interaction.
Preserve valid point estimates for non-resampleable segments and lock all supported uncertainty metrics to the core metric results under pairwise NaNs. Constraint: This commit belongs only to the uncertainty-aware feature branch. Rejected: Route batched bootstrap calculations through xarray metrics | it would discard the existing vectorized NumPy execution path. Confidence: high Scope-risk: narrow Directive: Maintain parity tests whenever supported uncertainty metric formulas change. Tested: Uncertainty tests; full pytest (1658 passed, 5 skipped); Ruff; compileall; diff check. Not-tested: Production-scale distributed bootstrap workloads.
Resolve scanned simulation fulllist artifacts before serializing init and sim-scan YAML so runtime validation no longer depends on the caller's working directory. Constraint: Generated station-list locations must survive config execution outside the YAML directory. Rejected: Teach every fulllist consumer about YAML-relative paths | the generated path is shared across existing root-relative consumers and fails before those consumers can recover it. Confidence: high Scope-risk: narrow Directive: Preserve absolute fulllist output for both openbench init and openbench sim scan. Tested: 29 sim-scan/init tests; Ruff; git diff --check. Not-tested: Full cross-platform suite before this commit.
Release the opt-in uncertainty workflow together with absolute generated station-list paths and document the resulting package version consistently. Constraint: PyPI and GitHub releases require a version newer than published 3.0.0b11. Rejected: Publish the feature branch as beta 11 | PyPI distributions are immutable and beta 11 is already live. Confidence: high Scope-risk: moderate Directive: Keep src/openbench/__init__.py as the package version source of truth and mirror release labels in conda metadata and README. Tested: Full pytest (1658 passed, 5 skipped); Ruff; compileall; isolated build; Twine 7 check; wheel import smoke; archive content audit. Not-tested: Production-scale distributed uncertainty workloads.
Apply the repository's existing Ruff formatter to the four uncertainty files reported by the release PR lint job. Constraint: CI enforces ruff format --check across src and tests. Rejected: Relax the format check | these files require only mechanical formatting. Confidence: high Scope-risk: narrow Directive: Run both ruff check and ruff format --check before future releases. Tested: Ruff check; Ruff format check; 94 uncertainty and loader tests; git diff --check. Not-tested: Full suite after formatting-only changes.
Owner
Author
|
Closing per release-scope clarification: uncertainty stays on its feature branch, while PyPI 3.0.0b12 will be built from the normal mainline with only the station fulllist fix. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
fulllistvalues as absolute paths from bothopenbench initandopenbench sim scan3.0.0b12Why
The uncertainty workflow is ready for beta distribution, while generated station YAML could fail when executed from a working directory other than the config directory because its
fulllistpath was serialized relatively.Validation
pytest -q— 1658 passed, 5 skippedruff check .python -m compileall -q src tests