feat: support ResStock and multifamily building handling#32
Merged
Conversation
- Migrate pyproject.toml from Poetry to PEP 621 + uv (dependency-groups for dev deps, tool.uv package = false since this is an app, not a library). Drop the unneeded pathlib backport dependency (stdlib covers this on Python 3.12+). - Bump all runtime and dev dependencies to current stable majors (pandas 3.x, requests 2.34, geopandas 1.1, pyarrow 25, pytest 9, mypy 2.x, pre-commit 4.6, etc.) and regenerate uv.lock, replacing poetry.lock. - Fix a pandas 3.0 breaking change (GroupBy.apply no longer retains grouping columns) that was causing the notebook step to fail with KeyError: in.county_name on every dependency-bump CI run; switch to GroupBy.sample() which is version-safe and avoids the issue. - Fix mypy config: it pointed at a nonexistent your_module.py and used python_version = 3.8, so it was silently never checking the real code. Point it at comstock_processor.py/tests/, target 3.12, and relax strict annotations for tests. This surfaced a real bug: process_building_time_series() was annotated to return None but actually returns a tuple of lists; fixed the annotation and added missing type hints. - Connect all tests to CI: 7 of 10 tests in tests/test_comstock_processor.py had no unit/integration marker, so --strict-markers plus -m unit / -m integration in CI silently never ran them. Added the correct marker to every test so the full suite executes. - Update pre-commit config: bump ruff-pre-commit and nbstripout to latest, add astral-sh/uv-pre-commit's uv-lock hook to keep uv.lock in sync. - Update CI workflow to install/run everything through uv (astral-sh/setup-uv) instead of Poetry, which was also failing to install on the Python 3.9 matrix job; drop the unsupported 3.9 job (project requires Python >=3.12) in favor of 3.12/3.13. - Update README installation/testing/commit instructions for uv. Verified locally: uv run pytest -m unit, uv run pytest -m integration (real network downloads), uv run mypy, uv run pre-commit run --all-files, and a full notebook execution all pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Unpinned major-only v9 tag does not exist for astral-sh/setup-uv, only full semver tags like v9.0.0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The newer ruff-pre-commit (v0.15.22) reformats the notebook cell I edited after the last local pre-commit run, and warns that the S320 rule referenced in ruff.toml ignore list no longer exists in this ruff version. Fix both so pre-commit passes cleanly in CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ComStock is periodically republished with a new set of released data, and the metadata file layout/paths have changed across releases (release_1 used a single national metadata/baseline.parquet file; release_2 and release_3 partition metadata per state/county instead). This adds explicit, configurable support for the last three releases so callers aren't locked to a single hardcoded, increasingly stale release. - Add a SUPPORTED_RELEASES registry (release_1/2/3, mapping to comstock_amy2018_release_N) plus DEFAULT_RELEASE = release_3. All three currently resolve to the 2025 OEDI mirror, which republishes every release using the same metadata_and_annual_results/by_state_and_county partitioned layout, so one code path now handles all three releases uniformly. ComStockProcessor takes a new optional `release` kwarg (defaulting to the latest), validated against the registry with a clear ValueError otherwise. - Rewrite process_metadata() to discover the relevant state/county partitions via the public S3 list-objects-v2 API (stdlib xml.etree, no new dependency), download them in parallel with the same ThreadPoolExecutor/caching pattern used for time series files, concatenate, and apply the same county/building-type filters as before. Partition downloads and the final selected_metadata CSV are namespaced by release so switching releases doesn't collide with or reuse another release's cache. - process_building_time_series() needed no structural changes: the timeseries_individual_buildings/by_state layout is identical across all three releases. - Fix 01_data_sampling_example.ipynb, which broke against the new default release: the by_state_and_county metadata schema renamed several columns with explicit unit suffixes (e.g. in.sqft -> in.sqft..ft2, out.electricity.total.energy_consumption -> ...energy_consumption..kwh, electricity_bill_median..usd split into _median_high/_median_low, electricity_bill_number_of_rates -> electricity_bill_num_bills). Time series column names are unchanged. - Update tests: add unit tests for release validation/URL construction, parametrize metadata download/filter tests across all three releases, and gate the state="All" test behind TEST_DATA=true since "All" now means downloading every state's and county's partition files instead of a single cached national parquet. - Document the release parameter and supported releases in README.md. Note: this changes the previous hardcoded default from comstock_amy2018_release_1 (2024 flat layout) to release_3 (2025 partitioned layout) — existing callers that didn't pass `release` will now get the newest release's data/schema. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ntime Use a higher I/O-bound worker count (IO_WORKERS=16) for S3 listing and metadata partition downloads instead of the CPU-bound worker count used for time series downloads; on CI runners with few cores this previously limited metadata downloads to essentially one file at a time. Also parallelize the per-state county listing (previously sequential) for state="All". Bound the CI TEST_DATA integration step with timeout-minutes + continue-on-error, since state="All" now downloads a very large number of small files and should not be allowed to hang the job indefinitely. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… full download The previous TEST_DATA-gated test_all_state_filter genuinely downloaded every state's/county's metadata partition (thousands of files) when TEST_DATA=true, which took long enough in CI to hit the platform's workflow run limits. Mock ComStockProcessor._available_states down to two small states (DE, RI) so the test still exercises the real state='All' discover/download/ concatenate/filter code path, but stays fast and deterministic. This removes the need for the TEST_DATA skip and keeps the test in the regular integration suite. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Each ComStock "upgrade" is a different energy-efficiency measure package applied
to the same baseline building sample. Every release also publishes an
upgrades_lookup.json (upgrade id -> package name) and a measure_name_crosswalk.csv
(a stable measure_id crosswalked to the upgrade id/name used for that measure in
that release and earlier releases, since upgrade ids/measures are release-specific
and not stable across releases).
- Add ComStockProcessor.list_upgrades(save_dir) to download/cache
upgrades_lookup.json and return {upgrade_id: package_name} for the configured
release.
- Add ComStockProcessor.get_measure_crosswalk(save_dir) to download/cache
measure_name_crosswalk.csv as a DataFrame, and find_upgrade_id(save_dir,
measure_id, target_release=None) to look up the upgrade id for a stable
measure_id in a specific release, raising a clear error if that release isn't
covered by the currently loaded crosswalk (a release's crosswalk only covers
itself and earlier releases -- release_3's crosswalk covers all three
currently-supported releases).
- Refactor process_metadata() into a thin wrapper around a new
_download_metadata_for_upgrade(save_dir, upgrade) helper parameterized by
upgrade instead of hardcoded self.upgrade, and add
process_metadata_for_upgrades(save_dir, upgrades=None) which downloads and
combines metadata for multiple upgrades (defaulting to every upgrade from
list_upgrades()) into one DataFrame. Every partition already includes an
`upgrade` id and `in.upgrade_name` column, so grouping the combined result by
bldg_id lets you compare a building's results across packages.
- Add unit tests (release validation, mocked default-to-every-upgrade behavior)
and integration tests (list_upgrades, get_measure_crosswalk, find_upgrade_id
across releases, and process_metadata_for_upgrades with an explicit small
upgrade list for Delaware/SmallOffice).
- Document the new methods and a building-comparison usage example in
README.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds support for NREL's residential building stock dataset, ResStock, alongside
the existing ComStock support, and documents/handles ResStock's per-dwelling-unit
representation of multifamily buildings.
Background: ResStock is hosted on the same OEDI data lake as ComStock
(nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/{year}/resstock_*),
but differs in several ways: metadata is partitioned only by state (not
state+county), building types are residential housing categories (not commercial),
county names don't include a state prefix, and the measure crosswalk is an Excel
file, not csv. Multifamily buildings are represented at the *dwelling unit* level:
one metadata row is one simulated unit (not a whole building), with
weight/in.units_represented for population scaling and
in.geometry_building_number_units_mf/in.geometry_building_horizontal_location_mf/
in.geometry_building_level_mf describing the unit's context within its building.
- Extract a shared BuildingStockProcessor base class (building_stock_processor.py)
containing the S3 listing/downloading, upgrade package lookup
(list_upgrades), and time series download logic that is identical between
ComStock and ResStock. ComStockProcessor now subclasses it, keeping its own
state+county partitioned metadata logic and csv-based measure crosswalk.
BuildingStockRelease replaces the former ComStock-only ComStockRelease
dataclass (same shape, shared name); public symbols used elsewhere
(SUPPORTED_RELEASES, DEFAULT_RELEASE, ComStockProcessor) are unaffected.
- Add ResStockProcessor (resstock_processor.py), a new subclass with its own
SUPPORTED_RELEASES (release_1 -> resstock_amy2018_release_1; ResStock hasn't
been remastered across multiple releases into one consistent layout yet, unlike
ComStock, so only this one is supported for now), state-only partitioned
metadata download, RESSTOCK_BUILDING_TYPES validation (residential housing
categories including the two multifamily categories), county_name filtering
without a state prefix, and an xlsx-based get_measure_crosswalk/find_upgrade_id.
process_metadata_for_upgrades works the same as ComStockProcessor's, for
comparing a dwelling unit across measure packages.
- Add openpyxl as a dependency (needed to read ResStock's .xlsx measure
crosswalk).
- Add tests/test_resstock_processor.py mirroring the ComStock test suite:
unit tests for release/building-type validation and mocked
default-to-every-upgrade behavior, and integration tests for metadata
download/filtering/caching (including both multifamily categories and the
county_name-without-state-prefix format), list_upgrades, get_measure_crosswalk,
find_upgrade_id, process_metadata_for_upgrades, and time series downloading.
All existing ComStockProcessor tests still pass unchanged after the refactor.
- Document ResStockProcessor, its differences from ComStockProcessor, and how to
handle multifamily buildings in README.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Each ComStock "upgrade" is a different energy-efficiency measure package applied
to the same baseline building sample. Every release also publishes an
upgrades_lookup.json (upgrade id -> package name) and a measure_name_crosswalk.csv
(a stable measure_id crosswalked to the upgrade id/name used for that measure in
that release and earlier releases, since upgrade ids/measures are release-specific
and not stable across releases).
- Add ComStockProcessor.list_upgrades(save_dir) to download/cache
upgrades_lookup.json and return {upgrade_id: package_name} for the configured
release.
- Add ComStockProcessor.get_measure_crosswalk(save_dir) to download/cache
measure_name_crosswalk.csv as a DataFrame, and find_upgrade_id(save_dir,
measure_id, target_release=None) to look up the upgrade id for a stable
measure_id in a specific release, raising a clear error if that release isn't
covered by the currently loaded crosswalk (a release's crosswalk only covers
itself and earlier releases -- release_3's crosswalk covers all three
currently-supported releases).
- Refactor process_metadata() into a thin wrapper around a new
_download_metadata_for_upgrade(save_dir, upgrade) helper parameterized by
upgrade instead of hardcoded self.upgrade, and add
process_metadata_for_upgrades(save_dir, upgrades=None) which downloads and
combines metadata for multiple upgrades (defaulting to every upgrade from
list_upgrades()) into one DataFrame. Every partition already includes an
`upgrade` id and `in.upgrade_name` column, so grouping the combined result by
bldg_id lets you compare a building's results across packages.
- Add unit tests (release validation, mocked default-to-every-upgrade behavior)
and integration tests (list_upgrades, get_measure_crosswalk, find_upgrade_id
across releases, and process_metadata_for_upgrades with an explicit small
upgrade list for Delaware/SmallOffice).
- Document the new methods and a building-comparison usage example in
README.md.
Co-authored-by: Nicholas Long <nicholas.long@nrel.gov>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Base automatically changed from
feat/support-measure-package-comparison
to
feat/support-multiple-comstock-releases
July 24, 2026 01:15
…eleases' into fix-resstock-conflict # Conflicts: # comstock_processor.py
…download their time series (#33) Adds the ability to find a specific set of buildings by criteria beyond state/county/building_type -- e.g. "all office buildings under 10,000 sqft in the Denver metro area" -- and feed the result directly into process_building_time_series() to download time series only for those buildings. - county_name now accepts a list of county names (in addition to a single name or "All") in both ComStockProcessor and ResStockProcessor, so a metro area spanning multiple counties (e.g. Denver, Arapahoe, Jefferson, Adams, Douglas, Broomfield) can be queried in one call instead of requiring state="All" (over-fetching the whole state) or one call per county. - Add optional min_sqft/max_sqft constructor parameters to both processors, filtering on the in.sqft..ft2 column shared by ComStock and ResStock metadata. Implemented as a shared _apply_sqft_filter() helper on BuildingStockProcessor since the column name is identical across products. - Add scope_label()/sqft_label() helpers to building_stock_processor.py that build filesystem-safe, deterministic cache-filename components for the new county-list and sqft-range filters, so differently-scoped searches against the same state/building_type/upgrade don't collide on stale cached CSVs. Note: this changes the selected_metadata.csv cache filename format (adds a square-footage-range segment), so existing cached files from before this change will no longer be reused and will be regenerated. - process_metadata()'s docstring and the county-filtering logic in both processors now use isin() against the (possibly multi-value) county list instead of a single equality check. - Add tests/test_building_stock_processor.py for the new scope_label/ sqft_label/validate_release pure functions, plus integration tests in both existing test files for multi-county filtering, sqft filtering, cache filename uniqueness across filter combos, and the full "search metadata, then download time series for the matching buildings" workflow. - Document the new filters and workflow in README.md with a Denver-area example. Co-authored-by: Nicholas Long <nicholas.long@nrel.gov> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Conflicts: # README.md # comstock_processor.py # pyproject.toml # tests/test_comstock_processor.py # uv.lock
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for NREL's residential building stock dataset, ResStock, alongside the existing ComStock support, and handles ResStock's representation of multifamily buildings.
Background: where ResStock lives, and multifamily handling
ResStock is hosted on the same OEDI data lake as ComStock (
nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/{year}/resstock_*), but differs in a few important ways I confirmed by inspecting the bucket directly:state=DE/DE_upgrade0.parquet), not state+county like ComStock.Single-Family Detached,Multi-Family with 5+ Units, etc.), stored inin.geometry_building_type_recs.in.county_namehas no state prefix ("Kent County", not"DE, Kent County").measure_name_crosswalk_res_{year}_{release}.xlsx), not csv.resstock_amy2018_release_1has been remastered into the current, consistent 2025 layout so far — the newerresstock_amy2018_release_2still lives at its original (differently-structured) 2024 path, so it isn't supported yet.Multifamily buildings: ResStock simulates individual dwelling units, not whole buildings — a "Multi-Family with 5+ Units" metadata row is one apartment unit, not the building. There's no shared "building id" tying units back to a real building; each is an independently sampled, weighted record. Relevant columns:
in.geometry_building_number_units_mf(units in that unit's building),in.geometry_building_horizontal_location_mf/in.geometry_building_level_mf(the unit's position, affecting heat transfer with neighbors), andweight/in.units_represented(population scaling).What changed
BuildingStockProcessor(building_stock_processor.py): a shared base class with the S3 listing/downloading,list_upgrades(), andprocess_building_time_series()logic that's identical between ComStock and ResStock (same bucket, same timeseries layout, sameupgrades_lookup.jsonconvention).ComStockProcessornow subclasses it, unchanged behaviorally — all existing tests pass without modification.ResStockProcessor(resstock_processor.py):SUPPORTED_RELEASES = {"release_1": ...}(only the one confirmed-working layout, for now).RESSTOCK_BUILDING_TYPES— the 5 validin.geometry_building_type_recsvalues, validated in the constructor.process_metadata()/process_metadata_for_upgrades()(simpler than ComStock's, no county sub-listing needed).county_namefiltering without the state prefix,get_measure_crosswalk()/find_upgrade_id()adapted to the xlsx format.openpyxldependency (to read the xlsx crosswalk).tests/test_resstock_processor.pymirrors the ComStock test suite — release/building-type validation, mocked default-upgrades behavior, and integration tests for metadata download/filtering (including both multifamily categories and county-name-without-prefix), caching,list_upgrades,get_measure_crosswalk,find_upgrade_id,process_metadata_for_upgrades, and time series downloads.Validation
uv run pytest tests/ -m unit✅ (16 passed — release/building-type validation, mocked behaviors, for both ComStock and ResStock)uv run pytest tests/ -m integration✅ (27 passed, ~1 min — full ComStock suite unchanged + new ResStock metadata/multifamily/crosswalk/timeseries tests for Delaware)uv run mypy✅ (6 source files)uv run pre-commit run --all-files✅find_upgrade_id, and time series downloads all work end-to-end against the live OEDI bucket