Correct tank energy scale (2.5x) and reversed field semantics

[eman]

Summary

Two independent bugs in the tank energy fields, plus docs that described them three mutually incompatible ways.

The functional change is one line: 10.0 → 4.0. Everything else is the rename it forces, the doc corrections, and tests.

On a 65-gallon unit the CLI goes from Total Capacity 15800.0 Wh / Available Capacity 11660.0 Wh to Full Recovery 6320.0 Wh / Energy to Setpoint 4664.0 Wh.

1. Unit scale

totalEnergyCapacity / availableEnergyCapacity were multiplied by 10, assuming the device reports 10 Wh units. It reports about 4 Wh per count.

totalEnergyCapacity is a whole-tank quantity, so regressing it against the setpoint measures the quantum with no stratification assumption. The modal value at each setpoint forms an arithmetic sequence. These are raw wire counts:

setpoint °F	totalEnergyCapacity		setpoint °F	totalEnergyCapacity
140.0	1369		145.4	1580
141.8	1439		146.3	1615
143.6	1510		148.1	1685

R² = 0.99999, slope 70.25 raw counts per Kelvin of whole-tank rise. Endpoints alone: (1685 − 1369) / 4.5 K = 70.2. The 1580 at 145.4 °F is the value in tests/conftest.py, and 1580 × 4 = the 6320 Wh above.

Converting that to Watt-hours needs a water mass, and a "65 gallon" tank does not hold 65 gallons. Rather than assume nominal volume and derive an odd-looking quantum, assume the quantum is round — as every other conversion in this protocol is (half-degrees, tenths) — and check which implies a sensible volume:

candidate	implied water volume
4 Wh (1/250 kWh)	241.7 L / 63.9 gal	✅ under nominal, as expected
1/240 kWh	251.8 L / 66.5 gal	❌ more than the nameplate
10 kJ	167.8 L / 44.3 gal	❌
15 kJ	251.7 L / 66.5 gal	❌

Two further checks agree. 183 individual heating recoveries give 4.11 Wh/count by a noisier route (p10 3.47, p90 4.45 — it depends on (upper+lower)/2 approximating mean tank temperature). And integrating current_inst_power over those recoveries:

scale	implied COP
10 Wh/count (before)	7.02 — impossible
4 Wh/count (now)	2.89 — ordinary for a HPWH

That last argument needs no tank volume, specific heat or mixing model at all — it rules out the old scale independently of everything else here, though it cannot identify the quantum.

The reference temperature is the strongest result

Extrapolating the setpoint regression to total = 0 gives 104.92 °F, against the device's dhw_temperature_min of 104.9 °F. That uses only the device's own two numbers — no mass, no specific heat, no thermistors, no assumption about the quantum — and establishes full_recovery_energy = k · (setpoint − dhw_temperature_min) as fact rather than inference.

Reported tank energy is now 2.5x smaller. Historical series need rescaling by 0.4 to compare.

2. Field semantics

availableEnergyCapacity is not available energy — it is the energy still needed to reach the setpoint. It falls as the tank heats and hits zero when fully charged, so anything treating it as stored energy had the signal inverted. Regressed against mean tank temperature: negative slope, R² = 0.93, zero crossing at the setpoint.

totalEnergyCapacity is not a fixed tank size either. It is the cost of a full recovery to the current setpoint, measured from the device minimum setpoint (104.9 °F), and moves ~140 Wh per 0.5 °C of setpoint change. Seven months of history shows 16 distinct values.

old (removed)	new
total_energy_capacity	full_recovery_energy
available_energy_capacity	energy_to_setpoint

Removed rather than aliased, per the backward compatibility policy. A missed rename fails with AttributeError instead of silently returning a value 2.5x too small — the better failure mode, since the scale correction lands in the same change. Wire field names are unchanged.

3. Documentation

Three incompatible published descriptions: Wh with no conversion (protocol reference), Wh with a ×10 scale (the code), and a 0–100 percentage (track-energy.rst, models.rst, history.rst). The percentage claim was never true of any version.

Also fixes four field names in track-energy.rst that do not exist on DeviceStatus (dhw_tank_upper_temp, dhw_tank_lower_temp, comp_temp, dhw_heatex_out_temp), and documents that dhwTemperature is measured inside the tank rather than in the outlet pipe despite its name — it tracks tankUpperTemperature to within one 0.5 °C step, and the device has no sensor downstream of itself.

New docs/explanation/tank-energy.rst records the evidence and a migration table. It is the largest file here; the scale change makes every reported number 2.5x smaller, so the justification seemed worth keeping somewhere.

The vendor app reads neither field

Navien's own NaviLink app was decompiled to look for confirmation. It never touches either field.

The current release — 2.03.00, versionCode 141, published March 2026 — yields 8,101 Java sources, and neither totalEnergyCapacity nor availableEnergyCapacity appears in any of them, nor in the raw dex string pool (so this is not obfuscation). The app's status model KDResponseMgppStatus.Status declares ~140 fields — including dhwChargePer, tankUpperTemperature, tankLowerTemperature, currentInstPower and mixingRate — and neither energy field is among them. It requests no field subset, so the device sends both and the app drops them on deserialization. Controls: dhwChargePer and tankUpperTemperature do appear in the dex strings, so the absence is a real result.

So there is no vendor label, scale factor or formula to check any of this against — the analysis above is the only account of these two fields that exists. It also explains the misleading protocol names: nothing Navien ships ever has to act on them.

The app does corroborate three surrounding facts, now cited inline in the docs:

App source	Confirms
MgppStatusFragment.getVolume() maps volumeCode 1/2/3 to 189.2 / 246.0 / 302.8 L, labelled "Volume"	The nominal volume the quantum candidates are judged against is Navien's own figure, not an assumption of ours
MgppControlFragment.makeTempMap() reads dhwTemperatureMin / 2.0f as °C	Half-degree-C encoding, so 40.5 °C / 104.9 °F is exactly where the regression lands. The app rounds and displays 105 °F
Status screen labels dhwTemperature "DHW Temp." beside "Upper Temp."/"Lower Temp.", with a separate "Discharge Temp." row for dischargeTemperature	The vendor UI does not treat dhwTemperature as an outlet reading

Corrections since first review

• The evidence tables were 10x the raw wire values. They had been logged through the pre-fix library (which multiplied by 10) and tabulated without undoing it, under headings naming the raw protocol fields — while the prose beside them ("70.25 raw counts per Kelvin", "35 counts") was already in true counts. The page contradicted itself by exactly the factor it exists to correct. Now in raw counts and self-verifying against tests/conftest.py. No numeric conclusion changed.
• hpUpperOnTemperatureSetting → hpUpperOnTempSetting, the real protocol name.
• 143 Wh per 0.5 °C → 140 Wh, which is what the documented slope gives at 4 Wh/count.
• "overstated by 2.43x" → 2.5x where the sentence describes what the library reported. 2.43 is 10/4.11, the measured estimate; the code applied 10/4.0.
• The data_conversions note led with the per-recovery method the explanation page had already demoted; it now leads with the setpoint slope.

Reviewer notes

• Single-device calibration. The 4 Wh quantum comes from one 65-gallon unit but is applied to all volume codes. Correct if the quantum is a fixed firmware constant — the natural design — but unverified for the 50 and 80 gallon variants. A recovery trace from either would settle it; the app's volume table gives nominal litres per volumeCode but says nothing about the firmware quantum.
• dhw_charge_per remains unreconciled. It agrees with neither corrected field (48-point mean gap over two weeks). Documented as an opaque vendor heuristic rather than guessed at. The app prints it unmodified as "DHW Charge", so it is device-computed and is the number users actually see.
• Version. Breaking changes; wants a major bump.

Not in this PR

A nwp500.energy module computing usable energy and deliverable volume from tank geometry is parked on feat/nwp500-energy. It was originally included here and removed to keep this PR to the fix.

Testing

• 638 passed, 4 skipped on Python 3.14; new coverage for the energy quantum, the renamed fields, and confirmation the old names are gone
• tests/conftest.py capacity fixtures were 0.0; now carry real raw counts
• ruff check src/ tests/ and ruff format --check clean; mypy clean across 54 files
• Docs build clean; tank-energy.rst renders with no warnings and the new cross-reference resolves
• Every figure on the page re-derived against the code: energy_count_to_wh(1580) == 6320.0, endpoint slope (1685 − 1369)/4.5 = 70.2 counts/K, zero crossing 140.0 − 1369/39.06 = 104.95 °F, 70.25 × 0.5 × 4 = 140 Wh
• Verified against a live device: the CLI renders the new labels and values

Two environment notes, neither caused by this branch and both present on main:

• The repo requires Python ≥ 3.14 (setup.cfg). On 3.13 the whole package fails to import — mqtt_events.py annotates dataclass fields with DeviceStatus, imported only under TYPE_CHECKING, and without from __future__ import annotations that is evaluated eagerly. 3.14's lazy annotations (PEP 649) make it fine. A stale 3.13 .venv therefore looks like a total breakage.
• ruff check . reports 15 findings, all in scripts/ and setup.py — the same newer-ruff sweep already applied to examples/ in b6af787. Untouched by this PR.

[Copilot]

Pull request overview

This PR corrects tank energy telemetry interpretation in DeviceStatus (unit scale and reversed semantics), updates CLI/examples/docs accordingly, and introduces a new nwp500.energy module with pure, first-principles hot-water availability calculations.

Changes:

• Replace the old ×10 “energy capacity” conversion with a fixed ~4 Wh/count quantum and rename the two energy fields to reflect their true meaning (with deprecation aliases).
• Add nwp500.energy with TankModel + helper functions for stored/usable energy, deliverable volume, and setpoint targeting.
• Update CLI labels, examples, and protocol/Python API docs; add extensive unit tests.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/test_models.py	Adds deprecation/alias behavior tests for renamed energy fields.
tests/test_model_converters.py	Updates converter tests for the new energy quantum conversion.
tests/test_energy.py	New test suite validating nwp500.energy calculations and invariants.
tests/conftest.py	Updates fixture values to use raw device energy counts (not Wh).
src/nwp500/models/status.py	Renames energy fields, fixes conversion, improves descriptions, and adds deprecated properties with warnings.
src/nwp500/models/init.py	Re-exports the new annotated converter type alias.
src/nwp500/energy.py	New pure-functions module for tank energy and deliverable hot-water modeling.
src/nwp500/converters.py	Introduces WH_PER_ENERGY_COUNT and energy_count_to_wh, removing mul_10.
src/nwp500/cli/presentation.py	Updates CLI labels/field selection to the corrected energy fields.
src/nwp500/init.py	Exposes tank-energy helpers at the top-level package API.
examples/intermediate/periodic_requests.py	Updates example output to reflect the corrected semantics/name.
docs/reference/python_api/models.rst	Fixes Python API docs to match corrected field meaning + adds deprecation warning.
docs/reference/protocol/device_status.rst	Updates protocol reference text and conversion notes for energy fields and dhwTemperature.
docs/reference/protocol/data_conversions.rst	Corrects energy conversion/scaling documentation and usage guidance.
docs/project/history.rst	Corrects historical documentation statement about energy capacity being a percentage.
docs/how-to/track-energy.rst	Updates guidance/examples for the corrected energy deficit semantics and points users to nwp500.energy.
docs/how-to/manage-units.rst	Updates units guide for the renamed energy fields.
docs/explanation/tank-energy.rst	New deep-dive explanation + migration guidance.
docs/explanation/index.rst	Adds the new explanation page to the docs index.
CHANGELOG.rst	Documents the breaking semantics/scale corrections and the new energy module.

Suppressed comments (2)

src/nwp500/init.py:202

• energy_to_setpoint is missing from __all__, so it won’t be included in star-imports and docs that rely on __all__ to reflect the public API. If it’s intended to be a top-level export, it should be listed alongside the other tank-energy helpers.

    # Tank energy calculations
    "TankModel",
    "tank_mean_temperature",
    "stored_energy",
    "usable_energy",
    "deliverable_volume",
    "draw_energy",
    "setpoint_for_energy",

docs/how-to/track-energy.rst:154

• Same line-block formatting issue for the full_recovery_energy description: continuation lines should keep the leading | so the block renders correctly.

| **Description:** Cost of a full recovery to the *current setpoint*,
  measured from the device reference temperature of 104.9 degF. It moves
  with the setpoint, so it is not a fixed tank capacity.

[eman]

Valid — fixed in b4b0f0e.

The docstring computed 6436 Wh from a nominal 65 gallons while the assertion expected 6320 Wh, so it read as asserting a value its own reasoning contradicted.

6320 Wh is 1580 counts × 4 Wh, and implies 241.5 kg of water over the 22.5 K recovery span — just under the 246.05 kg a nominal 65 gallons would weigh, which is expected since a "65 gallon" tank does not hold 65 gallons. The docstring now derives that and explains why anchoring on nominal volume is the wrong move: it would give a quantum of 4.07, which is not a round number, unlike every other conversion in this protocol.

[eman]

False positive — no change made.

Indented continuation lines in a reStructuredText line block are valid: the continuation is folded into the preceding line. Parsed with docutils, this block produces one line_block node containing four line nodes and zero system messages.

It is also the established style of this file and predates the PR — the same pattern is on main:

| **Field:** ``current_inst_power``
| **Type:** ``float``
| **Units:** Watts (W)
| **Description:** Total instantaneous power consumption of the entire
  unit, including heat pump compressor and electric heating elements.

Prefixing the continuation lines with | would change the rendering, splitting one logical line into two.

[eman]

Re: src/nwp500/__init__.py — energy_to_setpoint not exported

Stale. This review ran against an earlier revision. The nwp500.energy module has since been removed from this branch to keep the PR scoped to the fix, so there are no tank-energy helpers to export — __init__.py no longer adds any. The CHANGELOG entry listing them went at the same time.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.