feat(influxdb): export peak power, enriched Tempo, cost simulation, contract, address and collection health - #628
Conversation
Add ExportInfluxDB.max_power(), following the same pattern as tempo(): one point per day from the consumption_daily_max_power cache, with the peak power in VA, the exact timestamp of the peak, and the usage percentage against the contract's subscribed power when it is known. Wired conditionally into Job.run() (jobs.py), same pattern as the existing tempo export, and gated behind a new "power_max.enable" configuration key, disabled by default so existing installs are unaffected until opted in.
… day counts Extend ExportInfluxDB.tempo(): in addition to the color field already exported for every cached day, the current day's point now also gets color_tomorrow, the 6 configured Tempo prices (price_<color>) and the day-per-color counters (days_<color>), read from tempo_config. Since these extra fields share the same measurement/tags/timestamp as the existing color field, InfluxDB merges them into the same point instead of creating a new one.
Add ExportInfluxDB.cost_simulation(), sourced from the existing
statistic/price_consumption cache (already used by the cost simulation
page), a nested JSON {year: {month: {offer: {euro,kWh,Wh}}}} with
offer among BASE, HC, HP and TEMPO (itself split by sub-period).
One point is written per (year[, month], offer[, TEMPO period]), with
a tag model designed to let InfluxDB queries sum by offer and by year
without a cartesian product: `granularity` (year/month) separates the
yearly total from its monthly components, and `offer`/`period` avoid
mixing TEMPO's six sub-periods with the single BASE/HC/HP amounts.
Wired conditionally into Job.run() (jobs.py) and gated behind a new
"cost_simulation.enable" configuration key, disabled by default.
Add ExportInfluxDB.contract(), sourced from db.get_contract(), already used by ExportMqtt.contract(). Writes a single point in time (now), overwritten on every cycle, with the subscribed power, plan, meter type, segment, distribution tariff, contract status, last activation date and the 7 offpeak_hours_N fields. Subscribed power is additionally converted to VA (subscribed_power_va) for direct use in dashboards and alerting, alongside the original string. Wired conditionally into Job.run() (jobs.py) and gated behind a new "contract_export.enable" configuration key, disabled by default.
Add ExportInfluxDB.address(), sourced from db.get_addresse(), already used by ExportMqtt.address(). Writes a single point in time (now), overwritten on every cycle, with street, postal code, city and INSEE code as text fields; no tag beyond usage_point_id since none of these values has a cardinality worth indexing on. Wired conditionally into Job.run() (jobs.py) and gated behind a new "address_export.enable" configuration key, disabled by default.
Add ExportInfluxDB.health(), sourced from db.get_usage_point(), already used by ExportMqtt.status(). Writes a single point in time (now), overwritten on every cycle, with the API call quota used and its limit, last call timestamp, last error, and consentement_days_left: the number of days left before the Enedis consent expires (negative if already expired), computed here so Grafana/InfluxDB alerting does not need to recompute it. Wired conditionally into Job.run() (jobs.py) and gated behind a new "health_export.enable" configuration key, disabled by default.
contract(), address() and health() used self.tz.localize(datetime.now()): datetime.now() returns the container's local wall-clock time (e.g. Europe/Paris via the TZ env var), and localize() labels that value as-is in self.tz (UTC here) without converting it, so the point was written with a timestamp in the FUTURE (offset = the timezone, 2h in summer). Found during end-to-end verification: the points existed in InfluxDB but a `range(stop: now())` query without a future bound could not see them. Fix: use datetime.now(pytz.utc).astimezone(self.tz), which computes the correct current instant regardless of the configured timezone.
m4dm4rtig4n
left a comment
There was a problem hiding this comment.
Thanks for this — the pattern-matching against the existing ExportInfluxDB/ExportMqtt code is spot on, and the timestamp fix is a real one that was worth catching. No need to split the PR from my point of view: the six exports share the same three files and the same tiny shape, and each is gated independently, so reviewing them together is fine.
I checked the diff against main (patch -p1 applies cleanly, files compile) and verified the parts that are easy to get wrong:
- every DB method called exists with the signature used (
get_daily_max_power_all(usage_point_id, order),get_contract,get_addresse,get_usage_point,get_stat,get_tempo_config,get_tempo_range), and every model attribute exists too (Addresses.city/insee_code,Contracts.*,UsagePoints.last_error/quota_*/ban/consentement_expiration,ConsumptionDailyMaxPower.event_date/value); forceRoundis module-level inexport_influxdb.py, so no missing import;- the
price_consumptionshape matchesStat.generate_price()(zero-padded month keys,TEMPOalways pre-populated with the six sub-periods at 0), so_cost_simulation_writecan't hit aKeyError/None; data.date == todayis sound, sinceDB.set_tempo()stores naive midnight;- adding sections to
Config.defaultdoesn't upsetConfig.check(), which only walkshome_assistant.
Two things I'd like changed before merge, then a few design questions.
Blocking
1. tempo(): float(price) doesn't handle the comma decimal separator
fields[f"price_{color}"] = float(price)Tempo prices can reach the DB as strings using , as the decimal separator — that's exactly why convert_price() exists (export_home_assistant.py:44-55) and why Stat.generate_price() guards it:
if isinstance(tempo_price, str):
tempo_price = float(tempo_price.replace(",", "."))float("0,1296") raises ValueError. Because the only try/except is up in Job.export_influxdb(), an exception in tempo() aborts the whole InfluxDB export for that usage point — the five new exports and ecowatt() included. Could you route the prices through convert_price() (or the same isinstance(..., str) guard)? Same remark, less likely to bite, for int(days).
2. config.exemple.yaml (and the wiki) aren't updated
Config.load() replaces self.config with the parsed YAML — it doesn't merge self.default into it. So on an existing install none of the five new sections exist, power_max_config() & co. return False, and the exports stay off (which is the intent). The flip side is that the options are only discoverable in a freshly generated config.yaml: without a documented example, users can't realistically turn them on. Adding the five stanzas to config.exemple.yaml would fix that.
Design questions
Retention silently drops the historical points
InfluxDB.write() skips any point older than the bucket's retention window:
if self.retention == 0 or (date.replace(tzinfo=None) > date_max.replace(tzinfo=None)):cost_simulation stamps the yearly rows on 1 January and the monthly rows on the 1st of the month, so on a bucket with retention — even a one-year one — the current year's granularity="year" point falls out of the window and the sum(euro) group by (year) where granularity="year" query from the description comes back empty. Worth either documenting that this export needs an infinite-retention bucket, or stamping the yearly row at a date inside the window.
"overwritten on every cycle" isn't what happens
For contract(), address() and health(), each cycle writes a point at a new timestamp, so InfluxDB keeps them all: the series grows without bound for data that essentially never changes. Two options — stamp them at a stable instant (e.g. the current day's midnight) so re-writes really are idempotent, or just reword the docstrings. I'd lean towards the former; last() queries keep working either way and the series stays flat.
One try/except for six independent exports
Given the whole point is that each export is independently gated, it feels inconsistent that a failure in any one of them takes down the others plus ecowatt(). A per-export try/except (or a small helper wrapping name -> callable) would match the intent.
Tempo enrichment is conditional on today's point existing
The prices, per-colour day counters and color_tomorrow are only written when the cache already holds a row for today. If the day's colour hasn't been fetched yet, none of it lands — whereas ExportMqtt.tempo() publishes prices and counters unconditionally. Intentional?
Nits
- Naming: three of the five keys carry an
_exportsuffix and two don't; the method ismax_power()while the key and measurement arepower_max; the key ishealth_exportwhile the measurement iscollect_health. - These five keys are top-level config sections even though the exports are InfluxDB-only.
tempois top-level because it's shared with the MQTT/HA exports, so it isn't quite a precedent — nesting underinfluxdb:would keep the root namespace clean. event_timestampis written as a string, so it can't be used in arithmetic on the Grafana side. An epoch value (or usingevent_dateas the point timestamp) would be more usable.consentement_days_lefttruncates through.days(23 hours left reads as 0), and uses a naivedatetime.now()against a naive column — which works, but sits oddly in a PR that fixes timezone handling elsewhere.int(subscribed_power.split(" ")[0]) * 1000is now the fourth copy of that parsing (export_mqtt.py:401,ajax.py:740,stat.py:274, and twice here). Good candidate for a small helper.address()skipslocalityandcountry;contract()skipsusage_point_status. Deliberate?- No tests, while CI runs
pytest --cov. A single test asserting the tag model ofcost_simulation(mockingINFLUXDB.write) would lock in the part that's hardest to get right. - The description says "+343/-2"; GitHub reports +326/-2.
tox -e lintersenablesDTZ, so the baredatetime(...)/datetime.now()calls added here would be flagged. No CI workflow actually runs the linters (onlypytest.yaml), and the surrounding code does the same thing, so this is informational.
Nice side effect
max_power() mirrors ExportMqtt.max_power() closely — same query, same value/percentage_usage naming — and in doing so it fixes a bug the MQTT version has: there, threshold_usage = int(100 * value_w / max_value) sits outside the max_value != 0 guard (export_mqtt.py:415), so it raises ZeroDivisionError whenever subscribed_power is missing. Your if subscribed_power_va: guard is the right shape. That looks like a plausible cause for #542 / #551 — I'll follow up separately on the MQTT side.
What this adds
Six new, independently-gated InfluxDB exports, each following the existing patterns already
used by
ExportInfluxDB(for the data shape) andExportMqtt(for the source query), plus onebug fix discovered while validating the three "instant" exports end-to-end. All are disabled
by default — nothing changes for existing installs until a user opts in.
power_max— daily peak power. One point per day from theconsumption_daily_max_powercache: peak power in VA, exact timestamp of the peak, andusage percentage against the contract's subscribed power when known.
ExportInfluxDB.tempo()now also writes, on the current day's pointonly,
color_tomorrow, the 6 configured Tempo prices (price_<color>) and theday-per-color counters (
days_<color>). These share the same measurement/tags/timestamp asthe existing
colorfield, so InfluxDB merges them into the same point.cost_simulation— simulated costs by tariff offer (BASE/HC/HP/TEMPO with its 6sub-periods), sourced from the existing
statistic/price_consumptioncache. Tag model(
granularity= year/month,offer,period) is designed so asum(euro) group by (year) where granularity="year"query gives the yearly total per offerwithout a cartesian product against the monthly rows.
contract_export— contract details (subscribed power, also converted to VA; plan;meter type; segment; distribution tariff; status; last activation date; the 7
offpeak_hours_Nfields), sourced fromdb.get_contract(), already used byExportMqtt.contract().address_export— delivery point address (street, postal code, city, INSEE code),sourced from
db.get_addresse(), already used byExportMqtt.address().health_export— collection health (API quota used/limit, last call timestamp, lasterror, and
consentement_days_left: days left before the Enedis consent expires, negativeif already expired), sourced from
db.get_usage_point(), already used byExportMqtt.status().Each export adds: one method on
ExportInfluxDB(src/models/export_influxdb.py), oneconditional call in
Job.run()(src/models/jobs.py), and one configuration key withenable: Falseby default plus a<name>_config()accessor (src/models/config.py).Included fix: wrong timestamp on the three "instant" exports
contract(),address()andhealth()write a single point per cycle withself.tz.localize(datetime.now()).datetime.now()returns the container's local wall-clocktime (e.g. Europe/Paris via the
TZenv var);localize()then labels that value as-is inself.tz(UTC in the default config) without converting it — so the point lands in thefuture by exactly the timezone offset (2h in summer, observed end-to-end: the points existed
in InfluxDB but a
range(stop: now())query without a future bound would not surface them).Fixed by using
datetime.now(pytz.utc).astimezone(self.tz), which computes the correct instantregardless of the configured timezone. This only affects the three exports introduced in this
PR, so it is included here rather than filed as a separate fix.
Scope
3 files, 7 commits, +343/-2 lines total:
src/models/config.py(+65)src/models/export_influxdb.py(+249/-2, includes the timestamp fix)src/models/jobs.py(+31)Notes for reviewers
grouped them here since they share the same three files and the same small pattern.
jobs.pyis gated behind its own<name>.enableconfiguration flag, defaulting toFalse.