Skip to content

feat(influxdb): export peak power, enriched Tempo, cost simulation, contract, address and collection health - #628

Open
Germwalker wants to merge 7 commits into
MyElectricalData:mainfrom
Germwalker:feat/influxdb-additional-exports
Open

feat(influxdb): export peak power, enriched Tempo, cost simulation, contract, address and collection health#628
Germwalker wants to merge 7 commits into
MyElectricalData:mainfrom
Germwalker:feat/influxdb-additional-exports

Conversation

@Germwalker

Copy link
Copy Markdown

What this adds

Six new, independently-gated InfluxDB exports, each following the existing patterns already
used by ExportInfluxDB (for the data shape) and ExportMqtt (for the source query), plus one
bug fix discovered while validating the three "instant" exports end-to-end. All are disabled
by default
— nothing changes for existing installs until a user opts in.

  1. power_max — daily peak power. One point per day from the
    consumption_daily_max_power cache: peak power in VA, exact timestamp of the peak, and
    usage percentage against the contract's subscribed power when known.
  2. Tempo enrichmentExportInfluxDB.tempo() now also writes, on the current day's point
    only, color_tomorrow, the 6 configured Tempo prices (price_<color>) and the
    day-per-color counters (days_<color>). These share the same measurement/tags/timestamp as
    the existing color field, so InfluxDB merges them into the same point.
  3. cost_simulation — simulated costs by tariff offer (BASE/HC/HP/TEMPO with its 6
    sub-periods), sourced from the existing statistic/price_consumption cache. Tag model
    (granularity = year/month, offer, period) is designed so a
    sum(euro) group by (year) where granularity="year" query gives the yearly total per offer
    without a cartesian product against the monthly rows.
  4. contract_export — contract details (subscribed power, also converted to VA; plan;
    meter type; segment; distribution tariff; status; last activation date; the 7
    offpeak_hours_N fields), sourced from db.get_contract(), already used by
    ExportMqtt.contract().
  5. address_export — delivery point address (street, postal code, city, INSEE code),
    sourced from db.get_addresse(), already used by ExportMqtt.address().
  6. health_export — collection health (API quota used/limit, last call timestamp, last
    error, and consentement_days_left: days left before the Enedis consent expires, negative
    if already expired), sourced from db.get_usage_point(), already used by
    ExportMqtt.status().

Each export adds: one method on ExportInfluxDB (src/models/export_influxdb.py), one
conditional call in Job.run() (src/models/jobs.py), and one configuration key with
enable: False by default plus a <name>_config() accessor (src/models/config.py).

Included fix: wrong timestamp on the three "instant" exports

contract(), address() and health() write a single point per cycle with
self.tz.localize(datetime.now()). datetime.now() returns the container's local wall-clock
time (e.g. Europe/Paris via the TZ env var); localize() then labels that value as-is in
self.tz (UTC in the default config) without converting it — so the point lands in the
future by exactly the timezone offset (2h in summer, observed end-to-end: the points existed
in InfluxDB but a range(stop: now()) query without a future bound would not surface them).
Fixed by using datetime.now(pytz.utc).astimezone(self.tz), which computes the correct instant
regardless of the configured timezone. This only affects the three exports introduced in this
PR, so it is included here rather than filed as a separate fix.

Scope

3 files, 7 commits, +343/-2 lines total:

  • src/models/config.py (+65)
  • src/models/export_influxdb.py (+249/-2, includes the timestamp fix)
  • src/models/jobs.py (+31)

Notes for reviewers

  • Happy to split this into one PR per export if you'd rather review/merge them independently —
    grouped them here since they share the same three files and the same small pattern.
  • No existing behavior changes: every new call in jobs.py is gated behind its own
    <name>.enable configuration flag, defaulting to False.

Add ExportInfluxDB.max_power(), following the same pattern as tempo():
one point per day from the consumption_daily_max_power cache, with the
peak power in VA, the exact timestamp of the peak, and the usage
percentage against the contract's subscribed power when it is known.

Wired conditionally into Job.run() (jobs.py), same pattern as the
existing tempo export, and gated behind a new "power_max.enable"
configuration key, disabled by default so existing installs are
unaffected until opted in.
… day counts

Extend ExportInfluxDB.tempo(): in addition to the color field already
exported for every cached day, the current day's point now also gets
color_tomorrow, the 6 configured Tempo prices (price_<color>) and the
day-per-color counters (days_<color>), read from tempo_config. Since
these extra fields share the same measurement/tags/timestamp as the
existing color field, InfluxDB merges them into the same point instead
of creating a new one.
Add ExportInfluxDB.cost_simulation(), sourced from the existing
statistic/price_consumption cache (already used by the cost simulation
page), a nested JSON {year: {month: {offer: {euro,kWh,Wh}}}} with
offer among BASE, HC, HP and TEMPO (itself split by sub-period).

One point is written per (year[, month], offer[, TEMPO period]), with
a tag model designed to let InfluxDB queries sum by offer and by year
without a cartesian product: `granularity` (year/month) separates the
yearly total from its monthly components, and `offer`/`period` avoid
mixing TEMPO's six sub-periods with the single BASE/HC/HP amounts.

Wired conditionally into Job.run() (jobs.py) and gated behind a new
"cost_simulation.enable" configuration key, disabled by default.
Add ExportInfluxDB.contract(), sourced from db.get_contract(), already
used by ExportMqtt.contract(). Writes a single point in time (now),
overwritten on every cycle, with the subscribed power, plan, meter
type, segment, distribution tariff, contract status, last activation
date and the 7 offpeak_hours_N fields. Subscribed power is additionally
converted to VA (subscribed_power_va) for direct use in dashboards and
alerting, alongside the original string.

Wired conditionally into Job.run() (jobs.py) and gated behind a new
"contract_export.enable" configuration key, disabled by default.
Add ExportInfluxDB.address(), sourced from db.get_addresse(), already
used by ExportMqtt.address(). Writes a single point in time (now),
overwritten on every cycle, with street, postal code, city and INSEE
code as text fields; no tag beyond usage_point_id since none of these
values has a cardinality worth indexing on.

Wired conditionally into Job.run() (jobs.py) and gated behind a new
"address_export.enable" configuration key, disabled by default.
Add ExportInfluxDB.health(), sourced from db.get_usage_point(), already
used by ExportMqtt.status(). Writes a single point in time (now),
overwritten on every cycle, with the API call quota used and its
limit, last call timestamp, last error, and consentement_days_left:
the number of days left before the Enedis consent expires (negative if
already expired), computed here so Grafana/InfluxDB alerting does not
need to recompute it.

Wired conditionally into Job.run() (jobs.py) and gated behind a new
"health_export.enable" configuration key, disabled by default.
contract(), address() and health() used self.tz.localize(datetime.now()):
datetime.now() returns the container's local wall-clock time (e.g.
Europe/Paris via the TZ env var), and localize() labels that value
as-is in self.tz (UTC here) without converting it, so the point was
written with a timestamp in the FUTURE (offset = the timezone, 2h in
summer). Found during end-to-end verification: the points existed in
InfluxDB but a `range(stop: now())` query without a future bound could
not see them.

Fix: use datetime.now(pytz.utc).astimezone(self.tz), which computes
the correct current instant regardless of the configured timezone.

@m4dm4rtig4n m4dm4rtig4n left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this — the pattern-matching against the existing ExportInfluxDB/ExportMqtt code is spot on, and the timestamp fix is a real one that was worth catching. No need to split the PR from my point of view: the six exports share the same three files and the same tiny shape, and each is gated independently, so reviewing them together is fine.

I checked the diff against main (patch -p1 applies cleanly, files compile) and verified the parts that are easy to get wrong:

  • every DB method called exists with the signature used (get_daily_max_power_all(usage_point_id, order), get_contract, get_addresse, get_usage_point, get_stat, get_tempo_config, get_tempo_range), and every model attribute exists too (Addresses.city/insee_code, Contracts.*, UsagePoints.last_error/quota_*/ban/consentement_expiration, ConsumptionDailyMaxPower.event_date/value);
  • forceRound is module-level in export_influxdb.py, so no missing import;
  • the price_consumption shape matches Stat.generate_price() (zero-padded month keys, TEMPO always pre-populated with the six sub-periods at 0), so _cost_simulation_write can't hit a KeyError/None;
  • data.date == today is sound, since DB.set_tempo() stores naive midnight;
  • adding sections to Config.default doesn't upset Config.check(), which only walks home_assistant.

Two things I'd like changed before merge, then a few design questions.

Blocking

1. tempo(): float(price) doesn't handle the comma decimal separator

fields[f"price_{color}"] = float(price)

Tempo prices can reach the DB as strings using , as the decimal separator — that's exactly why convert_price() exists (export_home_assistant.py:44-55) and why Stat.generate_price() guards it:

if isinstance(tempo_price, str):
    tempo_price = float(tempo_price.replace(",", "."))

float("0,1296") raises ValueError. Because the only try/except is up in Job.export_influxdb(), an exception in tempo() aborts the whole InfluxDB export for that usage point — the five new exports and ecowatt() included. Could you route the prices through convert_price() (or the same isinstance(..., str) guard)? Same remark, less likely to bite, for int(days).

2. config.exemple.yaml (and the wiki) aren't updated

Config.load() replaces self.config with the parsed YAML — it doesn't merge self.default into it. So on an existing install none of the five new sections exist, power_max_config() & co. return False, and the exports stay off (which is the intent). The flip side is that the options are only discoverable in a freshly generated config.yaml: without a documented example, users can't realistically turn them on. Adding the five stanzas to config.exemple.yaml would fix that.

Design questions

Retention silently drops the historical points

InfluxDB.write() skips any point older than the bucket's retention window:

if self.retention == 0 or (date.replace(tzinfo=None) > date_max.replace(tzinfo=None)):

cost_simulation stamps the yearly rows on 1 January and the monthly rows on the 1st of the month, so on a bucket with retention — even a one-year one — the current year's granularity="year" point falls out of the window and the sum(euro) group by (year) where granularity="year" query from the description comes back empty. Worth either documenting that this export needs an infinite-retention bucket, or stamping the yearly row at a date inside the window.

"overwritten on every cycle" isn't what happens

For contract(), address() and health(), each cycle writes a point at a new timestamp, so InfluxDB keeps them all: the series grows without bound for data that essentially never changes. Two options — stamp them at a stable instant (e.g. the current day's midnight) so re-writes really are idempotent, or just reword the docstrings. I'd lean towards the former; last() queries keep working either way and the series stays flat.

One try/except for six independent exports

Given the whole point is that each export is independently gated, it feels inconsistent that a failure in any one of them takes down the others plus ecowatt(). A per-export try/except (or a small helper wrapping name -> callable) would match the intent.

Tempo enrichment is conditional on today's point existing

The prices, per-colour day counters and color_tomorrow are only written when the cache already holds a row for today. If the day's colour hasn't been fetched yet, none of it lands — whereas ExportMqtt.tempo() publishes prices and counters unconditionally. Intentional?

Nits

  • Naming: three of the five keys carry an _export suffix and two don't; the method is max_power() while the key and measurement are power_max; the key is health_export while the measurement is collect_health.
  • These five keys are top-level config sections even though the exports are InfluxDB-only. tempo is top-level because it's shared with the MQTT/HA exports, so it isn't quite a precedent — nesting under influxdb: would keep the root namespace clean.
  • event_timestamp is written as a string, so it can't be used in arithmetic on the Grafana side. An epoch value (or using event_date as the point timestamp) would be more usable.
  • consentement_days_left truncates through .days (23 hours left reads as 0), and uses a naive datetime.now() against a naive column — which works, but sits oddly in a PR that fixes timezone handling elsewhere.
  • int(subscribed_power.split(" ")[0]) * 1000 is now the fourth copy of that parsing (export_mqtt.py:401, ajax.py:740, stat.py:274, and twice here). Good candidate for a small helper.
  • address() skips locality and country; contract() skips usage_point_status. Deliberate?
  • No tests, while CI runs pytest --cov. A single test asserting the tag model of cost_simulation (mocking INFLUXDB.write) would lock in the part that's hardest to get right.
  • The description says "+343/-2"; GitHub reports +326/-2.
  • tox -e linters enables DTZ, so the bare datetime(...) / datetime.now() calls added here would be flagged. No CI workflow actually runs the linters (only pytest.yaml), and the surrounding code does the same thing, so this is informational.

Nice side effect

max_power() mirrors ExportMqtt.max_power() closely — same query, same value/percentage_usage naming — and in doing so it fixes a bug the MQTT version has: there, threshold_usage = int(100 * value_w / max_value) sits outside the max_value != 0 guard (export_mqtt.py:415), so it raises ZeroDivisionError whenever subscribed_power is missing. Your if subscribed_power_va: guard is the right shape. That looks like a plausible cause for #542 / #551 — I'll follow up separately on the MQTT side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants