Problem
AlertEngine keeps its dedup and cooldown state entirely in process memory:
CooldownTracker (tokenjam/core/alerts.py:109-129) suppresses repeat alerts of the same type for the same agent within a cooldown window, using an in-memory _last_fired: dict[tuple[str, str], datetime].AlertEngine._failure_rate_fired (tokenjam/core/alerts.py:149) is an in-memory set[str] of session ids that have already fired a FAILURE_RATE alert, so a struggling session fires at most one alert instead of one per error past the threshold.
Both are constructed fresh in AlertEngine.__init__, and AlertEngine is instantiated exactly once, in IngestPipeline.__init__ (tokenjam/core/ingest.py:591: alert_engine = AlertEngine(db=db, config=config)). Neither structure is hydrated from anything on startup. Any process restart — the background daemon restarting, or a fresh CLI ingest process — forgets which (type, agent) pairs already fired and which sessions already crossed the failure-rate threshold. If the underlying condition is still true after the restart (e.g. the session is still in its cooldown window, or still erroring), the engine treats it as new and inserts another alert row. If alert dispatch channels (ntfy, webhook, Discord, Telegram) are configured, each of those duplicate rows re-sends externally too.
Expected
A daemon restart within an alert's cooldown window should not re-fire (insert or dispatch) the same (type, agent) alert, and a session that already fired a FAILURE_RATE alert should not fire a second one after a restart.
Where to look
tokenjam/core/alerts.py:109-129 — CooldownTrackertokenjam/core/alerts.py:149 — AlertEngine._failure_rate_firedtokenjam/core/ingest.py:591 — the single AlertEngine construction site- The
alerts table (tokenjam/core/db.py, CREATE TABLE IF NOT EXISTS alerts) already carries fired_at, type, agent_id, and session_id columns — enough to reconstruct both pieces of state on startup
Done when
- On
AlertEngine.__init__, CooldownTracker and _failure_rate_fired are hydrated from the alerts table (e.g. the most recent fired_at per (type, agent_id) within the cooldown window, and the set of session_ids with a FAILURE_RATE row) instead of starting empty. No new table is added — the existing alerts table is the source.
- A restart-simulation test confirms a daemon restart within an alert's cooldown window does not insert or dispatch a duplicate
(type, agent_id) alert.
- External dispatch (ntfy/webhook/Discord/Telegram) is verified to respect the hydrated state too, not just the DB insert.
Problem
AlertEnginekeeps its dedup and cooldown state entirely in process memory:CooldownTracker(tokenjam/core/alerts.py:109-129) suppresses repeat alerts of the same type for the same agent within a cooldown window, using an in-memory_last_fired: dict[tuple[str, str], datetime].AlertEngine._failure_rate_fired(tokenjam/core/alerts.py:149) is an in-memoryset[str]of session ids that have already fired aFAILURE_RATEalert, so a struggling session fires at most one alert instead of one per error past the threshold.Both are constructed fresh in
AlertEngine.__init__, andAlertEngineis instantiated exactly once, inIngestPipeline.__init__(tokenjam/core/ingest.py:591:alert_engine = AlertEngine(db=db, config=config)). Neither structure is hydrated from anything on startup. Any process restart — the background daemon restarting, or a fresh CLI ingest process — forgets which (type, agent) pairs already fired and which sessions already crossed the failure-rate threshold. If the underlying condition is still true after the restart (e.g. the session is still in its cooldown window, or still erroring), the engine treats it as new and inserts another alert row. If alert dispatch channels (ntfy, webhook, Discord, Telegram) are configured, each of those duplicate rows re-sends externally too.Expected
A daemon restart within an alert's cooldown window should not re-fire (insert or dispatch) the same (type, agent) alert, and a session that already fired a
FAILURE_RATEalert should not fire a second one after a restart.Where to look
tokenjam/core/alerts.py:109-129—CooldownTrackertokenjam/core/alerts.py:149—AlertEngine._failure_rate_firedtokenjam/core/ingest.py:591— the singleAlertEngineconstruction sitealertstable (tokenjam/core/db.py,CREATE TABLE IF NOT EXISTS alerts) already carriesfired_at,type,agent_id, andsession_idcolumns — enough to reconstruct both pieces of state on startupDone when
AlertEngine.__init__,CooldownTrackerand_failure_rate_firedare hydrated from thealertstable (e.g. the most recentfired_atper(type, agent_id)within the cooldown window, and the set ofsession_ids with aFAILURE_RATErow) instead of starting empty. No new table is added — the existingalertstable is the source.(type, agent_id)alert.