fix(scheduler): record who initiated an execution (#1970) - #1974
Conversation
`schedule_executions` has carried five origin columns for audit since
AUDIT-001, and the backend populates them on every path it owns. The
scheduler is a separate service with its own DB module, and its
`create_execution()` listed none of them in the INSERT — nor accepted
them in its signature, so there was nowhere to put a caller even if one
had been forwarded. Every scheduler-created row was written with all
five NULL.
`triggered_by='manual'` therefore recorded *that* a human ran something
and never *who*. The attribution lived only in backend/MCP-server logs,
bounded by log retention, so past a few weeks the durable record could
not answer "did anyone trigger this run, and who?".
The identity was dropped at three points, not one:
1. the backend's delegating POST sent no body at all, so the
authenticated caller — in scope right there — never crossed the hop;
2. `_trigger_handler` had no parameter to receive one;
3. `create_execution()` had nowhere to put it.
An `ExecutionOrigin` value object is threaded through all three. One
object rather than five parallel parameters at four call depths: five
positional siblings is how one of them silently stops being forwarded.
Two paths the DB fix alone would have left blank are covered too. A
retry inherits the original run's origin — it has no caller of its own,
but a chain of retries that drops the initiator makes the first attempt
the only attributable one; the read is fail-open, since an audit lookup
must not be able to stop a retry from running. A reminder inherits the
provenance #1296 already persisted.
Cron ticks stay NULL. Attributing an autonomous fire to, say, the
schedule's owner would make the column actively misleading — a blank
reads as "unknown", a wrong name does not.
Also hardened while here:
- the untrusted trigger body is validated at the scheduler boundary.
`source_user_id` is dropped rather than coerced when it is not an int:
`bool` IS an `int` in Python, so `True` would have persisted as user
1, a real account attributed to a run it had nothing to do with.
Strings are length-capped and blank-to-None, so "" and NULL are not
two spellings of "unknown".
- the backend prefers the validated `current_user.agent_name` over the
raw `X-Source-Agent` header — the reverse of chat.py's precedence,
which is fine for a collaboration hint but would let a caller pin its
run on a sibling agent in an audit column.
- the MCP trigger tool forwards the origin headers `chat()` already
sends (Invariant #13). Without it an MCP-triggered run attributes to
the key OWNER but not to which key or agent fired it — the part that
identifies the actor when one human owns many of both.
Not a vulnerability: nothing authorizes on these columns.
Backward compatible in both rolling-deploy directions — an old scheduler
ignores the new body fields, and a new scheduler treats a bodyless POST
as an unattributed manual trigger.
tests/unit/test_1970_execution_origin.py — 27 checks, 25 of which fail
against the pre-fix tree.
Related to #1970
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Resolve by running |
/review — self-review (PR #1974, #1970)Branch: Self-review, so discount accordingly. Nothing blocking found. [I1] The retry-origin read is a new DB call on the retry path (Confidence: 8/10)original = self.db.get_execution(original_execution_id)Trading "the retry is unattributed" for "the retry never runs" would be a bad deal, so this is wrapped fail-open with a WARNING. Worth naming because the first version I wrote had the [I2] Attribution fields are caller-supplied at the scheduler boundary (Confidence: 7/10)
[I3] The 255-char cap is not derived from the column (Confidence: 6/10)
Clean categories
Summary
CI: 18/18 green. |
…ases)
Two real bugs in this PR's own `ExecutionOrigin`, found by a boundary +
property pass over `from_payload` and fixed here.
**1. Range was never checked.** The parser dropped wrong-TYPED values and
capped string LENGTH, but accepted any `int`. `source_user_id` lands in an
INTEGER column, and SQLite raises `OverflowError: Python int too large to
convert to SQLite INTEGER` rather than truncating — verified end-to-end
against a real `create_execution`. On this PR's path the exception lands in
`_execute_manual_trigger`'s blanket `except`, so the run is logged and
silently lost *after* the endpoint already answered `"triggered"`. A field
of an untrusted payload decided whether the schedule ran, which is the exact
thing this function exists to prevent — the range check is the missing
sibling of a guard that was otherwise present.
**2. `is_empty()` used truthiness.** `not any((self.user_id, ...))` reports
an origin carrying `user_id=0` as empty. Hypothesis shrank it to
`{"source_user_id": 0}`. Latent — nothing in `src/` calls it yet — which is
why it was worth fixing now, before the first caller inherited a helper that
lies about a valid value.
Both lived inside code with **100% statement and branch coverage** from
`test_1970_execution_origin.py`. Full coverage says every line ran, not that
every value class was tried; that gap is the whole reason for a separate
edge-case pass.
tests/unit/test_execution_origin_properties.py — 31 boundary rows + 6
Hypothesis properties.
Related to #1970
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/edge-cases — 3 real bugs found across today's PRs (2 in this one)Boundary-value analysis + Hypothesis over [B1] Range was never checked — a payload field could silently lose a run
Verified end-to-end against a real The range check is the missing sibling of a guard that was otherwise present — the function already drops unusable input for exactly this reason. [B2]
|
…sts/registry.json — rebuild: dev entries + this PR's two entries)
vybe
left a comment
There was a problem hiding this comment.
Validated via /validate-pr — complete three-surface fix (backend router + scheduler + MCP client/tool), 25/27 regression tests fail pre-fix plus Hypothesis properties, retry inheritance and rolling-deploy compatibility verified both directions, extraHeaders hook refuses Authorization. I added the missing 'Fixes #1970' keyword and resolved the registry.json conflict (rebuild: dev + this PR's two entries); scheduler/service.py auto-merged cleanly against #1975 as predicted. Full CI green post-rebase. Note: stacked #1976 auto-retargets to dev on branch delete. Known follow-up: architecture.md schedule_executions DDL omits the source_* columns (pre-existing drift).
…uced My rebase of this branch onto dev left `src/scheduler/models.py` with TWO `class ExecutionOrigin` definitions: this PR's at line 110 and dev's copy (#1974, as merged) at 187. Python keeps the last one, so the second shadowed the first and both of this PR's fixes became dead code — the None-comparing `is_empty` and the SQLite-range guard on `source_user_id`. Three `test_execution_origin_properties` cases went red on head, correctly. Git did not flag it. Both sides added the class at different offsets, so the textual auto-merge took both hunks and reported no conflict — a semantic duplicate that only a reader or an importer would notice. Two things on my side let it through: - I re-read only the files git marked as conflicted, not the whole merged result of a 4-commit rebase. - My post-rebase check was `-k "1968 or ent326 or timeline or scheduler or executions"`, which does not match `test_execution_origin_properties`. The filter was narrower than the blast radius, so 129 tests passed and said nothing about the file I had just broken. Kept the first copy: it is a strict superset (identical fields, `is_empty` comparing against None so `user_id=0` is not reported empty, and the `_SQLITE_INT_MIN/MAX` range check that stops an out-of-range id reaching the INSERT and taking the dispatch down). Deleted dev's older copy. Swept the other six branches I rebased for the same class of damage — duplicate top-level defs in any changed .py — all clean. 350 passed across origin/scheduler/1968/1969/1970/execution. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…uced My rebase of this branch onto dev left `src/scheduler/models.py` with TWO `class ExecutionOrigin` definitions: this PR's at line 110 and dev's copy (#1974, as merged) at 187. Python keeps the last one, so the second shadowed the first and both of this PR's fixes became dead code — the None-comparing `is_empty` and the SQLite-range guard on `source_user_id`. Three `test_execution_origin_properties` cases went red on head, correctly. Git did not flag it. Both sides added the class at different offsets, so the textual auto-merge took both hunks and reported no conflict — a semantic duplicate that only a reader or an importer would notice. Two things on my side let it through: - I re-read only the files git marked as conflicted, not the whole merged result of a 4-commit rebase. - My post-rebase check was `-k "1968 or ent326 or timeline or scheduler or executions"`, which does not match `test_execution_origin_properties`. The filter was narrower than the blast radius, so 129 tests passed and said nothing about the file I had just broken. Kept the first copy: it is a strict superset (identical fields, `is_empty` comparing against None so `user_id=0` is not reported empty, and the `_SQLITE_INT_MIN/MAX` range check that stops an out-of-range id reaching the INSERT and taking the dispatch down). Deleted dev's older copy. Swept the other six branches I rebased for the same class of damage — duplicate top-level defs in any changed .py — all clean. 350 passed across origin/scheduler/1968/1969/1970/execution. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ted (#1968) (#1976) * fix(scheduler): record who initiated an execution (#1970) `schedule_executions` has carried five origin columns for audit since AUDIT-001, and the backend populates them on every path it owns. The scheduler is a separate service with its own DB module, and its `create_execution()` listed none of them in the INSERT — nor accepted them in its signature, so there was nowhere to put a caller even if one had been forwarded. Every scheduler-created row was written with all five NULL. `triggered_by='manual'` therefore recorded *that* a human ran something and never *who*. The attribution lived only in backend/MCP-server logs, bounded by log retention, so past a few weeks the durable record could not answer "did anyone trigger this run, and who?". The identity was dropped at three points, not one: 1. the backend's delegating POST sent no body at all, so the authenticated caller — in scope right there — never crossed the hop; 2. `_trigger_handler` had no parameter to receive one; 3. `create_execution()` had nowhere to put it. An `ExecutionOrigin` value object is threaded through all three. One object rather than five parallel parameters at four call depths: five positional siblings is how one of them silently stops being forwarded. Two paths the DB fix alone would have left blank are covered too. A retry inherits the original run's origin — it has no caller of its own, but a chain of retries that drops the initiator makes the first attempt the only attributable one; the read is fail-open, since an audit lookup must not be able to stop a retry from running. A reminder inherits the provenance #1296 already persisted. Cron ticks stay NULL. Attributing an autonomous fire to, say, the schedule's owner would make the column actively misleading — a blank reads as "unknown", a wrong name does not. Also hardened while here: - the untrusted trigger body is validated at the scheduler boundary. `source_user_id` is dropped rather than coerced when it is not an int: `bool` IS an `int` in Python, so `True` would have persisted as user 1, a real account attributed to a run it had nothing to do with. Strings are length-capped and blank-to-None, so "" and NULL are not two spellings of "unknown". - the backend prefers the validated `current_user.agent_name` over the raw `X-Source-Agent` header — the reverse of chat.py's precedence, which is fine for a collaboration hint but would let a caller pin its run on a sibling agent in an audit column. - the MCP trigger tool forwards the origin headers `chat()` already sends (Invariant #13). Without it an MCP-triggered run attributes to the key OWNER but not to which key or agent fired it — the part that identifies the actor when one human owns many of both. Not a vulnerability: nothing authorizes on these columns. Backward compatible in both rolling-deploy directions — an old scheduler ignores the new body fields, and a new scheduler treats a bodyless POST as an unattributed manual trigger. tests/unit/test_1970_execution_origin.py — 27 checks, 25 of which fail against the pre-fix tree. Related to #1970 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(scheduler): return a real execution_id, and 409 when nothing started (#1968) `_trigger_handler` was fire-and-forget: it spawned the run with `asyncio.create_task` and responded immediately, *before* the execution record existed. So it had no id to return. The backend relayed the same id-less fields, and the MCP tool interpolated the missing key — telling every agent `Execution started with ID 'undefined'`, on every trigger, while the execution ran fine. Callers could not correlate a trigger with its run, poll it, or fetch its result; the workaround was to guess from `list_recent_executions` by timestamp. The same ordering hid a second problem. The response was emitted before `_execute_manual_trigger` had even attempted the distributed lock, so a trigger suppressed because the schedule was already running still answered `"status": "triggered"`. A suppressed trigger and a real one were byte-identical to the caller. The handler now acquires the lock and creates the row synchronously, then hands both to the background task. That makes two facts sayable that simply did not exist yet at response time: which execution this is, and whether one was started at all. * 200 carries a real `execution_id`, valid the moment the caller receives it — a fast poller must not 404. * 409 `already_running` replaces the false "triggered", with no id and no row, because nothing ran. Exactly one row per trigger: `_execute_schedule_with_lock` takes the pre-created execution and skips its own create. Two rows would hand the caller an id naming a row that never runs while a second did the work. Because a row can now exist before a gate decides not to run, an abandoned run FAILs its pre-created row rather than leaving it `running` forever — canary E-01's exact signature, and a task the UI would show indefinitely. The handler also now holds the lock across a DB write, which is new, so every exit from that window releases it: creation raising, creation returning None, the run raising, and normal completion — exactly once each. A second release is the dangerous one, since a lock re-acquired by the next run in between would be freed out from under it. Relayed through the remaining surfaces: * the backend forwards `execution_id` (and records it on the audit row, so a trigger and its run are joinable after the fact) and maps 409 rather than flattening it into "Failed to trigger schedule" — a worse lie than the original, since it claims failure where the schedule is healthily busy; * the MCP tool returns a structured `already_running` instead of throwing, so an agent gets a decision it can act on, and GUARDS the id instead of interpolating it — an older backend still omits the field, and swapping one confident lie for another is not a fix; * `ScheduleTriggerResult.execution_id` becomes optional. Typing it as a required `string` while the wire never sent it is precisely why the compiler stayed happy through every `undefined`; * the UI reads 409 as "already running" rather than "nothing was changed — try again", and the CLI prints the id it was already fetching and discarding. tests/unit/test_1968_trigger_execution_id.py — 22 checks, 17 of which fail against the pre-fix tree. Related to #1968 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(scheduler): keep a strong reference to the spawned trigger task (#1968) Self-review finding on this PR. The event loop holds only a WEAK reference to a task, so a bare `asyncio.create_task(...)` whose result nobody keeps can be garbage-collected mid-flight — the asyncio docs say so outright. The bare call predates this PR, but this PR changes what it costs. Before, a collected task meant the run silently did not happen. Now the lock is acquired and the execution row created BEFORE the task is spawned, so a collected task strands a `running` execution whose id the caller already holds and pins the schedule's lock until its Redis TTL. Uses the `_inflight` set + `add_done_callback(discard)` shape the #1083 result-callback path already established (`agent_server/services/result_callback.py`), so the set cannot grow without bound. Guarded by `test_the_spawned_task_is_strongly_referenced`, which checks the reference is held in the window BEFORE the task runs — the only window where collection is possible — and released after. Related to #1968 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(learnings): record the create_task-owns-state class surfaced by #1968 review Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(scheduler): remove the duplicate ExecutionOrigin my rebase introduced My rebase of this branch onto dev left `src/scheduler/models.py` with TWO `class ExecutionOrigin` definitions: this PR's at line 110 and dev's copy (#1974, as merged) at 187. Python keeps the last one, so the second shadowed the first and both of this PR's fixes became dead code — the None-comparing `is_empty` and the SQLite-range guard on `source_user_id`. Three `test_execution_origin_properties` cases went red on head, correctly. Git did not flag it. Both sides added the class at different offsets, so the textual auto-merge took both hunks and reported no conflict — a semantic duplicate that only a reader or an importer would notice. Two things on my side let it through: - I re-read only the files git marked as conflicted, not the whole merged result of a 4-commit rebase. - My post-rebase check was `-k "1968 or ent326 or timeline or scheduler or executions"`, which does not match `test_execution_origin_properties`. The filter was narrower than the blast radius, so 129 tests passed and said nothing about the file I had just broken. Kept the first copy: it is a strict superset (identical fields, `is_empty` comparing against None so `user_id=0` is not reported empty, and the `_SQLITE_INT_MIN/MAX` range check that stops an out-of-range id reaching the INSERT and taking the dispatch down). Deleted dev's older copy. Swept the other six branches I rebased for the same class of damage — duplicate top-level defs in any changed .py — all clean. 350 passed across origin/scheduler/1968/1969/1970/execution. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(webhooks): a busy schedule is a healthy 202, not a 503 (#1968) The blast-radius table listed three consumers of the scheduler's `/trigger` endpoint and asserted none regress. There is a fourth — `routers/webhooks.py`, the unauthenticated public trigger — and it is the one that does. With #1968's 409, `if response.status_code not in (200, 202)` caught a HEALTHY, busy schedule and turned it into an ERROR log line plus a 503 'Trigger failed — try again later' to a public caller, advice that only hits the same lock. Worse, the `except HTTPException` arm then called `idempotency_service.fail(idem)`, releasing the #525 dedup claim for a delivery that never failed — so a retry could fire a second execution the moment the lock cleared. Before #1968 that delivery returned 202. Now a 409 is recognised as its own outcome: INFO log, the claim COMPLETED with a snapshot recording what happened, and 202 with `status: "already_running"` and a message saying the delivery was coalesced into the run in flight rather than claiming a fresh execution started. Genuine scheduler errors (500/502/400) still surface as 503 — the carve-out is on 409 alone, and a test pins that it does not widen. tests/test_webhook_triggers.py asserts `status_code in (202, 503)` throughout, which is why CI accepted the regression silently, so the new tests name the status instead of tolerating a set. They run in-process (TestClient + faked scheduler) rather than against a live instance: carve-out removed: 3 failed, 3 passed as shipped: 6 passed Two of the non-blocking notes, both one-liners: * `_abandon_precreated_execution` now passes `expected_status=RUNNING` to a new optional CAS precondition on the scheduler's `update_execution_status`. It was safe by argument (every abandon gate runs before dispatch); #1082 exists to retire exactly that kind of argument, and the clause makes it safe by construction. Default None keeps every other caller byte-identical. * `schedules.ts` logged `execution_id: undefined` above the `!result.execution_id` guard — the exact string this issue is named after. Moved below it. Related to #1968 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Problem
schedule_executionshas carried five execution-origin columns for audit since AUDIT-001, andsrc/backend/database.pypopulates them on every path the backend owns. The scheduler is a separate service with its own DB module, and itscreate_execution()listed none of them:Nor did the signature accept them, so there was nowhere to put a caller even if one had been forwarded. Every scheduler-created row was written with all five NULL — observed across 100+ runs of an actively used schedule.
triggered_by='manual'recorded that a human ran something and never who. The attribution existed only in backend / MCP-server logs, bounded by log retention, so past a few weeks the durable record could not answer "did anyone trigger this run, and who?".Not a vulnerability — nothing authorizes on these columns.
The identity was dropped at three points, not one
routers/schedules.py::trigger_schedulePOSTto the scheduler sent no body at all. The authenticated caller was in scope right there and never crossed the hop.scheduler/main.py::_trigger_handlerscheduler/database.py::create_executionFixing only #3 — the line the issue points at — changes nothing observable, because nothing upstream would have had anything to pass.
Fix
An
ExecutionOriginvalue object threadedbackend → _trigger_handler → _execute_manual_trigger → _execute_schedule_with_lock → create_execution.One object rather than five parallel parameters at four call depths: five positional siblings is exactly how one of them silently stops being forwarded, which is the failure mode already on display here.
Cron ticks stay NULL. Attributing an autonomous fire to, say, the schedule's owner would make the column actively misleading — a blank reads as "unknown", a wrong name does not. Pinned by a test, because it is the half of the behaviour that must not change.
Two more paths the issue's fix would have left blank
Neither is mentioned in the issue, and both would have stayed anonymous after a literal reading of it:
triggered_by='retry') inherits the original run's origin. A retry has no caller of its own, but it exists only because someone started the original; a chain that drops the initiator makes attempt 1 the only attributable attempt. The read is fail-open — an audit lookup must not be able to stop a retry from running.triggered_by='reminder') inherits the provenance feat: agent self-reminders — agent-callable one-shot deferred self-trigger ("remind me to do X later") #1296 already persisted (owner_id,created_by_email,source_agent_name,source_mcp_key_id). This needed those four fields surfaced on the scheduler'sRemindermodel; the mapper guards them the way_row_to_executionguards its optional columns, so a table predating them doesn't take the fire path down.Hardening the issue didn't ask for
Boundary validation of the trigger body. The scheduler's trigger endpoint takes an untrusted JSON body (as it already did for
triggered_by).source_user_idis dropped rather than coerced when it is not an int —boolIS anintin Python, soTruewould have persisted as user1: a real account attributed to a run it had nothing to do with. Strings are length-capped and blank-to-None, so""andNULLare not two spellings of "unknown". A malformed body costs attribution, never the run.Precedence flipped versus
chat.py. The backend prefers the validatedcurrent_user.agent_nameover the rawX-Source-Agentheader:Header-first is fine for a collaboration hint; in an audit column it would let an agent-scoped caller pin its run on a sibling agent. The header now fills only the case where it is the sole signal (a user-scoped key, whose
agent_nameisNone).MCP third surface (Invariant #13).
trigger_agent_schedulenow forwards the same origin headerschat()already sends. Without it an MCP-triggered run attributes to the key owner but not to which key or which agent fired it — the part that identifies the actor when one human owns many of both. The genericextraHeadershook this required explicitly refusesAuthorization, so an attribution convenience can't become an auth-substitution surface.Compatibility
Backward compatible in both rolling-deploy directions:
triggered_bystill read as before;All five parameters are keyword-optional, so an un-migrated
create_executioncall site still inserts a valid row.Verification
tests/unit/test_1970_execution_origin.py— 27 checks, 25 of which fail against the pre-fix tree:The 2 that pass either way are deliberate: the cron-stays-NULL guard and the optional-parameter back-compat guard both assert behaviour the fix must preserve, so passing before and after is the correct result for them.
The DB assertions run against a real temp SQLite file whose table declares the five columns — a mock would have happily accepted the pre-fix signature, and a table without the columns would let a regressed INSERT pass by having nothing to fill.
Also green:
tsc --noEmiton the MCP server, and the scheduler-adjacent suites (test_1808,test_schedule_status_observability,test_1945,test_1713— 70 tests).Acceptance criteria
create_execution()accepts and writes all fivesource_*columns_trigger_handlerreads it and passes it downSibling issues
#1968 (
execution_id: undefined) and #1969 (lock-denied cron tick leaves no record) are untouched here. The issue notes both are "fixed most cleanly by creating the execution record synchronously in_trigger_handler()" — a restructure of the trigger path that this PR deliberately does not attempt, so the attribution fix can land without carrying that risk. TheExecutionOriginobject is already in scope at_trigger_handler, so it composes with that change rather than competing with it.Related to #1970
🤖 Generated with Claude Code
Fixes #1970