[Worker] No telemetry outcome for shutdown-released claims (abandoned_shutdown), and TranscriptTriageWorker unchecked for the stranded-Failed window

[Chris0Jeky]

Residuals from #1605 / PR #1631 (posted on the issue thread; re-homed here so they survive the Closes).

1. Both shutdown paths added by PR Release a shutdown-abandoned queue claim instead of charging it a retry #1631 (claim release in the cancellation guard, retry-transition completion under cancellation) record no WorkerItemsProcessed outcome tag. OutboundWebhookDeliveryWorker already has the pattern (its shutdown tag) — add an equivalent abandoned_shutdown / cancelled_requeued outcome so operators can distinguish clean-shutdown releases from real failures in metrics.

2. TranscriptTriageWorker was outside PR Release a shutdown-abandoned queue claim instead of charging it a retry #1631's ownership and was NOT examined for the same defect class: a stranded-Failed row when cancellation lands inside the backoff delay after MarkAsFailed commits, and retry-budget burn on graceful shutdown. Audit it against the two mechanisms fixed in LlmQueueToProposalWorker and port the ReleaseClaim/ApplyRetryTransition idiom if it shares them.

Refs #1605, PR #1631.

[Chris0Jeky]

Third residual: pre-try claim-window cancellation still burns a retry via the sweep

Surfaced by the adversarial review of #1631 (finding M2). Recording it here rather than fixing it in
that PR, which is scoped to the two cancellation windows it already closes.

What #1631 actually guarantees. Its AC-1 is worded "a graceful shutdown mid-item does not
increment RetryCount". That holds only once execution has entered the try block in
ProcessSingleItemAsync — the release guard is a catch on that try, so it can only fire for
cancellation raised inside it.

The gap. Four ct-taking awaits run before that try, all after the row has already been
flipped to Processing. Cancellation observed by any of them throws straight past the release
guard, leaving the row Processing. RecoverStuckProcessingItemsAsync then cannot distinguish it
from a crash and charges the budget (MarkAsFailed, RetryCount++) — the exact accounting #1605
was opened to stop, just through a narrower door.

Sites, at 97806b30 in backend/src/Taskdeck.Api/Workers/LlmQueueToProposalWorker.cs:

Line	Await	Note
276	unitOfWork.LlmQueue.TryClaimProcessingAsync(itemId, expectedUpdatedAt, ct)	The claim itself; cancellation after the UPDATE commits but before the call returns leaves the row Processing with no in-memory handle at all.
285	unitOfWork.LlmQueue.GetByIdAsync(itemId, ct)	Post-claim re-fetch.
315	unitOfWork.AutomationProposals.GetBySourceReferenceAsync(ProposalSourceType.Queue, item.Id.ToString(), ct)	Idempotency guard read.
320	unitOfWork.SaveChangesAsync(ct)	The existing-proposal short-circuit's own MarkAsCompleted write.

Severity: low, and strictly better than before. This is pre-existing, not introduced by #1631 —
that PR narrows the vulnerable window rather than widening it, since everything from
ParseInstructionAsync onward is now covered. The window is also small: 276 and 285 are fast local
SQLite round-trips, and 315/320 only run for the already-has-a-proposal path. The 320 case is the
mildest of the four, because a proposal already exists, so the restart's own idempotency guard
completes the item on the next tick regardless — it costs one sweep-charged retry, not the work.

Possible fix, if it is ever worth it. Move the claim-to-try region inside the same guard, i.e.
open the try immediately after TryClaimProcessingAsync returns true and let
ReleaseClaimOnShutdownAsync cover it (it is already status-guarded — it no-ops unless the row is
still Processing). The 276 case cannot be covered that way, since there is no entity to release
yet; that one needs the sweep either way, or a claim that stamps a shutdown-distinguishable marker.

Filing rather than fixing: it needs its own tests for four distinct cancellation points, and the
payoff is one retry per unlucky shutdown on an already-narrow window.

[Chris0Jeky]

Coordinator disposition (2026-09-06, Codex-session PR carried under the standing D-12 route: fresh-context review, MEDIUM+ fixed in-PR, merged as a merge commit).

PR #2746 (Add shutdown outcome telemetry to LLM queue worker) reviewed FIX-FIRST → fixed in-PR by the coordinator (commit 2f6531cea, worker and resilience classes 67/67): HIGH — the test's MeterListener on the process-global WorkerItemsProcessed counter filtered only on outcome, appended to an unsynchronised list and asserted exact single-element sequences while other test classes drive the same counter in parallel; it now runs in a DisableParallelization collection, filters on worker.name, and locks the list. MEDIUM — abandoned_shutdown was emitted when the row was successfully released and nothing when the release write genuinely failed; the outcomes are now released_on_shutdown and shutdown_release_failed (the latter emitted only from the release write's catch, not when the row had already moved on). Merges when its updated hosted run is green. LOWs on the PR: RecordWorkerOutcome inside the requeue try; CompleteRetryTransitionOnShutdownAsync returns true on a no-op transition.

[Chris0Jeky]

PR #2746 merged as 0b56a12aa at head 763ec6f9c (hosted run green after the coordinator's fixes recorded above).