Skip to content

feat: async prompt dispatch (kind:"prompt" leaf) - #170

Merged
pdettori merged 10 commits into
rossoctl:mainfrom
pdettori:docs/async-prompt-dispatch-spec
Aug 26, 2026
Merged

feat: async prompt dispatch (kind:"prompt" leaf)#170
pdettori merged 10 commits into
rossoctl:mainfrom
pdettori:docs/async-prompt-dispatch-spec

Conversation

@pdettori

Copy link
Copy Markdown
Member

Summary

Adds a kind:"prompt" leaf envelope that dispatches a free-form prompt through the existing async KEDA/Redis-Streams queue, so a /turn-style interaction can be backgrounded and polled for completion — mirroring the existing kind:"solve" precedent.

Governing principle: cleanest design over fewest changes. Exactly one new discriminant is threaded end-to-end — prompt kind → responded result status → text payload — reusing every existing seam (async enqueue, KEDA ScaledJob drain, /runs/status polling, terminal classification). /turn parity comes from a shared exported executeTurn() core (parameterized only by createIfAbsent), with runTurn reduced to a thin wrapper, plus a shared sumBranchUsage helper (no third copy).

Design docs: docs/specs/2026-08-25-async-prompt-dispatch-design.md and docs/adrs/0028-async-prompt-dispatch.md.

What's in it

  • Envelope + result: LeafEnvelope gains kind:"prompt" + prompt; LeafResult gains { status:"responded"; text; usage? } (one status → one payload).
  • Persistence: leaf-result-store persists text: string | null with a dedicated responded branch in toResultRecord.
  • Turn core: executeTurn(input) extracted from runTurn; behavior-preserving — the /turn 404 "no session in backend" contract is preserved bit-for-bit (createIfAbsent:false).
  • Dispatch: runPromptLeaf maps stopReasonresponded (with aborted/error special-cased); wired into runLeaf with an injectable deps.executeTurn.
  • Routing: isPromptEnvelope folds into isRunEnvelope; a bare {sessionId, kind:"prompt", prompt, async:true} enqueues → 202 {status:"accepted"}; handleLeafStatus emits the responded wire shape. Zero /runs route branching change.
  • Boundary guard (review fix): a workloadId on a prompt envelope still gates existence (404 if absent) but its pool selector is intentionally not injected — the API boundary logs a warning rather than injecting a selector executeTurn would silently drop (ADR 0028; prompt leaves inherit /turn's sandbox model, no per-leaf pool isolation).

Testing

Full suite green (make test, EXIT:0): @sh/knative-server 179 passed, harness 201 passed / 3 skipped, all other packages passed. Skips are the pre-existing gated live-model / live-Redis suites — not regressions. Includes new unit coverage for the prompt envelope, the responded record, the max_tokens → responded mapping, and the workload-gated-but-selector-ignored boundary.

Follow-up (non-blocking)

  • Hermetic variant of the executeTurn 404-contract test with an injected backend, so the contract is asserted unconditionally in a Redis-less CI leg (the contract is currently also pinned by server.ts:94).

Resolves #168.

🤖 Generated with Claude Code

Add the design for a `kind:"prompt"` leaf that dispatches a free-form
prompt over the existing async KEDA queue, mirroring the `solve`
precedent. Realizes issue rossoctl#168.

Key decisions (ADR-0028):
- dedicated LeafResult discriminant `{status:"responded";text}`, not a
  `done`+text overload — preserves the union's one-discriminant→one-
  payload invariant
- full `/turn` parity via a shared `executeTurn` core, parameterized only
  by session-open policy (createIfAbsent) so `/turn`'s 404 contract is
  preserved while the leaf gets create-or-resume
- leaf-family model precedence; `/turn` sandbox model (no solve pool lease)

No new queue, worker, scaler, route, or status endpoint; zero `/runs`
route changes.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
Extend the leaf-job-runner processOne return union with "responded" (a
forced consequence of the new LeafResult member; a prompt leaf is a
terminal success that writes a record + acks). leaf-result-store's
responded mapping is a later task.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
…sage

runTurn becomes a thin createIfAbsent:false wrapper over executeTurn, so a
prompt leaf can run the identical /turn stack. The 404-on-missing-session
contract is preserved and now pinned by a regression test.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
DRY the assistant-message branch-usage summation that Task 3 had mirrored
verbatim into executeTurn and runSolveLeaf. One source of truth in run-turn.ts;
run-leaf.ts consumes it over the existing value-import edge (no new cycle).
Both call sites keep their best-effort try/catch guard.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
…ecuteTurn

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
Add 'solved|responded' to poll_status terminal states and a Claim 7 that
dispatches a bare kind:prompt async envelope and asserts status=responded
with non-empty .text. Gated by ASYNC_LIVE_SMOKE=1.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
Address final-review findings for kind:prompt leaves:
- I1: resolveRunWorkload keeps the workloadId existence/404 gate but no
  longer injects sandboxPoolSelector for kind:prompt (executeTurn ignores
  it by design, ADR 0028); logs a warn instead of silently dropping it.
- M2: split the stale spec §2 'no change' row — leaf-job-runner had a
  type-level touch (processOne return union + 'responded').
- M3: add an explicit max_tokens -> responded test for the prompt path.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
Comment thread packages/knative-server/src/server.ts Fixed

@pdettori pdettori left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Reviewed — would APPROVE (self-approval blocked by GitHub, so posting as a comment).

Clean, well-documented async prompt-dispatch feature. Exactly one discriminant (prompt kind → responded status → text payload) threaded end-to-end, reusing every existing seam. Verified "responded" is terminal end-to-end: classifyOutcome uses a default-ack model, so no new branching was needed there. Model-selection precedence, the import type cycle guard, and the selector-drop boundary guard all check out, with thorough unit coverage. All CI green; 9 commits, all signed-off.

One non-blocking suggestion inline (CodeQL log-injection cleanup). The executeTurn err.message.includes("no session in backend") string-match is brittle but you've already noted the hermetic 404-contract test as a follow-up — right call.

Assisted-By: Claude Code

Comment thread packages/knative-server/src/server.ts Outdated
// The workloadId still gates existence (404 above), but its pool selector is intentionally
// ignored here rather than injected and then silently dropped by executeTurn downstream.
if (record.sandboxSelector) {
console.warn(`workload '${body.workloadId}': sandbox pool selector ignored for kind:prompt leaf (ADR 0028)`);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CodeQL flags this as log injection (alert #45). Not practically exploitable — findWorkload is an exact-key lookup and workload names are validated against WORKLOAD_NAME (RFC-1123, ≤50 chars) at creation, so reaching this line means the id is already validated. But logging the stored record.workloadId instead of raw body.workloadId makes that safety self-evident and clears the alert without a suppression.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 6733c12 — the boundary warning now logs the resolved record.workloadId (the exact key findWorkload matched, already validated against WORKLOAD_NAME at creation) instead of the raw body.workloadId. Clears alert #45 without a suppression; no behavior change. The existing I1 test asserts only that the warn fires, so it stayed green.

…on (alert rossoctl#45)

The kind:prompt boundary warning interpolated the raw request field
body.workloadId. Log the resolved record.workloadId instead — the exact
key findWorkload matched, already validated against WORKLOAD_NAME
(RFC-1123, <=50 chars) at creation. Clears the alert without a
suppression and makes the safety self-evident. No behavior change; the
existing I1 test still asserts the warn fires.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
@pdettori
pdettori merged commit a606184 into rossoctl:main Aug 26, 2026
10 checks passed
@pdettori
pdettori deleted the docs/async-prompt-dispatch-spec branch August 26, 2026 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: async free-form prompt dispatch (kind:"prompt") — background /turn over the KEDA queue

2 participants