Skip to content

Stall detection now fires while agent is busy - #252

Merged
bkrabach merged 3 commits into
mainfrom
feature/goal-progress-payload-hygiene
Aug 3, 2026
Merged

Stall detection now fires while agent is busy#252
bkrabach merged 3 commits into
mainfrom
feature/goal-progress-payload-hygiene

Conversation

@bkrabach

@bkrabach bkrabach commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

This branch brings three related improvements to the goal loop's stall detection and observability:

  1. Goal progress payload hygiene — The orchestrator now emits the goal condition on every orchestrator:goal_progress event in its fully-expanded form (after file references are resolved), so anyone reading the event sees exactly what the evaluator judged. A schema-version field was added so consumers can distinguish an absent key from a null one. The run summary is now stored in full and truncated only at display time.

  2. Evaluator input hygiene — Evaluator input was previously unbounded; a large tool result inside the 40-message window shipped in full every turn. Input is now bounded, tool-call arguments are included (so the evaluator sees what was asked), and the system prompt discloses to the evaluator what it cannot see.

  3. Stall detection now fires while agent is busy — Previously a stall was only detectable on a continuation turn that made zero tool calls. This missed the dominant real-world failure mode: the three worst stalls in the corpus (31, 24, and 54 turns) made tool calls every turn while the goal had become unsatisfiable, so the detector never fired.

    A second trigger, independent of tool activity, now runs alongside the original. It uses a cheap deterministic pre-filter over recent evaluator reasons every turn (no model call) and only consults the stall judge when the pre-filter trips, preserving the invariant that a mechanical condition alone never trips a stall.

    The stall judge was reframed from binary yes/no into a verdict taxonomy: resolvable, time-locked, structure-locked, or history-locked. The last three all mean stop now and name the dead end, which users need to rewrite the goal. The judge's prompts were split by trigger, since the original prompt hard-codes the assertion that the assistant took no tool actions.

    The CLI's stalled-state wording now derives from that verdict instead of from a distinct-blocker count. The old counter compared strings with only whitespace and case normalization, so an evaluator that rephrased a blocker each turn produced a distinct signature per turn, yielding a false "flailing" verdict.

    The evaluator reason list is now capped before being passed to the summary model; it previously grew unbounded and a 54-turn run shipped all 54 reasons.

Verification

  • Tests: 74 goal tests passed (full app-cli suite has 14 pre-existing failures in test_session_spawner_subprocess, test_cleanup_observability, test_always_render_final_response, test_handler_methods, test_session_lifecycle_events — confirmed present on main before any of this work)
  • Real-world measurement: Replayed against the three recorded reason chains, new pre-filter first consults judge at turn 14 (stalled at 31), turn 12 (stalled at 24), and turn 18 (stalled at 54). On achieved organic runs from the same sessions: zero false trips.

Files Changed

  • amplifier_app_cli/goal_progress_hook.py — stall detection reframe, verdict taxonomy, input bounding, CLI wording
  • tests/test_goal_progress_hook.py — coverage for verdict detection, stall judge confirmation, reason capping

Goal progress renderer now clips run summaries to display length (120 chars),
moved from orchestrator so stored events retain full text while terminal output
stays single-line. Blocker and reason prose are deliberately never clipped.

Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Previously a stall was only detectable on a continuation turn that made zero
tool calls. This missed the dominant real-world failure mode: the three worst
stalls in the corpus (31, 24, and 54 turns) made tool calls every turn while
the goal had become unsatisfiable, so the detector never fired.

A second trigger, independent of tool activity, now runs alongside the
original. It uses a cheap deterministic pre-filter over recent evaluator
reasons every turn (no model call) and only consults the stall judge when the
pre-filter trips, preserving the invariant that a mechanical condition alone
never trips a stall.

The stall judge was reframed from binary yes/no into a verdict taxonomy:
resolvable, time-locked, structure-locked, or history-locked. The last three
all mean stop now and name the dead end, which users need to rewrite the
goal. The judge's prompts were split by trigger, since the original prompt
hard-codes the assertion that the assistant took no tool actions.

The CLI's stalled-state wording now derives from that verdict instead of from
a distinct-blocker count. The old counter compared strings with only
whitespace and case normalization, so an evaluator that rephrased a blocker
each turn produced a distinct signature per turn, yielding a false "flailing"
verdict.

The evaluator reason list is now capped before being passed to the summary
model; it previously grew unbounded and a 54-turn run shipped all 54 reasons.

Replayed against the three recorded reason chains: new pre-filter first
consults judge at turn 14 (stalled at 31), turn 12 (stalled at 24), and turn
18 (stalled at 54). On achieved organic runs from the same sessions: zero
false trips.

Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
goalify composes a /goal stop-condition from the current conversation
and lints it against known termination-failure patterns before showing
it to the user. It has a hard prerequisite — it is only meaningful when
the CLI provides the /goal command — so co-locating it with the CLI
satisfies that prerequisite by construction rather than by configuration.

The skill lives at amplifier_app_cli/data/skills/goalify/ and is
registered by appending a package-relative path in
_ensure_default_skills_dirs(). Resolution is by installed package
location, never by git URI: a git reference would have co-located the
files in the repo while still versioning them independently of the code
they depend on, which defeats the purpose. The wheel build declares
packages = ["amplifier_app_cli"], so repo-root files are excluded
from the distribution entirely.

The skill directory is appended in Python rather than declared through
a bundle's tool-skills config, because bundle-level skills config
replaces rather than appends — the same hazard _ensure_default_skills_dirs
was originally written to work around. Verified that workspace and user
skills still load alongside the packaged one.

The skill body contains only instructions to the invoking agent. Authoring
rationale and the lint-rule evidence base were moved to sibling
PROVENANCE.md, which load_skill does not read.

The lint output table now reports "no known pattern detected" rather
than "PASS", because a column of PASS reads as a validation claim the
linter cannot support — it detects known patterns from a finite corpus.

A repo-root AGENTS.md records the decision rule for what may live in
amplifier_app_cli/data/ versus an external bundle: the asset must depend
on something the CLI uniquely provides, no non-CLI host would want it,
and unconditional triggers belong in the bundle layer while gated ones
stay in Python. Assets resolve by package path only, and this location
is auto-loaded for every user in every session, so its token budget
discipline is stricter than anywhere else.

Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@bkrabach
bkrabach merged commit 00a49ad into main Aug 3, 2026
7 checks passed
@bkrabach
bkrabach deleted the feature/goal-progress-payload-hygiene branch August 3, 2026 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants