Stall detection now fires while agent is busy - #252
Merged
Conversation
Goal progress renderer now clips run summaries to display length (120 chars), moved from orchestrator so stored events retain full text while terminal output stays single-line. Blocker and reason prose are deliberately never clipped. Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Previously a stall was only detectable on a continuation turn that made zero tool calls. This missed the dominant real-world failure mode: the three worst stalls in the corpus (31, 24, and 54 turns) made tool calls every turn while the goal had become unsatisfiable, so the detector never fired. A second trigger, independent of tool activity, now runs alongside the original. It uses a cheap deterministic pre-filter over recent evaluator reasons every turn (no model call) and only consults the stall judge when the pre-filter trips, preserving the invariant that a mechanical condition alone never trips a stall. The stall judge was reframed from binary yes/no into a verdict taxonomy: resolvable, time-locked, structure-locked, or history-locked. The last three all mean stop now and name the dead end, which users need to rewrite the goal. The judge's prompts were split by trigger, since the original prompt hard-codes the assertion that the assistant took no tool actions. The CLI's stalled-state wording now derives from that verdict instead of from a distinct-blocker count. The old counter compared strings with only whitespace and case normalization, so an evaluator that rephrased a blocker each turn produced a distinct signature per turn, yielding a false "flailing" verdict. The evaluator reason list is now capped before being passed to the summary model; it previously grew unbounded and a 54-turn run shipped all 54 reasons. Replayed against the three recorded reason chains: new pre-filter first consults judge at turn 14 (stalled at 31), turn 12 (stalled at 24), and turn 18 (stalled at 54). On achieved organic runs from the same sessions: zero false trips. Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
goalify composes a /goal stop-condition from the current conversation and lints it against known termination-failure patterns before showing it to the user. It has a hard prerequisite — it is only meaningful when the CLI provides the /goal command — so co-locating it with the CLI satisfies that prerequisite by construction rather than by configuration. The skill lives at amplifier_app_cli/data/skills/goalify/ and is registered by appending a package-relative path in _ensure_default_skills_dirs(). Resolution is by installed package location, never by git URI: a git reference would have co-located the files in the repo while still versioning them independently of the code they depend on, which defeats the purpose. The wheel build declares packages = ["amplifier_app_cli"], so repo-root files are excluded from the distribution entirely. The skill directory is appended in Python rather than declared through a bundle's tool-skills config, because bundle-level skills config replaces rather than appends — the same hazard _ensure_default_skills_dirs was originally written to work around. Verified that workspace and user skills still load alongside the packaged one. The skill body contains only instructions to the invoking agent. Authoring rationale and the lint-rule evidence base were moved to sibling PROVENANCE.md, which load_skill does not read. The lint output table now reports "no known pattern detected" rather than "PASS", because a column of PASS reads as a validation claim the linter cannot support — it detects known patterns from a finite corpus. A repo-root AGENTS.md records the decision rule for what may live in amplifier_app_cli/data/ versus an external bundle: the asset must depend on something the CLI uniquely provides, no non-CLI host would want it, and unconditional triggers belong in the bundle layer while gated ones stay in Python. Assets resolve by package path only, and this location is auto-loaded for every user in every session, so its token budget discipline is stricter than anywhere else. Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch brings three related improvements to the goal loop's stall detection and observability:
Goal progress payload hygiene — The orchestrator now emits the goal condition on every
orchestrator:goal_progressevent in its fully-expanded form (after file references are resolved), so anyone reading the event sees exactly what the evaluator judged. A schema-version field was added so consumers can distinguish an absent key from a null one. The run summary is now stored in full and truncated only at display time.Evaluator input hygiene — Evaluator input was previously unbounded; a large tool result inside the 40-message window shipped in full every turn. Input is now bounded, tool-call arguments are included (so the evaluator sees what was asked), and the system prompt discloses to the evaluator what it cannot see.
Stall detection now fires while agent is busy — Previously a stall was only detectable on a continuation turn that made zero tool calls. This missed the dominant real-world failure mode: the three worst stalls in the corpus (31, 24, and 54 turns) made tool calls every turn while the goal had become unsatisfiable, so the detector never fired.
A second trigger, independent of tool activity, now runs alongside the original. It uses a cheap deterministic pre-filter over recent evaluator reasons every turn (no model call) and only consults the stall judge when the pre-filter trips, preserving the invariant that a mechanical condition alone never trips a stall.
The stall judge was reframed from binary yes/no into a verdict taxonomy: resolvable, time-locked, structure-locked, or history-locked. The last three all mean stop now and name the dead end, which users need to rewrite the goal. The judge's prompts were split by trigger, since the original prompt hard-codes the assertion that the assistant took no tool actions.
The CLI's stalled-state wording now derives from that verdict instead of from a distinct-blocker count. The old counter compared strings with only whitespace and case normalization, so an evaluator that rephrased a blocker each turn produced a distinct signature per turn, yielding a false "flailing" verdict.
The evaluator reason list is now capped before being passed to the summary model; it previously grew unbounded and a 54-turn run shipped all 54 reasons.
Verification
Files Changed
amplifier_app_cli/goal_progress_hook.py— stall detection reframe, verdict taxonomy, input bounding, CLI wordingtests/test_goal_progress_hook.py— coverage for verdict detection, stall judge confirmation, reason capping