feat(model-routing): add per-call selection, strategies, and ordered fallback - #9
Draft
JackYPCOnline wants to merge 2 commits into
Draft
feat(model-routing): add per-call selection, strategies, and ordered fallback#9JackYPCOnline wants to merge 2 commits into
JackYPCOnline wants to merge 2 commits into
Conversation
JackYPCOnline
force-pushed
the
feature/model-routing-selection
branch
from
July 27, 2026 18:30
d7ce19b to
766e13c
Compare
JackYPCOnline
force-pushed
the
feature/model-routing-selection
branch
9 times, most recently
from
July 27, 2026 21:19
e79647a to
fdd370b
Compare
…fallback Add a RoutingStrategy protocol and RoutingContext, and a selection middleware on InvokeModelStage that picks a candidate per invocation (cached in invocation_state), sets the per-call model, and runs before the agentic capability middleware. Add router-owned ordered fallback: on a model failure the retry strategy did not re-request, the router advances to the next untried candidate in declaration order, resets the retry budget, and requests another attempt. Register the fallback hook after ModelRetryStrategy. Bundle two local strategies: FallbackStrategy (select-first, the default) and ContextFitStrategy (context-window capacity). Add a public ModelRetryStrategy.reset() so the router can reset the budget when advancing. Refs: strands-agents#364
JackYPCOnline
force-pushed
the
feature/model-routing-selection
branch
from
July 28, 2026 01:30
fdd370b to
28c2633
Compare
Initialize the router plugin once before capability middleware so it owns selection, fallback, and cleanup registration. Validate cached state against both router and agent ownership, reject malformed state, and reject duplicate candidate instances.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Phase C of model routing, stacked on Phase B (
feature/model-routing-core). Where Phase B resolves a single model at construction, this adds per-call selection and ordered fallback, matching the P0 "router core and fallback" + local strategy items in the design (team/designs/0016-model-routing.md).A
RoutingStrategychooses a candidate per model call; a selection middleware runs it once per invocation (cached ininvocation_state), sets the per-call model, and is registered before the agentic capability middleware. Independently, the router installs an ordered-fallback hook: when the selected model's retries are exhausted, it advances to the next untried candidate in declaration order and requests another attempt. This composes with the existingModelRetryStrategyrather than replacing it.Public API (provisional; package is internal pending API review)
RoutingStrategy(Protocol):async select(context) -> RoutingCandidate; the result must be one of the candidates.RoutingContext:messages,system_prompt,tool_specs,candidates,invocation_state.ModelRouter(models, *, strategy=None)—strategydefaults toFallbackStrategy(selects the first candidate). Selection recurses into nested routers.FallbackStrategy(select-first) andContextFitStrategy— routes by capacity, estimating tokens with each candidate's owncount_tokensand treating a candidate as fitting when usage stays withinthresholdof its window (default: the conversation manager'sDEFAULT_COMPRESSION_THRESHOLD, so a fit won't immediately trip proactive compression). Picks the smallest fitting window, the largest when none fit; reuses the sharedDEFAULT_CONTEXT_WINDOW_LIMITfor models with no declared window and is best-effort if a count fails.Follow-ups (not in this PR)
Notes
Stacked on
feature/model-routing-core; review that PR first. Adds one small change to shared retry code (ModelRetryStrategy.reset()) so the router can reset the retry budget when advancing. Integration tests (real Bedrock) cover context-fit selection and fallback past a broken model.