Skip to content

feat(model-routing): add per-call selection, strategies, and ordered fallback - #9

Draft
JackYPCOnline wants to merge 2 commits into
feature/model-routing-corefrom
feature/model-routing-selection
Draft

feat(model-routing): add per-call selection, strategies, and ordered fallback#9
JackYPCOnline wants to merge 2 commits into
feature/model-routing-corefrom
feature/model-routing-selection

Conversation

@JackYPCOnline

@JackYPCOnline JackYPCOnline commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Description

Phase C of model routing, stacked on Phase B (feature/model-routing-core). Where Phase B resolves a single model at construction, this adds per-call selection and ordered fallback, matching the P0 "router core and fallback" + local strategy items in the design (team/designs/0016-model-routing.md).

A RoutingStrategy chooses a candidate per model call; a selection middleware runs it once per invocation (cached in invocation_state), sets the per-call model, and is registered before the agentic capability middleware. Independently, the router installs an ordered-fallback hook: when the selected model's retries are exhausted, it advances to the next untried candidate in declaration order and requests another attempt. This composes with the existing ModelRetryStrategy rather than replacing it.

Public API (provisional; package is internal pending API review)

from strands.models.routing import ModelRouter, RoutingCandidate, FallbackStrategy, ContextFitStrategy

# Ordered fallback (default strategy): try sonnet, fall over to opus on failure.
agent = Agent(model=ModelRouter(models=[sonnet, opus]))

# Capacity-based local selection.
agent = Agent(model=ModelRouter(models=[haiku, sonnet, opus], strategy=ContextFitStrategy()))
  • RoutingStrategy (Protocol): async select(context) -> RoutingCandidate; the result must be one of the candidates.
  • RoutingContext: messages, system_prompt, tool_specs, candidates, invocation_state.
  • ModelRouter(models, *, strategy=None)strategy defaults to FallbackStrategy (selects the first candidate). Selection recurses into nested routers.
  • Ordered fallback is always active, under any strategy: on an unretried model failure the router advances through untried candidates in declaration order, resetting the retry budget each hop, until one succeeds or all are exhausted.
  • Bundled strategies: FallbackStrategy (select-first) and ContextFitStrategy — routes by capacity, estimating tokens with each candidate's own count_tokens and treating a candidate as fitting when usage stays within threshold of its window (default: the conversation manager's DEFAULT_COMPRESSION_THRESHOLD, so a fit won't immediately trip proactive compression). Picks the smallest fitting window, the largest when none fit; reuses the shared DEFAULT_CONTEXT_WINDOW_LIMIT for models with no declared window and is best-effort if a count fails.

Follow-ups (not in this PR)

  • Model-driven routing (Phase D): a judge model classifies the request and names a candidate.
  • P1: cost-aware, cache-affinity (sticky), classifier/semantic.

Notes

Stacked on feature/model-routing-core; review that PR first. Adds one small change to shared retry code (ModelRetryStrategy.reset()) so the router can reset the retry budget when advancing. Integration tests (real Bedrock) cover context-fit selection and fallback past a broken model.

Draft: per-call selection + ordered fallback, pending API review and the model-driven follow-up.

@github-actions github-actions Bot added the enhancement New feature or request label Jul 27, 2026
@JackYPCOnline
JackYPCOnline force-pushed the feature/model-routing-selection branch from d7ce19b to 766e13c Compare July 27, 2026 18:30
@JackYPCOnline JackYPCOnline changed the title feat(model-routing): add per-call selection strategy and middleware feat(model-routing): add per-call selection, strategies, and ordered fallback Jul 27, 2026
@JackYPCOnline
JackYPCOnline force-pushed the feature/model-routing-selection branch 9 times, most recently from e79647a to fdd370b Compare July 27, 2026 21:19
…fallback

Add a RoutingStrategy protocol and RoutingContext, and a selection
middleware on InvokeModelStage that picks a candidate per invocation
(cached in invocation_state), sets the per-call model, and runs before
the agentic capability middleware.

Add router-owned ordered fallback: on a model failure the retry
strategy did not re-request, the router advances to the next untried
candidate in declaration order, resets the retry budget, and requests
another attempt. Register the fallback hook after ModelRetryStrategy.

Bundle two local strategies: FallbackStrategy (select-first, the
default) and ContextFitStrategy (context-window capacity). Add a
public ModelRetryStrategy.reset() so the router can reset the budget
when advancing.

Refs: strands-agents#364
@JackYPCOnline
JackYPCOnline force-pushed the feature/model-routing-selection branch from fdd370b to 28c2633 Compare July 28, 2026 01:30
Initialize the router plugin once before capability middleware so it owns selection, fallback, and cleanup registration. Validate cached state against both router and agent ownership, reject malformed state, and reject duplicate candidate instances.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant