Skip to content

feat(genai): production-grade model-agnostic FallbackPolicy middleware#2771

Open
giterinhub wants to merge 3 commits into
googleapis:mainfrom
giterinhub:feature/add-fallback-policy
Open

feat(genai): production-grade model-agnostic FallbackPolicy middleware#2771
giterinhub wants to merge 3 commits into
googleapis:mainfrom
giterinhub:feature/add-fallback-policy

Conversation

@giterinhub

Copy link
Copy Markdown

Summary & Context

This PR introduces and enhances the Model-Agnostic Fallback Middleware (FallbackPolicy) for the google-genai Python SDK.

The policy enables automatic failovers across dynamic model sequences and Vertex AI regions when encountering retryable API errors (e.g. 429 Rate Limit, 503 Service Unavailable, 504 Gateway Timeout).


Key Improvements & Enhancements

  1. Universal Exception Code Extraction:

    • _extract_status_code dynamically inspects APIError.code, exc.status_code, and exc.response.status_code, enabling seamless compatibility across google-genai, httpx, requests, and google.api_core.
  2. Telemetry & Audit Callback (on_fallback):

    • Accepts an optional on_fallback(exc, attempt_idx, next_payload) hook allowing downstream applications to record metrics (Prometheus, OpenTelemetry, Datadog) or custom failover audit logs.
  3. Configurable Backoff (backoff_factor):

    • Supports exponential backoff delays (backoff_factor * 2^(attempt-1)) for both synchronous (time.sleep) and asynchronous (asyncio.sleep) execution paths.
  4. Payload Immutability Guarantee:

    • Each attempt uses copy.deepcopy(kwargs) to ensure nested objects (contents, config) remain completely unmutated across retry iterations.
  5. Multi-Dimensional Fallback Matrix:

    • Sequential matrix evaluation when both fallback_models and fallback_locations are specified, evaluated up to max_retries.

Verification & Testing

  • Ruff Linting: python3 -m ruff check google/genai/_fallback.py google/genai/tests/test_fallback.py google/genai/tests/test_fallback_unittest.py -> Passed (0 errors)
  • Ruff Formatting: python3 -m ruff format --check google/genai/_fallback.py google/genai/tests/test_fallback.py google/genai/tests/test_fallback_unittest.py -> Passed (0 unformatted files)
  • Pytest Suite: pytest google/genai/tests/test_fallback.py -v -> 13 / 13 Passed
  • Unittest Suite: python3 google/genai/tests/test_fallback_unittest.py -> 12 / 12 Passed
  • Concurrency Stress Test: 100 concurrent async tasks executed through execute_async via asyncio.gather().

Fixes googleapis#1288
Closes googleapis#1288

- Add automatic status code extraction across APIError, httpx, requests, and response objects
- Add backoff_factor parameter supporting configurable exponential backoff delays
- Add on_fallback telemetry callback hook for custom audit logging and metrics
- Add comprehensive pytest and unittest test suites covering max retries threshold, non-retryable fast-failing error codes, region+model matrix fallbacks, payload mutability audits, and 100-task high-concurrency async stress tests
@giterinhub
giterinhub force-pushed the feature/add-fallback-policy branch from ac02e12 to 94139be Compare July 23, 2026 18:08
@Venkaiahbabuneelam Venkaiahbabuneelam self-assigned this Jul 24, 2026
@Venkaiahbabuneelam Venkaiahbabuneelam added the size:XL Code changes > 100 lines label Jul 24, 2026
…gleapis#1288)

- Narrow return type in _extract_status_code to prevent no-any-return mypy error
- Add Dict[str, Any] type parameters for generic dict annotations
- Explicitly export APIError in __all__ to resolve attr-defined mypy error in __init__.py
@giterinhub
giterinhub force-pushed the feature/add-fallback-policy branch from 40f8786 to 43d3116 Compare July 24, 2026 08:04
@giterinhub

Copy link
Copy Markdown
Author

Note on mypy 3.11 static checking:
mypy flags 9 pre-existing type errors in google/genai/types.py and google/genai/tunings.py

caused by static AST evaluation of optional try: 
from IPython import display
except
ImportError: IPython_display = None branches and untyped IPython calls when strict checking individual files.

To keep this PR strictly targeted to FallbackPolicy, those 9 pre-existing IPython type fixes have been isolated into a separate PR: #2773.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL Code changes > 100 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants