Add Retry, Backoff, and Request Diagnostics to the HTTP Transport

[cferrys]

Summary

The Python client currently performs each HTTP request once and returns a generic Response(0, ...) for network failures. Add configurable retry/backoff behavior and lightweight request diagnostics to make hlquery integrations more reliable under transient server, network, and load-related failures.

Context

hlquery is positioned as a high-performance search engine/database wrapper around RocksDB, but the Python client transport in lib/request.py is a single-attempt synchronous urllib wrapper. Any temporary URLError, timeout, 429, 502, 503, or 504 immediately surfaces as a failed response with limited context.

This matters most for production search and ingestion paths:

• lib/documents.py exposes import_documents, which may submit large batches.
• lib/search.py exposes latency-sensitive search, vector search, SQL-backed search, and multi-search.
• lib/response.py only preserves status, headers, body, raw body, and a string error, so callers cannot inspect retry attempts, elapsed time, or failure category.

For a high-throughput RocksDB-backed service, transient overload or connection churn should not force every application using the client to hand-roll retry policy and observability.

Proposed Implementation

Add first-class transport resilience options while keeping current defaults backward compatible:

1. Extend client options in utils/config.py and lib/client.py:

   • retries: default 0 or conservative 2
   • retry_backoff_ms: default base delay
   • retry_statuses: default [429, 502, 503, 504]
   • retry_methods: default idempotent methods only, with opt-in support for import/search POST routes

2. Update lib/request.py to:

   • retry eligible transport errors and retryable HTTP statuses
   • use exponential backoff with jitter
   • honor Retry-After when present
   • avoid retrying validation errors, auth errors, and normal 4xx responses
   • track elapsed time and attempt count

3. Extend lib/response.py with non-breaking accessors:

   • get_elapsed_ms()
   • get_attempts()
   • get_retryable()
   • optionally get_error_type()

4. Add focused tests or examples covering:

   • timeout/connection failure retry
   • 503 retry then success
   • non-retryable 400/401
   • disabled retry behavior matching today’s behavior

Impact

This improves reliability for production ingestion, search, vector search, and multi-search workloads without requiring every hlquery user to build their own transport wrapper. It also gives operators better visibility into client-side latency and transient failures, which is critical for tuning a high-performance search system under real traffic.