Fix dangling retry processor and add resilience/validation to AI client

[balisdev]

Summary

• startRetryProcessor #139 — startRetryProcessor fires tick without await: the retry processor started its first tick with a fire-and-forget call and gated the recursive loop with a plain boolean. Restarting the processor while an earlier tick was still in flight (e.g. stopRetryProcessor() immediately followed by startRetryProcessor()) let the stale tick see the flag flipped back on and reschedule itself, leaving two overlapping processing loops running. Fixed by awaiting the first tick and gating scheduling with a generation counter instead of a boolean, so a stale in-flight tick can never reschedule itself after a restart.
• AI #140 — AI client fetch has no timeout: this is a duplicate of AI client fetch has no timeout #74, which was already fixed and merged (1535a22, PR fix(quizzes): add configurable timeout to AI service fetch #85) — the fetch call already has an AbortController with a configurable AI_TIMEOUT_MS timeout. No functional change needed; adding a regression test to lock in the behavior and closing this as a duplicate.
• AI #141 — AI client has no resilience patterns: unlike Stellar calls, the AI service client had no retry or circuit breaker, so a persistent outage hung every quiz request for the full timeout duration. Extracted the existing Stellar retry/circuit-breaker logic into a reusable factory (src/utils/resilience.ts) and wired up an independent retry + circuit breaker instance for the AI client (bounded to 3 attempts per call so it still fails predictably rather than retrying forever). src/stellar/resilience.ts keeps its exact public API, now backed by the shared factory — no behavior change for Stellar.
• AI #142 — AI client does not validate response shape: response.json() was cast straight to AiQuizResponse with no runtime check, so a malformed body could crash instead of falling back gracefully. Added a zod schema and replaced the unsafe cast with safeParse, throwing a descriptive error on shape mismatch.

Both AI failure modes are still caught by quizService.generateQuiz's existing fallback to placeholder questions, so no caller changes were needed.

Test plan

• npm run typecheck — clean
• npm run lint — clean (only pre-existing warnings elsewhere)
• New regression test for the retry-processor restart race (fails on the old code, passes with the fix)
• New AI client test suite covering: valid response parsing, malformed response rejection (AI #142), retry on transient failure and circuit breaker opening (AI #141), and timeout behavior (AI #140 regression)
• Existing resilience.test.ts and stellar/client.ts behavior unchanged after the factory extraction
• Full suite run; pre-existing unrelated failures (no local Redis, Fastify version mismatch, Stellar account validation) confirmed present on main before these changes too

Closes #139, Closes #140, Closes #141, Closes #142

[drips-wave]

@balisdev Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits