Plan: Implement retry policies with exponential backoff and dead-letter handling (5.2.3)

[leynos]

Summary

This draft pull request delivers the ExecPlan for backend roadmap item 5.2.3:
retry policies with exponential backoff and dead-letter handling for the
Apalis (PostgreSQL) queue adapter. It is a planning-only change; no
implementation begins until the plan is approved.

Plan document (branch-wise link):
docs/execplans/5-2-3-retry-policies-with-exponential-backoff-and-dead-letter-handling.md

The plan proposes:

• A pure domain retry policy (JobRetryPolicy) computing bounded, jittered
  exponential backoff under a hard total-backoff budget, with a
  retryable/fatal error classification, executed in production via a
  delegating tower::retry::Policy so the verified function is the running
  function.
• A DeadLetterQueue domain port with a PostgreSQL adapter over the
  apalis.jobs table: ordered listing, guarded compare-and-swap requeue
  with provenance, and guarded discard-as-delete.
• Storage-level dead-lettering of fatal failures, compensating for the
  incomplete abort-to-Killed path in apalis-postgres 1.0.0-rc.6.
• Verification via rstest unit tests, proptest invariants, googletest
  and pretty_assertions assertions, insta snapshot stability, and
  rstest-bdd behavioural scenarios against embedded PostgreSQL, plus
  rc-coupling canary tests to catch upstream drift on the eventual 1.0-final
  pin bump.
• Two ADRs (retry mechanism; dead-letter representation), an operator
  runbook, and documentation updates, with worker consumption and
  deployment explicitly deferred to roadmap 5.3.1.

The draft was revised after a six-lens expert design review (structure,
contracts, scaling, operations, alternatives, and long-term viability);
the revision note at the end of the plan summarizes the changes.

References

• Lody session: https://lody.ai/leynos/sessions/393f0df6-4a19-41c6-a76a-35fb9e814fa0

🤖 Generated with Claude Code

Summary by Sourcery

Documentation:

• Introduce a detailed ExecPlan outlining retry policy design, dead-letter queue semantics, and associated operational and testing guidance for backend roadmap item 5.2.3.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 2d050341-e3b8-43dd-926c-6c99a89ad059

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[sourcery-ai]

Reviewer's Guide

Planning-only ExecPlan document for backend roadmap item 5.2.3, introducing a domain-level retry policy with bounded exponential backoff and jitter, a DeadLetterQueue port with a PostgreSQL adapter over apalis.jobs, storage-level fatal dead-lettering, verification strategy (unit/property/BDD/integration tests), ADRs, and documentation/runbook updates—no runtime behaviour changes yet.

Sequence diagram for planned retry and fatal dead-letter handling

sequenceDiagram
    participant Worker
    participant JobHandler
    participant JobRetryPolicy
    participant RetryLayer
    participant DeadLetterQueueAdapter
    participant ApalisJobs

    Worker->>RetryLayer: process_job()
    RetryLayer->>JobHandler: handle()
    JobHandler-->>RetryLayer: Err(JobErrorKind_Retryable)
    RetryLayer->>JobRetryPolicy: decide(attempt, Retryable, jitter)
    JobRetryPolicy-->>RetryLayer: RetryDecision_Retry(delay)
    RetryLayer->>RetryLayer: wait_backoff(delay)
    RetryLayer->>JobHandler: handle() (retry)

    alt [fatal error]
      JobHandler-->>RetryLayer: Err(JobErrorKind_Fatal)
      RetryLayer->>JobRetryPolicy: decide(attempt, Fatal, jitter)
      JobRetryPolicy-->>RetryLayer: RetryDecision_DeadLetter
      RetryLayer->>DeadLetterQueueAdapter: mark_fatal_dead(job_id)
      DeadLetterQueueAdapter->>ApalisJobs: UPDATE status = Killed
      DeadLetterQueueAdapter-->>RetryLayer: Ok
    end

Loading

File-Level Changes

Change	Details	Files
Add detailed ExecPlan document for implementing retry policies with bounded exponential backoff, fatal/retryable classification, and dead-letter handling on the Apalis PostgreSQL queue adapter, including interfaces, invariants, milestones, verification obligations, and ADR/runbook work.	Define domain-level JobRetryPolicy concept with bounded, jittered exponential backoff, max_attempts, and total-backoff budget requirements, plus JobErrorKind and RetryDecision classification to be executed via a delegating tower::retry::Policy in adapters. Specify DeadLetterQueue domain port and PostgreSQL adapter design over apalis.jobs for listing, guarded compare-and-swap requeue with provenance, and guarded discard-as-delete, with dead predicate based on Killed or exhausted Failed rows. Capture constraints, tolerances, risks, external Apalis axioms, and rc-coupling canaries, including AXIOM-FACADE, AXIOM-ATTEMPT, AXIOM-ACK, and explicit fatal dead-lettering requirements compensating for missing abort-to-Killed semantics in apalis-postgres rc.6. Lay out verification plan with property tests for backoff invariants, BDD scenarios on embedded PostgreSQL for retry/dead-letter behaviour, and rc-canary tests to catch upstream drift, plus Makefile gate usage. Define plan-of-work milestones (EP-M0–EP-M4) covering approval/compile probe, domain policy implementation, Apalis retry wiring and bounded attempts, dead-letter port and adapter implementation, and ADR/documentation/runbook updates. Record architectural decisions (ADR-002, ADR-003) and a decision log around using Apalis's retry facade with a delegating Policy, in-process backoff bounded by TOTAL_BACKOFF_BUDGET, dead letters as apalis.jobs rows with guarded transitions, storage-level fatal kill, optional partial index, and per-queue retry policy configuration.	docs/execplans/5-2-3-retry-policies-with-exponential-backoff-and-dead-letter-handling.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[codescene-access]

No quality gates enabled for this code.