Skip to content

channel-memory: add async LLM digest worker and cost caps #268

Description

@mostlydev

Motivation

Deterministic channel-memory digesting is the safety baseline, but the Phase 2 design also needs an off-hot-path LLM worker for higher-quality topic and sequence rollups. This must not add latency to channel-awareness serving or blur exact source provenance.

Depends on #265. It can land after #267 if deterministic raw_window+digest ships first.

Scope

  • Add an async worker that processes queued source-message windows into structured digest blocks.
  • Keep hard events verbatim or near-verbatim; use lossy prose only for sparse topic_rollup and sequence_rollup style blocks.
  • Require structured JSON output and reject malformed or provenance-free results.
  • Cache work by source message ids plus content hashes.
  • Store model/provider/version, generated time, sparse flag, source window, and source-message provenance for every generated block.
  • Enforce conservative per-channel/per-pod daily call and cost caps with clear telemetry.
  • Fall back to deterministic-only digest output when the worker is disabled, over budget, or failing.

Out of scope

  • The base SQLite adapter schema beyond extensions required for queued work and processor metadata.
  • claw-wall push ingest and channel-awareness serving changes.
  • Public YAML tuning surfaces beyond internal environment overrides.
  • Release pin, changelog, or nav changes.

Acceptance

  • Unit tests cover queue ordering, malformed LLM output rejection, cache reuse by content hash, cost-cap enforcement, and deterministic fallback.
  • Processor tests prove edit/delete/forget marks affected derived blocks stale or dirty.
  • Fixture tests show older verbose channel material compresses into sparse blocks while hard events remain faithful.
  • No LLM call occurs during /digest hot-path lookup.
  • Public docs and examples stay deployment-agnostic.

Refs #262
Refs #263

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions