Horizontal scaling with Redis pub/sub

[karutoil]

Summary

The backend currently runs as a single instance. For horizontal scaling (multiple backend instances behind a load balancer), we need Redis pub/sub to synchronize WebSocket state, cache invalidation, and task scheduling across instances.

Current State

• WebSocket gateway (src/websocket/gateway.ts) tracks agent connections in an in-memory CappedMap
• Agent registry is in-process — only the instance that received the agent handshake can route commands to that agent
• Task scheduler runs in-process — only one instance should schedule tasks to avoid duplicate execution
• Permission caching uses in-memory 30-second TTL — no cross-instance invalidation
• Redis is already in the Docker Compose stack but only used for future pub/sub (not currently connected)
• catalyst-backend/package.json does not include ioredis or any Redis client yet

Requirements

• Add Redis client (ioredis) to backend dependencies
• Agent registry: publish agent connection/disconnection events via Redis pub/sub so all instances know which agent is connected where
• WebSocket routing: when a client request needs to reach an agent on a different instance, forward the command via Redis pub/sub
• Task scheduling: use Redis-based distributed lock (e.g., Redlock) so only one instance runs scheduled tasks at a time
• Cache invalidation: broadcast permission/config cache invalidation across instances via Redis pub/sub
• Session affinity: document that WebSocket connections require sticky sessions or that the pub/sub bridge handles cross-instance routing transparently

Acceptance Criteria

• Redis client added to backend with connection configuration
• Agent registry syncs across multiple backend instances via pub/sub
• Server control commands route to the correct backend instance that holds the agent WS connection
• Scheduled tasks execute exactly once across N instances (distributed lock)
• Permission cache invalidation propagates across instances
• Docker Compose example with 2+ backend instances
• Health check endpoint reports Redis connectivity status

[karutoil]

@pullfrog has there been any progress on this on our main branch to get this implemented.

[pullfrog]

No — Redis pub/sub horizontal scaling is not implemented on main yet.

Checked current main (f539caa) against the acceptance criteria:

Criterion	Status on main
Redis client (ioredis / similar)	Not added — not in catalyst-backend/package.json
Agent registry cross-instance sync	Not done — still process-local
Cross-instance WS command routing	Not done
Distributed lock for scheduled tasks	Not done — task-scheduler is still in-process
Cross-instance permission cache invalidation	Not done for multi-instance; only multi-worker IPC
Multi-backend Compose example	Not done — single backend service
Health check reports Redis connectivity	Not done — REDIS_URL is never read in backend source

What is on main (related, but not this issue):

• Redis container + REDIS_URL wiring in Docker Compose / .env.example (infra only; backend does not connect)
• Node cluster multi-worker mode (WORKERS) with in-process cache-bus IPC for cache invalidation across workers on the same machine
• Explicit comments in lib/cache.ts, lib/cache-bus.ts, and cluster.ts: “There is no Redis” — invalidations stay on cluster IPC
• Architecture docs still mark multi-backend-instance scaling as partial

No linked PRs or commits reference this issue. Issue remains open with the original scope unstarted.

If you want this implemented next, say the word and we can plan/build it.

Task list (3/3 completed)

• Check issue 139 comments/events for related work
• Search main branch for Redis pub/sub implementation progress
• Report findings on issue via report_progress

  ^{｜ View workflow run ｜ via Pullfrog ｜ Using openai-compatible/ck-grok-4.5 ｜ 𝕏}

[karutoil]

@pullfrog, please create a plan to get this implemented.

[pullfrog]

Plan: Horizontal scaling with Redis pub/sub (#139)

Goal

Enable multi-backend-instance deployments behind a load balancer by wiring Redis for:

1. Cross-instance agent registry + command routing
2. Cross-instance permission cache invalidation
3. Distributed lock so background jobs (task scheduler, etc.) run once
4. Health reporting Redis connectivity
5. Compose example with 2+ backends

REDIS_URL is already injected by Compose; the backend never reads it today. Single-host WORKERS cluster + IPC cache-bus remains supported and should keep working when Redis is absent.

---

Current state (baseline)

Area	Today	Gap for multi-instance
Redis	Container + REDIS_URL only	No client / no process.env.REDIS usage
Agent WS	Process-local agents Map in gateway.ts	sendToAgent / requestFromAgent fail if agent is on another process
Cache invalidation	Cluster IPC in cache-bus.ts	Does not cross hosts
Background jobs	shouldRunBackgroundJobs() → worker id 1	Only valid inside one Node primary; multi-pod double-runs
Health	/health checks Postgres only	No Redis status
Compose	Single backend service	No multi-replica example

Out of scope for the first ship (document as follow-ups unless trivial): SFTP token store, file-tunnel queues, cross-instance backup binary relay, shared permission values in Redis (invalidation-only is enough).

---

Design principles

• Opt-in: If REDIS_URL is empty, behavior stays single-instance / cluster-IPC only (no hard dependency on Redis for small installs).
• Local-first: Always try the local agent map / local cache handlers first; use Redis only on miss or for broadcast.
• Stable instance id: Each process gets INSTANCE_ID (env or generated UUID at boot); register agents under that id.
• Channels (proposed):
  • catalyst:cache-invalidate — same payload shape as today’s IPC message
  • catalyst:agent-registry — connect / disconnect / heartbeat ownership
  • catalyst:agent-command — fire-and-forget commands to a target instance
  • catalyst:agent-rpc — request/response for requestFromAgent (correlation via requestId + reply channel or instance-scoped channel)
  • catalyst:client-events (optional phase) — fan-out routeToClients / admin events to other instances’ local WS/SSE subscribers
• Keep process-local caches (SimpleCache); Redis is the invalidation + routing bus, not a shared cache store in v1.

---

Milestone 1 — Redis client + config + health

Files: new src/lib/redis.ts; package.json (ioredis); index.ts bootstrap; /health; .env.example; docs.

1. Add ioredis dependency.
2. Implement getRedis() / isRedisEnabled():
   • Connect only when REDIS_URL is set
   • Separate command client and subscriber client (ioredis requirement for pub/sub)
   • Graceful connect/retry logging; never crash boot if Redis is down when optional — decide policy: degraded mode (log + health yellow) vs fail-fast in production (prefer: start HTTP, mark health degraded when Redis configured but unreachable)
3. Extend GET /health (and optionally admin health) with:

   "redis": { "configured": true|false, "ok": true|false, "latencyMs"?: number }

   When not configured, omit or report "configured": false without failing the check. When configured and ping fails → 503 (or keep 200 for DB and expose redis separately — recommend: overall health stays green if DB ok; add GET /health/ready or include redis only when REDIS_URL set and fail if down so multi-instance deploys don’t serve without the bus).
4. Document REDIS_URL, INSTANCE_ID (optional).

Acceptance slice: Redis client present; health reflects connectivity; no behavior change when REDIS_URL empty.

---

Milestone 2 — Cache invalidation over Redis

Files: src/lib/cache-bus.ts, existing permission/agent-auth call sites (no API change), tests.

1. Extend broadcastCacheInvalidate:
   • Always apply local handlers
   • If clustered: keep existing IPC path
   • If Redis enabled: also PUBLISH catalyst:cache-invalidate with { channel, payload, originInstanceId }
2. On subscribe: ignore messages from self (originInstanceId); call applyLocal
3. Update file header comments (remove “There is no Redis”)
4. Unit tests: mock publisher/subscriber; assert dual-path; assert self-skip
5. Optional integration test against Redis in CI if test Compose already has redis

Acceptance slice: Role/API-key mutation on instance A invalidates caches on instance B.

---

Milestone 3 — Agent registry + fire-and-forget routing

Files: new src/lib/agent-bus.ts (or similar); websocket/gateway.ts (finalizeAgentConnection, disconnect, sendToAgent, heartbeat).

1. On agent auth complete: SET catalyst:agent:{nodeId} → instanceId with TTL (e.g. 90s); refresh on heartbeat; DEL on clean disconnect. Optionally pub/sub event for observability.
2. sendToAgent(nodeId, message):
   • If local open socket → send (unchanged)
   • Else if Redis: lookup owner; if present, PUBLISH command envelope { targetInstanceId, nodeId, message } on catalyst:agent-command
   • Owning instance receives, calls local send; return success/failure via optional ack channel or best-effort bool (document best-effort for fire-and-forget)
3. Heartbeat / stale keys: registry TTL + agent heartbeat keep ownership accurate after crashes
4. Reconnect: new instance overwrites registry key (last writer wins — same as current “replace socket for nodeId”)

Acceptance slice: HTTP on instance A reaches agent WS on instance B for one-way commands.

---

Milestone 4 — RPC routing (requestFromAgent / binary variants)

Files: gateway.ts pending maps; agent-bus reply path.

1. Envelope includes requestId, replyToInstanceId, kind: json|binary
2. Origin instance parks a local pending promise (same as today) keyed by requestId
3. Owner instance forwards to agent; agent reply path: if request was remote, publish response to origin instead of local pending map
4. Timeouts remain origin-side (existing timeout logic)
5. Binary streaming (streamBinaryFromAgent, relayBackupStream): v1 document limitation — require sticky affinity or same-instance agents for backup relay; full binary bridge is a later milestone (high complexity / bandwidth)

Acceptance slice: requestFromAgent works cross-instance for JSON control plane commands used by API/task executor.

---

Milestone 5 — Distributed lock for background jobs

Files: src/cluster.ts or new src/lib/job-lock.ts; index.ts job start block (~1716+).

1. Introduce acquireBackgroundJobLeadership():
   • No Redis: keep shouldRunBackgroundJobs() (worker id 1 / single process)
   • With Redis: use a simple lock key catalyst:bg-jobs:leader with instance id value + TTL (e.g. 30s), renewed by interval while leader; only leader starts scheduler/alerts/retention
2. Prefer simple SET NX EX + renew over full Redlock unless multi-Redis; single Redis is enough for Compose/single-primary Redis
3. On lock loss: stop scheduling new work (best-effort); DB claim lock in task-scheduler already reduces double execution for tasks
4. Log backgroundJobOwnerLabel() including lock state

Acceptance slice: N backends with Redis → exactly one runs cron/retention; without Redis → unchanged cluster behavior.

---

Milestone 6 — Client event fan-out (recommended for parity)

Files: gateway.ts routeToClients / admin broadcasts.

1. When publishing server-scoped events, also Redis-publish; each instance delivers to local WS clients + SSE only
2. Deduplicate with originInstanceId / event id if needed
3. Without this, UI on instance A may miss live agent events owned by B (sticky sessions help clients but not always)

Acceptance slice: Live updates visible regardless of which instance holds the browser WS (or document sticky sessions as required if this milestone is deferred).

---

Milestone 7 — Compose multi-backend + docs

Files: catalyst-docker/docker-compose.yml (or docker-compose.scale.yml example); docs/architecture.md; docs/environment-variables.md; cluster/cache comments; copilot-instructions if needed.

1. Example: two backend services (backend-1, backend-2) or deploy.replicas with shared REDIS_URL + Postgres; distinct container names; same image
2. Document session affinity:
   • Agent WebSockets: Redis bridge removes need for agent stickiness for control plane
   • Browser WebSockets: sticky sessions still recommended until client fan-out is solid; or state that pub/sub fan-out makes sticky optional
3. Update architecture “Horizontal Scaling” table from Partial → supported with Redis
4. Align docs that currently claim Redis is already used for cache/session storage

Acceptance slice: Compose example with 2+ backends; docs match code.

---

Suggested implementation order (PRs)

PR	Scope	Risk
PR1	M1 Redis client + health	Low
PR2	M2 cache-bus Redis	Low–med
PR3	M3+M4 agent registry + routing	High — core feature
PR4	M5 job lock	Med
PR5	M6 client fan-out (if not bundled in PR3)	Med
PR6	M7 Compose + docs	Low

Can also ship M1–M2+M5 first (ops readiness) then agent routing.

---

Test plan

• Unit: redis wrapper disabled path; cache-bus dual publish; agent-bus local hit / remote publish; lock acquire/renew/steal after TTL
• Integration (Compose redis): two Node processes or mocked dual subscribers; connect agent on A, command from B
• Regression: WORKERS>1 without Redis still uses IPC; single process without Redis unchanged
• Manual: scale compose to 2 backends, connect one agent, run server control + scheduled task + role change

---

Risks & decisions

Risk	Mitigation
Binary backup relay cross-instance	Explicit non-goal v1; sticky or same-instance only
Redis as SPOF	Document; health fails when configured+down; optional later Sentinel
Message size / fan-out load	Keep payloads small; don’t put binary on Redis
Split brain job lock	Short TTL + renew; Postgres task claim as second line
Double-delivery of client events	origin id + idempotent UI handling

Open product choice (default in plan): When REDIS_URL is set but Redis is unreachable, fail readiness for multi-instance safety rather than silently running split-brain jobs/caches.

---

Effort estimate

~1–2 weeks focused backend work if sequential PRs; M3–M4 is the bulk. M1+M2+M5 alone are a few days and unlock safer multi-pod for non-WS paths.

---

Out of scope (track separately)

• Redis-backed SFTP tokens / file-tunnel
• Shared session store in Redis (Better Auth already uses Postgres)
• Full binary stream bridge for backups across instances
• Replacing in-memory permission caches with Redis values

---

Ready to implement from this plan via the implement action on this issue when you want it built.

Task list (3/3 completed)

• Analyze current architecture (WS gateway, cache, scheduler, cluster)
• Draft structured implementation plan with milestones
• Post plan via report_progress

  ^{｜ Implement plan ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using openai-compatible/ck-grok-4.5 ｜ 𝕏}