fix(noderesource): raise cosmos-exporter memory ceiling 384Mi -> 512Mi (stopgap)

[bdchatham]

Summary

cosmos-exporter sidecars are OOMKilling roughly every ~2h fleet-wide (observed on atlantic-2/archive-0-0-0: 7 restarts over 14h). This raises the hardcoded memory limit in defaultCosmosExporterResources() from 384Mi to 512Mi as a stopgap to reduce restart frequency — it does not fix the underlying leak.

Root cause

Investigated as part of a broader NodeSystemSaturation remediation (unrelated node/incident, cosmos-exporter surfaced as a side finding). Root cause is believed to be the goroutine-per-event fan-out in HandleBankTransferEvent (sei-cosmos-exporter/events.go) — every qualifying transfer unconditionally spawns a goroutine with a 5-minute time.Sleep before deleting its metric label, so under a rising event-rate burst, live goroutines scale with (event rate × 300s), outrunning the drain and climbing toward OOM. An /xreview systems-engineering pass specifically ruled out the originally-suspected culprit — the eventStream.Reset()/Run() reconnect path — after reading the actual code: it's synchronous, single-goroutine, with a clean handoff on MissedItemsError; no leak there.

Follow-up needed: file the leak issue against sei-cosmos-exporter (not this repo) pointing at the goroutine-per-event fan-out, not the reconnect path.

Why this reaches every live node, not just image-drift-triggered ones

An /xreview controller-boundary pass corrected an earlier assumption in this investigation: the StatefulSet template is re-rendered and re-applied via SSA on every controller reconcile pass, completely independent of the image-drift gate — that gate only controls whether the running pod gets force-replaced. So:

1. This change reaches every live SeiNode's StatefulSet template on its next reconcile (all ~31 nodes fleet-wide, template-only — no synchronous pod restart).
2. A pod that isn't automatically rolled (no image drift) picks up the new limit on its next natural recreation — either its own OOM-restart cycle, or an operator-initiated kubectl delete pod (safe under OnDelete: recreated pods reflect the current template, not a frozen one).
3. This holds even for a SeiNode CR whose Git source has been removed from a Kustomization (as with atlantic-2/archive-0-0-0 currently) — the controller reconciles off the live CR object, not off Git, so Git-detachment doesn't block this specific fix.

Blast radius

Applies fleet-wide (cosmos-exporter is added unconditionally to every SeiNode/mode). Only the limits.memory moves; requests are unchanged (64Mi), so there's no scheduling/bin-packing impact. Runtime risk: the leak means every exporter trends toward its limit, so the request:limit ratio widens from 6x to 8x — worth a capacity sign-off on nodes co-hosting multiple memory-heavy exporters, flagged here rather than blocking this PR on it.

Test plan

• go build ./internal/noderesource/... passes
• go test ./internal/noderesource/... passes (updated TestCosmosExporter_DefaultResources to expect 512Mi)
• After merge + deploy: confirm OOM cadence on a sample of live cosmos-exporter sidecars trends from ~2h toward longer (expect a modest improvement, not a fix — leak growth is bursty/super-linear per observed data, so don't expect proportional improvement)
• File the upstream sei-cosmos-exporter leak issue (goroutine-per-event fan-out in HandleBankTransferEvent)

🤖 Generated with Claude Code

[cursor]

PR Summary

Low Risk
Sidecar limit-only change with unchanged scheduling requests; modest extra memory headroom per node until the exporter leak is fixed upstream.

Overview
Raises the cosmos-exporter sidecar memory limit from 384Mi to 512Mi in defaultCosmosExporterResources() as a fleet-wide stopgap while sei-cosmos-exporter is believed to leak memory (goroutine fan-out in bank transfer handling). CPU and memory requests stay 64Mi/50m; only the OOM ceiling moves.

Comments in code document that this does not fix the leak—it should reduce restart frequency until an upstream fix. TestCosmosExporter_DefaultResources is updated to assert 512Mi.

^{Reviewed by Cursor Bugbot for commit 329dd20. Bugbot is set up for automated code reviews on this repo. Configure here.}