Skip to content

fix: revert broad metadata batching#2

Open
ctxswitch wants to merge 2 commits into
mainfrom
fix/revert-broad-metadata-batching
Open

fix: revert broad metadata batching#2
ctxswitch wants to merge 2 commits into
mainfrom
fix/revert-broad-metadata-batching

Conversation

@ctxswitch

@ctxswitch ctxswitch commented Jul 16, 2026

Copy link
Copy Markdown

Summary

Narrow the bbolt batching rollback to the operations that caused the production lock contention, while retaining request-path batching controls and telemetry.

The failing production image changed serialized S3FIFO maintenance from immediate transactions to delayed bbolt.Batch calls. Eviction holds queueMu while scanning pinned entries, so every candidate paid the batch delay twice while the global admission lock remained held. Expiry was also changed from one write transaction per reaper batch to one delayed batch call per expired entry.

What changed

  • Restore immediate bbolt.Update transactions for serialized S3FIFO eviction and ghost maintenance.
  • Restore one write transaction per envelope-expiry batch.
  • Keep batching only on the paths that already used it in the stable image: blob puts, blob touches, envelope puts, S3FIFO admissions, and ghost hits.
  • Retain METADATA_BATCH_SIZE and METADATA_BATCH_DELAY configuration.
  • Set defaults to the stable-image behavior: batch size 1000 and delay 1ms.
  • Retain metadata batching and bbolt write telemetry.
  • Add regressions proving eviction maintenance does not wait for the request batch window and expiry batches commit in one write transaction.

Compatibility

No data-format or migration impact. The configuration knobs remain available. Their defaults preserve the effective behavior of the previous production image: the service explicitly used a 1 ms delay and inherited bbolt v1.5.0 batch size 1000.

Configuration

Option Environment variable Default
Metadata batch size METADATA_BATCH_SIZE 1000
Metadata batch delay METADATA_BATCH_DELAY 1ms

These settings affect only operations that use request-path batching. Serialized eviction maintenance and bulk expiry cleanup do not wait on this batch window.

Validation

  • Regression tests: TestPopTailDoesNotWaitForRequestBatchWindow and TestEnvelopeReaperDeletesEachBatchInSingleWriteTransaction
  • Full suite: GOCACHE=/tmp/codex-go-build go test ./...
  • gofmt and git diff --check

Restore the pre-8d2998b transaction model: keep batching on concurrent request paths, use immediate transactions for serialized S3FIFO maintenance, and delete each expiry batch in one transaction. This prevents the bbolt batch window from extending the global eviction lock across a full pinned scan.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant