feat: bound concurrent postings with an optional semaphore

[nicolasburtey]

Why

Under sustained posting load (lana stress test, 2 loans/s target), the balance-poster lock prelude in BalanceRepo::find_for_update was the #1 DB-time consumer by far: ~95% of all DB time, windowed mean 683ms at hour 1 rising to 976ms at hour 4, with 100% buffer hits — i.e. pure advisory-lock wait. Every hot shared account (omnibus, top-of-chart sets) serializes postings on pg_advisory_xact_lock(journal|account|currency), and when arrival rate exceeds the serialized service rate, the queue forms inside Postgres, where each waiter pins:

• a pool connection,
• an open transaction / MVCC snapshot (impedes vacuum under sustained load),
• an app task that is invisible to any app-level backpressure.

That pileup is what turned a throughput ceiling into a stability incident (liveness-probe kill ~20 min into the soak).

What

New CalaLedgerConfig::max_concurrent_postings: Option<usize> (default None = unchanged behavior). When set, CalaLedger holds a tokio::sync::Semaphore and every posting acquires a permit first, moving the queue in-process: waiters hold no connection, no snapshot, and their wait shows up in traces under the cala_ledger.posting_permit.acquire span.

Gating covers both entry points exactly once:

• post_transaction acquires before creating its DbOp (waiters don't even hold a connection), then delegates to an ungated inner fn.
• post_transaction_in_op acquires per call; a caller composing several posts into one op acquires/releases one permit at a time, so no self-deadlock (covered by the new test with max_concurrent_postings(1)).

Explicitly not claimed: this does not raise the serialization ceiling of a hot account (that's the eventually_consistent escape hatch + shorter hold times). It keeps the system stable at the ceiling instead of collapsing below it, and makes saturation observable.

Test

• New transaction_post_with_bounded_concurrency: limit 1, single post, two sequential posts in one composed op, then 4 concurrent posters; asserts final balance. Would deadlock if any path double-acquired.
• Full cala-ledger suite: 95/95 pass.

Follow-ups (not in this PR)

• Wire a LANA__LEDGER__MAX_CONCURRENT_POSTINGS (or similar) through lana-app once a cala release includes this.
• Validate on a stress-testing sandbox: expect windowed mean of the find_for_update lock query to drop from ~1s to ~raw hold time, no liveness events, achieved loans/s flat across the 4h soak.

---

Note

Medium Risk
Touches the core posting path and can throttle throughput when the limit is set too low, but default is off and behavior is opt-in with explicit backpressure intent.

Overview
Adds CalaLedgerConfig::max_concurrent_postings (Option<usize>, default None for unchanged behavior). When set, CalaLedger keeps a tokio::sync::Semaphore and acquires a permit before posting work touches the DB.

post_transaction waits for a permit before opening a DbOp, then runs the existing logic via a new post_transaction_in_op_ungated helper so the outer path does not double-acquire. post_transaction_in_op acquires one permit per call (release between sequential posts in a composed op) to avoid self-deadlock.

Excess posters queue in-process instead of stacking on Postgres advisory-lock wait with pinned connections and transactions; permit wait is visible under cala_ledger.posting_permit.acquire.

Adds transaction_post_with_bounded_concurrency (max_concurrent_postings(1): standalone post, two posts in one op, four concurrent posters, balance assertion).

^{Reviewed by Cursor Bugbot for commit e8a3b37. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e8a3b37. Configure here.}

[cursor]

Composed posts can deadlock

High Severity

The posting_permits semaphore is acquired after a database connection is taken and released before the transaction commits. This timing mismatch means the semaphore doesn't prevent connections from being held by waiting tasks and can lead to deadlocks where tasks holding permits block on advisory locks held by other tasks waiting for permits.

 

^{Reviewed by Cursor Bugbot for commit e8a3b37. Configure here.}

[cursor]

Zero bound hangs all postings

Medium Severity

Configuring max_concurrent_postings to 0 creates a semaphore with zero permits without validation. This causes acquire_posting_permit to block indefinitely, silently freezing all transaction posting instead of failing fast. This can happen if 0 is mistaken for unbounded concurrency.

 

^{Reviewed by Cursor Bugbot for commit e8a3b37. Configure here.}

[cursor]

Pool and permit can deadlock

High Severity

The new posting_permits can lead to a circular-wait deadlock. post_transaction acquires a permit before a database connection, while post_transaction_in_op (when used with begin_operation) acquires a connection first. This inverted resource acquisition order, under a bounded semaphore and saturated connection pool, causes posting operations to stall or time out.

 

^{Reviewed by Cursor Bugbot for commit e8a3b37. Configure here.}

[github-actions]

📊 Performance Report

Commit: e8a3b37
Updated: 2026-07-24 20:28:39 UTC

Cala Performance Benchmark Results (non-representative)

Criterion Benchmark Results (single-threaded)

Benchmark	Time per Run	Throughput	% vs Baseline
post_simple_transaction	6.708ms	149 tx/s	0 (baseline)
post_and_recalculate_ec_account_set	11.067ms	90 tx/s	-64.0%
post_and_batch_recalculate_ec_account_set	9.080ms	110 tx/s	-35.0%
post_multi_layer_transaction	36.671ms	27 tx/s	-446.0%
post_simple_transaction_with_effective_balances	25.900ms	38 tx/s	-286.0%
post_simple_transaction_with_skipped_velocity	8.207ms	121 tx/s	-22.0%
post_simple_transaction_with_velocity	5.164ms	193 tx/s	+23.0%
post_simple_transaction_with_hit_velocity	2.134ms	468 tx/s	+68.0%
post_simple_transaction_with_one_account_set	21.454ms	46 tx/s	-219.0%
post_simple_transaction_with_five_account_sets	45.738ms	21 tx/s	-581.0%
post_simple_transaction_with_ec_account_set	8.507ms	117 tx/s	-26.0%

Load Testing Results (parallel-execution)

Scenario	tx/s
1 parallel	167.98
2 parallel	215.88
5 parallel	246.48
10 parallel	270.81
20 parallel	292.85
2 contention	159.60
5 contention	188.83
2 acct_sets	135.26
5 acct_sets	166.58

---

Note: Performance results may vary based on system resources and database state.

Last updated by commit e8a3b37