Replace ledger state machine with OCaml implementation

.github/workflows/tb_ocaml_ci.yml

@@ -4,20 +4,20 @@ on:
   push:
     paths:
       - ".github/workflows/tb_ocaml_ci.yml"
-      - "tb_ocaml/**"
+      - "ocam/**"
       - "README.md"
   pull_request:
     paths:
       - ".github/workflows/tb_ocaml_ci.yml"
-      - "tb_ocaml/**"
+      - "ocam/**"
       - "README.md"
 
 jobs:
   checks:
     runs-on: ubuntu-latest
     defaults:
       run:
-        working-directory: tb_ocaml
+        working-directory: ocam
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -35,15 +35,18 @@ jobs:
           opam install -y ocamlformat.0.28.1
 
       - name: Build
-        run: dune build
+        run: opam exec -- dune build
 
       - name: Build docs
-        run: dune build @doc
+        run: opam exec -- dune build @doc
 
       - name: Run Dune runtest
-        run: dune runtest
+        run: opam exec -- dune runtest
+
+      - name: Build benchmark
+        run: opam exec -- dune build @bench
 
       - name: Check formatting
         run: |
-          find . -path './_build' -prune -o \( -name '*.ml' -o -name '*.mli' \) -print0 \
+          find . -name _build -prune -o \( -name '*.ml' -o -name '*.mli' \) -print0 \
             | xargs -0 opam exec -- ocamlformat --check

.github/workflows/tb_ocaml_coverage.yml

@@ -4,20 +4,20 @@ on:
   push:
     paths:
       - ".github/workflows/tb_ocaml_coverage.yml"
-      - "tb_ocaml/**"
+      - "ocam/**"
       - "README.md"
   pull_request:
     paths:
       - ".github/workflows/tb_ocaml_coverage.yml"
-      - "tb_ocaml/**"
+      - "ocam/**"
       - "README.md"
 
 jobs:
   coverage:
     runs-on: ubuntu-latest
     defaults:
       run:
-        working-directory: tb_ocaml
+        working-directory: ocam
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -38,16 +38,17 @@ jobs:
         run: |
           rm -rf _coverage
           mkdir -p _coverage
-          BISECT_SILENT=YES dune runtest --instrument-with bisect_ppx --force
+          BISECT_SILENT=YES BISECT_FILE="$PWD/_coverage/bisect" \
+            opam exec -- dune runtest --instrument-with bisect_ppx --force
 
       - name: Generate coverage summary
-        run: opam exec -- bisect-ppx-report summary
+        run: opam exec -- bisect-ppx-report summary --coverage-path _coverage
 
       - name: Generate HTML report
-        run: opam exec -- bisect-ppx-report html
+        run: opam exec -- bisect-ppx-report html --coverage-path _coverage
 
       - name: Upload HTML report
         uses: actions/upload-artifact@v4
         with:
           name: tb_ocaml-coverage-html
-          path: tb_ocaml/_coverage
+          path: ocam/_coverage

.gitignore

@@ -0,0 +1 @@
+ocam/_build/

AGENTS.md

@@ -25,6 +25,17 @@ These instructions apply to the whole repository.
 - After `jj commit`, `@` is normally the new working change and `@-` is the
   commit just created. Do not try to fix this with Git checkout commands.
 
+### Commit and push workflow
+
+When the user explicitly asks to commit and push:
+
+1. Inspect `jj status`, `jj diff`, `jj log`, and `jj bookmark list`.
+2. Each commit should represent one coherent change; use explicit paths and
+   never include generated artifacts.
+3. Move the actual bookmark to `@-`: `jj bookmark move <bookmark> --to @-`.
+4. Push it: `jj git push --remote origin --bookmark <bookmark>`.
+5. Fetch and verify the remote bookmark matches the expected commit.
+
 ## Working style
 
 - Inspect the README, manifests, code, tests, and real execution path before

BENCHMARK_COMPARISON.md

@@ -0,0 +1,41 @@
+# State-machine benchmark comparison
+
+This repository benchmarks the pinned TigerBeetle Zig state machine and the
+OCaml rewrite with the same workload: two pre-created accounts followed by
+30,000 successful posted transfers in prebuilt batches of 30. Request creation
+is outside the timed interval in both runners.
+
+Run both benchmarks from the repository root:
+
+```sh
+sh ocam/bench/run_state_machine_comparison.sh
+```
+
+The script uses the Zig binary bundled with the pinned TigerBeetle submodule
+(`path/to/tigerbeetle/zig/zig`, version 0.14.1) and the active Opam switch for
+the OCaml executable. Set `ZIG=/path/to/zig` to use a compatible local Zig
+binary. The native harness imports the pinned source directly; it does not
+modify the submodule. On this macOS host, the command pins the native target to
+macOS 15 because the bundled Zig 0.14.1 cannot link its test runner against the
+host's newer macOS SDK target.
+
+| Implementation | Timed path | Metrics |
+| --- | --- | --- |
+| TigerBeetle Zig | The pinned `StateMachine` commit path through TigerBeetle's in-memory test storage, including multi-batch encoding, prefetch, and LSM indexes. | Operations/second and mean batch latency. |
+| OCaml | `create_transfers` in the deterministic in-memory rewrite, including creation-result allocation. | Operations/second, mean batch latency, and OCaml heap words allocated. |
+
+The figures are useful for tracking each implementation on the same machine,
+but are not a server-throughput comparison. The Zig path includes its LSM/VSR
+test fixture; the OCaml path intentionally has no storage or replication
+adapter yet. Run several times on an otherwise idle machine and compare medians.
+
+## Latest local run
+
+Populate this table with the output of the command above when recording a
+machine-specific result. Do not compare results from different hosts, compiler
+versions, or optimization modes.
+
+| Implementation | Operations/s | Mean batch latency (ms) | Allocation |
+| --- | ---: | ---: | --- |
+| TigerBeetle Zig | pending first run | pending first run | not reported by this harness |
+| OCaml | pending first run | pending first run | pending first run |

README.md

@@ -1,20 +1,66 @@
 # 6666
 
-An experimental OCaml implementation of the TigerBeetle ledger database. The upstream TigerBeetle Zig repository is pinned as a Git submodule at `path/to/tigerbeetle` for source and behavior reference.
+An experimental OCaml rewrite of TigerBeetle's ledger state machine. It keeps
+the pinned upstream TigerBeetle source at `path/to/tigerbeetle` as the behavior
+reference; do not treat this repository as a replacement TigerBeetle server.
 
-See [OCAML_REWRITE_PLAN.md](OCAML_REWRITE_PLAN.md) for the component-by-component
-rewrite, compatibility, benchmarking, and analysis plan.
+## Layout
 
+- `path/to/tigerbeetle/` — pinned upstream Zig source and behavior oracle.
+- `ocam/` — a copy of that pinned revision (`97c7a8ef385270ebe0e1b75959d3d21d134629df`),
+  with `src/state_machine.zig` replaced by the OCaml state machine and its
+  interface.
+- `ocam/src/` — deterministic ledger core, built with Dune and Base.
+- `ocam/test/` and `ocam/bench/` — equivalence scenarios and a state-machine
+  benchmark. See [`BENCHMARK_COMPARISON.md`](BENCHMARK_COMPARISON.md) to run
+  the matching Zig and OCaml state-machine measurements.
 
-The experimental version is written in ocam folder, library usage guide
+The intended toolchain is the Jane Street/OxCaml ecosystem. The current core
+is synchronous and keeps its state and wire/storage representations explicit;
+Async belongs at an integration boundary rather than in the ledger logic.
 
-I plan to use the janestreet ocaml ecosystem libs, instead of normal ocaml, oxcaml, core, base, async, magic-trace etc etc
+## Build, test, and benchmark
 
-use the following
-https://github.com/janestreet/async
-https://github.com/janestreet/magic-trace
-https://github.com/janestreet/core
-https://github.com/oxcaml/oxcaml
-https://github.com/oxcaml/odoc
-https://github.com/oxcaml/ocaml-lsp
-https://github.com/oxcaml/ocamlformat
+From `ocam/`:
+
+```sh
+opam install . --deps-only --with-test
+opam exec -- dune build
+opam exec -- dune runtest
+opam exec -- dune build @bench
+```
+
+The benchmark reports operations per second, per-batch latency, and allocation
+figures.
+
+To run the paired comparison from the repository root:
+
+```sh
+sh ocam/bench/run_state_machine_comparison.sh
+```
+
+## Documentation
+
+Start with the [ledger-core guide](doc/README.md). It describes the state
+machine's public operations, timestamp and linked-batch semantics, and the
+boundary between this in-memory core and the unchanged upstream components.
+
+The API reference is generated from the OCaml interface with `odoc`:
+
+```sh
+cd ocam
+opam exec -- dune build @doc
+open _build/default/_doc/_html/index.html
+```
+
+The generated HTML is a build artifact and is intentionally not checked in.
+
+## Current boundary
+
+The copied Zig LSM and VSR sources are unchanged. The OCaml state machine does
+not yet link into the copied Zig server: that needs a C-ABI adapter and
+wire-compatible 128-byte codecs. Current tests cover core account and transfer
+operations, linked rollback, lookups, and queries; full TigerBeetle equivalence
+still requires the complete Zig corpus and several protocol and edge-case
+areas. See [`ocam/OCAML_REWRITE.md`](ocam/OCAML_REWRITE.md) for the detailed
+coverage and remaining work.

bench/test_options.zig

@@ -0,0 +1 @@
+pub const benchmark = true;

bench/vsr_options.zig

@@ -0,0 +1,4 @@
+pub const git_commit: ?[40]u8 = "bee71e0000000000000000000000000000bee71e".*;
+pub const config_verify = true;
+pub const release = "65535.0.0";
+pub const release_client_min = "0.16.4";

doc/README.md

@@ -0,0 +1,99 @@
+# OCaml ledger core
+
+This is the reader's guide for the experimental OCaml rewrite in
+[`ocam/src/state_machine.ml`](../ocam/src/state_machine.ml). It is a
+deterministic, in-memory implementation of a portion of TigerBeetle's ledger
+state-machine behavior. It is not a TigerBeetle server and it does not yet
+replace the pinned upstream Zig implementation.
+
+Read this guide first, then use [the architecture and behavior notes](architecture.md)
+while reading the code. The detailed implementation boundary and test coverage
+are tracked in [`ocam/OCAML_REWRITE.md`](../ocam/OCAML_REWRITE.md).
+
+## Generate the API reference
+
+The public interface is [`ocam/src/state_machine.mli`](../ocam/src/state_machine.mli).
+It is rendered with `odoc`, which is already declared as the package's
+documentation dependency:
+
+```sh
+cd ocam
+opam install . --deps-only --with-doc
+opam exec -- dune build @doc
+open _build/default/_doc/_html/index.html
+```
+
+`_build/default/_doc/_html/` is generated output. Do not edit or commit it;
+change the `.mli` comments or these Markdown guides instead.
+
+## The public model
+
+Call `State_machine.empty ()` to make a state. The state owns accounts,
+transfers, pending-transfer status, and the most recently committed timestamp.
+All public operations are synchronous and mutate that state in the caller's
+thread. There is deliberately no Async, storage, network, or clock dependency
+in this layer.
+
+Identifiers and amounts are `U128.t`. `U128` exposes construction, comparison,
+addition, and subtraction explicitly so balance arithmetic can report overflow
+or underflow rather than silently wrapping.
+
+An account has four monotonic balance fields:
+
+| Debit side | Credit side |
+| --- | --- |
+| `debits_pending` | `credits_pending` |
+| `debits_posted` | `credits_posted` |
+
+A normal transfer increases posted debit and credit balances by the same
+amount. A pending transfer increases the pending fields instead. Consequently,
+across a set of accounts participating in successful transfers, total pending
+debits equal total pending credits and total posted debits equal total posted
+credits.
+
+## Operations at a glance
+
+| Operation | What it does |
+| --- | --- |
+| `create_accounts` | Validates and stores account requests. |
+| `create_transfers` | Validates and applies normal, pending, post-pending, and void-pending transfers. |
+| `expire_pending_transfers` | Removes balances for timed-out pending transfers and marks them expired. |
+| `lookup_accounts`, `lookup_transfers` | Looks up supplied IDs, keeping request order and omitting unknown IDs. |
+| `query_accounts`, `query_transfers` | Filters by non-zero fields, sorts by timestamp, then applies `limit`. |
+| `get_account_transfers` | Queries transfer history for the debit and/or credit side of one account. |
+| `get_account_balances` | Queries per-transfer balance snapshots for accounts created with the history flag. |
+
+Each create operation returns one result per input request. A successful result
+contains the assigned timestamp; an unsuccessful result normally has timestamp
+zero and a status explaining the validation or state conflict. Exact retry
+requests are idempotent and return an `*_exists` status rather than changing
+balances again.
+
+## A minimal execution
+
+Construct `account` and `transfer` records using zero-valued balance and
+metadata fields, set a non-zero `id`, `ledger`, and `code`, then submit them in
+commit order:
+
+```ocaml
+let state = State_machine.empty () in
+let accounts = State_machine.create_accounts state ~timestamp:1L [ account_1; account_2 ] in
+let transfers = State_machine.create_transfers state ~timestamp:3L [ transfer ] in
+```
+
+Requests without the `imported` flag must carry `timestamp = 0L`; the state
+machine assigns timestamps from the supplied operation timestamp. The caller
+is responsible for supplying operations in commit order. See
+[the behavior notes](architecture.md) for batch, pending, and query rules.
+
+## What is and is not covered
+
+The Dune tests exercise account creation, normal and pending transfers,
+posting, voiding, expiry, linked rollback, lookups, queries, U128 boundaries,
+and several deterministic properties. They do not establish full compatibility
+with TigerBeetle. In particular, the OCaml core has no C ABI or 128-byte wire
+codec, does not link into the copied Zig server, and has remaining parity work
+listed in `ocam/OCAML_REWRITE.md`.
+
+The pinned tree at [`path/to/tigerbeetle`](../path/to/tigerbeetle) remains the
+upstream behavior reference. Do not change it while working on this rewrite.