Skip to content

feat: persist request history so Transactions survives a restart - #126

Merged
flyworker merged 1 commit into
mainfrom
feat/persist-request-history
Sep 4, 2026
Merged

feat: persist request history so Transactions survives a restart#126
flyworker merged 1 commit into
mainfrom
feat/persist-request-history

Conversation

@flyworker

Copy link
Copy Markdown
Member

The Transactions view read a 1000-entry in-memory ring, so every restart emptied it. You hit this directly: after a deploy the list showed 17 rows and total_requests read 3.

This node restarts often enough that the earnings panel routinely reports counter resets, and each one wiped the list — which also meant the source filter and pager from #118 could only ever page through traffic since the last restart.

What changed

Each served request is written to a request_history table beside the existing metrics_history, and /inference/requests reads from it. The in-memory ring stays as a fallback and as what the model detail view reads.

Design constraints worth reviewing

Recording must never slow down serving. The database allows a single open connection (SetMaxOpenConns(1)), so a synchronous insert per request would serialise inference behind SQLite. Writes are queued and batched by a background writer; a queue full enough to overflow drops records and counts them rather than blocking. Shutdown drains what is queued.

One row per request, so retention has two limits. RetentionDays (default 7) is how far back it stays useful; MaxRows (default 200000) stops a burst filling the disk before the age limit ever applies. Whichever binds first wins; pruning runs hourly and on start. Error reasons are truncated to 512 bytes for the same reason — an upstream body is unbounded and there is one per row.

Unlabelled rows count as Hub. Rows written before the source column carry none, and all arrived over the WebSocket. Filtering to Hub includes them, matching the in-memory path — otherwise older history vanishes the moment you touch the filter.

On SQLite and data files

SQLite is already a pure-Go dependency (glebarez/sqlitemodernc.org/sqlite), so it is compiled into the binary and works under the CGO_ENABLED=0 cross-compiles — nothing needed there.

No data file was ever tracked, but .gitignore had no *.db pattern and the docs say CP_PATH can point at the repo. Now ignores *.db, *.db-wal, *.db-shm, *.sqlite, *.sqlite3.

Verified

8 new tests against a real SQLite database, not a stand-in: paging and newest-first ordering, model and source filters, unlabelled rows counting as hub, history surviving a simulated restart (a second store over the same file sees the first one's writes), pruning by age, pruning by row cap keeping the newest, Record never blocking when the queue is full, and long error truncation.

8/8 packages pass, go vet and gofmt clean. Config documented in docs/configuration.md.

The Transactions view read a 1000-entry in-memory ring, so every restart
emptied it. This node restarts often enough that the earnings panel routinely
reports counter resets, and each one wiped the list — which also meant the
source filter and pager added in #118 could only ever page through traffic
since the last restart.

Each served request is now written to a request_history table beside the
existing metrics_history, and /inference/requests reads from it. The in-memory
ring stays as the fallback and as what the model detail view reads.

Recording must never slow down serving, so writes are queued and batched by a
background writer rather than made per request — the database allows a single
open connection, and a synchronous insert would serialise inference behind
SQLite. A queue full enough to overflow drops records and counts them instead
of blocking, and shutdown drains what is queued.

One row per request, so retention has two limits: RetentionDays for how far
back the data stays useful, and MaxRows so a burst cannot fill the disk before
the age limit applies. Error reasons are truncated for the same reason — an
upstream body is unbounded and there is one per row.

Rows written before the source column carry none, and all of them arrived over
the WebSocket, so filtering to Hub includes them. Without that an operator's
older history would vanish the moment they touched the filter, which is what
the in-memory path already does.

Also ignores *.db and its WAL sidecars: the database normally lives in
$CP_PATH, but CP_PATH can be pointed at the repo during development.
@flyworker
flyworker merged commit 58afeec into main Sep 4, 2026
1 check passed
@flyworker
flyworker deleted the feat/persist-request-history branch September 4, 2026 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant