feat: persist request history so Transactions survives a restart - #126
Merged
Conversation
The Transactions view read a 1000-entry in-memory ring, so every restart emptied it. This node restarts often enough that the earnings panel routinely reports counter resets, and each one wiped the list — which also meant the source filter and pager added in #118 could only ever page through traffic since the last restart. Each served request is now written to a request_history table beside the existing metrics_history, and /inference/requests reads from it. The in-memory ring stays as the fallback and as what the model detail view reads. Recording must never slow down serving, so writes are queued and batched by a background writer rather than made per request — the database allows a single open connection, and a synchronous insert would serialise inference behind SQLite. A queue full enough to overflow drops records and counts them instead of blocking, and shutdown drains what is queued. One row per request, so retention has two limits: RetentionDays for how far back the data stays useful, and MaxRows so a burst cannot fill the disk before the age limit applies. Error reasons are truncated for the same reason — an upstream body is unbounded and there is one per row. Rows written before the source column carry none, and all of them arrived over the WebSocket, so filtering to Hub includes them. Without that an operator's older history would vanish the moment they touched the filter, which is what the in-memory path already does. Also ignores *.db and its WAL sidecars: the database normally lives in $CP_PATH, but CP_PATH can be pointed at the repo during development.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Transactions view read a 1000-entry in-memory ring, so every restart emptied it. You hit this directly: after a deploy the list showed 17 rows and
total_requestsread 3.This node restarts often enough that the earnings panel routinely reports counter resets, and each one wiped the list — which also meant the source filter and pager from #118 could only ever page through traffic since the last restart.
What changed
Each served request is written to a
request_historytable beside the existingmetrics_history, and/inference/requestsreads from it. The in-memory ring stays as a fallback and as what the model detail view reads.Design constraints worth reviewing
Recording must never slow down serving. The database allows a single open connection (
SetMaxOpenConns(1)), so a synchronous insert per request would serialise inference behind SQLite. Writes are queued and batched by a background writer; a queue full enough to overflow drops records and counts them rather than blocking. Shutdown drains what is queued.One row per request, so retention has two limits.
RetentionDays(default 7) is how far back it stays useful;MaxRows(default 200000) stops a burst filling the disk before the age limit ever applies. Whichever binds first wins; pruning runs hourly and on start. Error reasons are truncated to 512 bytes for the same reason — an upstream body is unbounded and there is one per row.Unlabelled rows count as Hub. Rows written before the source column carry none, and all arrived over the WebSocket. Filtering to Hub includes them, matching the in-memory path — otherwise older history vanishes the moment you touch the filter.
On SQLite and data files
SQLite is already a pure-Go dependency (
glebarez/sqlite→modernc.org/sqlite), so it is compiled into the binary and works under theCGO_ENABLED=0cross-compiles — nothing needed there.No data file was ever tracked, but
.gitignorehad no*.dbpattern and the docs sayCP_PATHcan point at the repo. Now ignores*.db,*.db-wal,*.db-shm,*.sqlite,*.sqlite3.Verified
8 new tests against a real SQLite database, not a stand-in: paging and newest-first ordering, model and source filters, unlabelled rows counting as hub, history surviving a simulated restart (a second store over the same file sees the first one's writes), pruning by age, pruning by row cap keeping the newest,
Recordnever blocking when the queue is full, and long error truncation.8/8 packages pass,
go vetandgofmtclean. Config documented indocs/configuration.md.