You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Goal: an OpenAI-compatible endpoint so a coding harness can drive jlib's inference. This is the design sketch, not a plan of work.
What already exists
sys::server — accepts, optionally TLS, hands each connection to a handler as a live socketstream&, dispatching through a job_queue. threads = 0 serves inline; threads = N uses a pool. Admission control bounds descriptors.
net::http::server — parses a request against the RFC 9112 grammar, calls a handler, writes a response.
generate() — takes an on_token callback fired per token, returning false to stop.
util::json — a facade over json-c that survives a stranger's input.
Both ends of a streaming path exist. The middle does not.
The blocker is streaming, not HTTP/1
net::http::server's handler is void(const Request&, response&), and response holds a std::string m_body serialised and written in one go. Its header says so:
No streaming: a response is accumulated whole and written in one go, which is what lets a handler that throws still be answered with a 500, and which means a body has to fit in memory.
At ~92 tok/s a 2000-token reply is 20+ seconds of silence and then a dump. Every coding harness expects incremental output.
Not blockers, for the record:
Pipelining. Unsupported, and irrelevant — real clients open parallel connections. Nothing expects it.
HTTP/2. Unnecessary; OpenAI-compatible APIs run over HTTP/1.1 with SSE.
Connection: close per response. A handshake per request. Wasteful, survivable, and deliberate — keep-alive framing is where request smuggling lives, and the message layer refused it on purpose.
The correction that shapes everything else
The obvious design is a second job_queue of model workers fed by the connection queue. That cannot work as stated, because a model<T> is not a stateless compute engine:
The key-value cache is the conversation. Two requests through one model instance would interleave their keys and produce nonsense — silently, in the way this codebase keeps finding.
So a model instance serves one conversation at a time, and the second tier is a serialisation point, not a pool. Concretely: job_queue(1) per loaded model, or a mutex, and those are the same thing.
That also means the chunk hand-back may not be needed. If the connection handler acquires the model and runs generate() itself with an on_token that writes SSE straight to its own socketstream&, there is no queue hop and no buffer between tiers. The handler simply is the model worker while it holds the model.
A hand-back is needed only if the two are genuinely decoupled — see batching below.
Lifecycle
What the user asked for, and it is right: a model is loaded once and shared. Gemma 2 at q8_0 is 2.8 GB and 7.7 s to load; per-request loading is not on the table.
engine
name -> loaded model (loaded on first use, never twice)
per model: a serialisation point (job_queue(1), or a mutex)
per model: a reset between requests (reset_cache, and m_seq)
Concurrency across models is free — two different models are two instances. Concurrency within a model is not; see below.
Reset between requests.reset_cache() exists. Whether the engine resets or the caller does, and what happens if a request dies mid-generation leaving a dirty cache, is a correctness question with a silent failure mode.
Concurrency within one model: two options, both real work
Share the weights, duplicate the state. Weights are read-only after load(); the cache and scratch are not. Splitting model<T> into a shared weight object and a per-conversation state object would let N conversations run against one 2.8 GB copy. Clean, and a real refactor — every tensor in model and block has to be sorted into one bucket or the other.
Batch them. The bigger win, and the reason production servers have a model queue at all. Decode is bandwidth-bound on weights: one sweep serves one token today, and could serve N sequences' tokens for nearly the same cost. jlib's forward() already takes many columns — but as one sequence's positions, not independent sequences with separate caches. That is substantial work and it is the thing the queue design should not preclude.
If batching is ever wanted, the second tier must be a queue with a hand-back, because the model worker then serves several connections at once and cannot write to any one socket directly. sys::ringbuffer — lock-free, one producer and one consumer, written for the audio callback — is the right shape for that per-request hand-back.
Recommendation: build the serialisation-point version first, and keep the token path behind an interface narrow enough that a ring buffer can be slid in later.
The question the current design was built to avoid
Streaming means bytes are on the wire before the handler finishes. A handler that throws at token 500 cannot be answered with a 500 status — the 200 went out long ago.
That is exactly what net::http::server bought by refusing streaming, and it has to be given up knowingly. The options are to close the connection abruptly and let the client see a truncated stream, or to emit an SSE error event and close cleanly. The second is friendlier and still leaves the client having received a partial completion, which the OpenAI protocol does not really model.
Whatever is chosen, http_server.hh's claim about accumulate-and-write stops being true and the header has to say what replaced it.
Other things a coding harness will want
Cancellation. Harnesses abort. on_token returning false already stops generation; the connection needs to notice a client disconnect and signal it. Without this an abandoned request holds a model for its full length.
Backpressure. A deep model queue means a client waits with no output. A bounded queue and a 503 beats a timeout.
sys::server and net::http::server both say in their headers that they are not hardened for a public port. A local coding harness on loopback is within that. If this ever listens on anything else, that claim has to be revisited rather than quietly outgrown — which is how a narrow thing becomes a broad one.
Goal: an OpenAI-compatible endpoint so a coding harness can drive jlib's inference. This is the design sketch, not a plan of work.
What already exists
sys::server— accepts, optionally TLS, hands each connection to a handler as a livesocketstream&, dispatching through ajob_queue.threads = 0serves inline;threads = Nuses a pool. Admission control bounds descriptors.net::http::server— parses a request against the RFC 9112 grammar, calls a handler, writes a response.generate()— takes anon_tokencallback fired per token, returning false to stop.util::json— a facade over json-c that survives a stranger's input.Both ends of a streaming path exist. The middle does not.
The blocker is streaming, not HTTP/1
net::http::server's handler isvoid(const Request&, response&), andresponseholds astd::string m_bodyserialised and written in one go. Its header says so:At ~92 tok/s a 2000-token reply is 20+ seconds of silence and then a dump. Every coding harness expects incremental output.
Not blockers, for the record:
Connection: closeper response. A handshake per request. Wasteful, survivable, and deliberate — keep-alive framing is where request smuggling lives, and the message layer refused it on purpose.The correction that shapes everything else
The obvious design is a second
job_queueof model workers fed by the connection queue. That cannot work as stated, because amodel<T>is not a stateless compute engine:The key-value cache is the conversation. Two requests through one model instance would interleave their keys and produce nonsense — silently, in the way this codebase keeps finding.
So a model instance serves one conversation at a time, and the second tier is a serialisation point, not a pool. Concretely:
job_queue(1)per loaded model, or a mutex, and those are the same thing.That also means the chunk hand-back may not be needed. If the connection handler acquires the model and runs
generate()itself with anon_tokenthat writes SSE straight to its ownsocketstream&, there is no queue hop and no buffer between tiers. The handler simply is the model worker while it holds the model.A hand-back is needed only if the two are genuinely decoupled — see batching below.
Lifecycle
What the user asked for, and it is right: a model is loaded once and shared. Gemma 2 at q8_0 is 2.8 GB and 7.7 s to load; per-request loading is not on the table.
Open questions, none of them decided:
reset_cache()exists. Whether the engine resets or the caller does, and what happens if a request dies mid-generation leaving a dirty cache, is a correctness question with a silent failure mode.Concurrency within one model: two options, both real work
Share the weights, duplicate the state. Weights are read-only after
load(); the cache and scratch are not. Splittingmodel<T>into a shared weight object and a per-conversation state object would let N conversations run against one 2.8 GB copy. Clean, and a real refactor — every tensor inmodelandblockhas to be sorted into one bucket or the other.Batch them. The bigger win, and the reason production servers have a model queue at all. Decode is bandwidth-bound on weights: one sweep serves one token today, and could serve N sequences' tokens for nearly the same cost. jlib's
forward()already takes many columns — but as one sequence's positions, not independent sequences with separate caches. That is substantial work and it is the thing the queue design should not preclude.If batching is ever wanted, the second tier must be a queue with a hand-back, because the model worker then serves several connections at once and cannot write to any one socket directly.
sys::ringbuffer— lock-free, one producer and one consumer, written for the audio callback — is the right shape for that per-request hand-back.Recommendation: build the serialisation-point version first, and keep the token path behind an interface narrow enough that a ring buffer can be slid in later.
The question the current design was built to avoid
Streaming means bytes are on the wire before the handler finishes. A handler that throws at token 500 cannot be answered with a 500 status — the 200 went out long ago.
That is exactly what
net::http::serverbought by refusing streaming, and it has to be given up knowingly. The options are to close the connection abruptly and let the client see a truncated stream, or to emit an SSE error event and close cleanly. The second is friendlier and still leaves the client having received a partial completion, which the OpenAI protocol does not really model.Whatever is chosen,
http_server.hh's claim about accumulate-and-write stops being true and the header has to say what replaced it.Other things a coding harness will want
on_tokenreturning false already stops generation; the connection needs to notice a client disconnect and signal it. Without this an abandoned request holds a model for its full length.--tokensreservation arithmetic is the same problem: reject or trim when prompt plus reply cannot fit./v1/models. Trivial, and harnesses call it.Scope note
sys::serverandnet::http::serverboth say in their headers that they are not hardened for a public port. A local coding harness on loopback is within that. If this ever listens on anything else, that claim has to be revisited rather than quietly outgrown — which is how a narrow thing becomes a broad one.