Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions doc/rfc/submitqueue/status-list-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ The append-only request log is not shaped for the second or third query. Serving
8. The data model uses separate query-shaped stores rather than secondary indexes.
9. Existing rows are not backfilled. Requests received before rollout may be unavailable through the new read model.
10. Retention and pruning are outside the scope of this RFC.
11. Land receipts begin in the internal `accepting` state. `accepting` requests are excluded from `Status` and `List` until successful pipeline publication promotes them to `accepted`, or a later pipeline event proves that publication succeeded.

## Vocabulary

Expand Down Expand Up @@ -75,7 +76,7 @@ The gateway owns the append-only request log and three new logical read models.

The authoritative request summary is keyed by sqid. It contains immutable request context plus the current materialized request-log winner and the reconciliation state needed to compare a later log entry without rereading historical logs.

The immutable context is queue, change URIs, and receipt time. The mutable response state is status, last error, and metadata. Optimistic-lock state is internal to the projection and is not part of the API response.
The immutable context is queue, change URIs, and receipt time. The mutable response state is status, last error, and metadata. The internal `accepting` admission state is not returned by the API. Optimistic-lock state is internal to the projection and is not part of the API response.

The sqid key supports authoritative lookup and conditional status updates for one request without a secondary index.

Expand All @@ -101,9 +102,13 @@ The URI is stored in the canonical form received from the validated Land request

### Land Receipt

After synchronous validation, Land generates the sqid and one receipt timestamp. The gateway persists the authoritative summary, URI mappings, and queue projection before appending the initial `accepted` request log and before publishing the request to the orchestrator.
After synchronous validation, Land generates the sqid and one receipt timestamp. The gateway persists the authoritative summary in the internal `accepting` state, then publishes the request to the orchestrator. It does not create the URI or queue projections while the request remains `accepting`.

This ordering guarantees that immutable request context exists before the request can produce later status logs or enter the asynchronous pipeline. The logical writes are independent and must be safe to retry for the same sqid and values; conflicting duplicate data is an error.
After publication succeeds, Land appends the initial `accepted` request log. Materializing `accepted`, `started`, or any later event promotes the authoritative summary out of `accepting` and creates the URI and queue projections. This handles the race where the orchestrator emits `started` before Land finishes persisting `accepted`.

An `accepted` event is the lowest public lifecycle state. A late `accepted` event is retained in request history but must not replace `started` or any later materialized status.

Pipeline publication is the Land success boundary. If publication fails, Land returns an error and leaves the hidden `accepting` receipt for operational cleanup. If publication succeeds but appending or materializing `accepted` fails, Land still returns the sqid because retrying the RPC would submit a duplicate request that is already in the pipeline. A later pipeline event activates and repairs the public projections.

### Request-Log Materialization

Expand All @@ -115,30 +120,32 @@ The winner comparison preserves the existing `Status` behavior:
2. Between versioned terminal entries, the greater request version wins.
3. Equal terminal versions use the greater log timestamp as a tie-breaker.
4. When no versioned terminal winner exists, the greater log timestamp wins.
5. Any retained lifecycle event promotes an `accepting` summary into the public projections.
6. `accepted` cannot replace `started` or any later status, even when the accepted log arrives later.

Materialization uses optimistic concurrency so stale or out-of-order consumers cannot replace a newer winner. Version arithmetic and reconciliation decisions belong to the materialization component; stores perform only mechanical creates, reads, conditional updates, and bounded page queries.

## Read Flows

### Status by Sqid

The gateway reads the authoritative summary by sqid and returns its current materialized state and immutable context. It does not fall back to parsing the sqid, reconciling logs, or reading orchestrator stores.
The gateway reads the authoritative summary by sqid and returns its current materialized state and immutable context. An `accepting` summary is treated as not found because it has not crossed the public admission boundary. The gateway does not fall back to parsing the sqid, reconciling logs, or reading orchestrator stores.

### Status by Change URI

The gateway performs a bounded newest-first scan of the URI reverse mapping and then reads the authoritative summary for each resolved sqid. Results preserve the mapping order. No mapping is a not-found result; exceeding the 100-request maximum is an error. A mapping whose authoritative summary is missing is an internal consistency error that fails the lookup; it is not returned as user-facing not-found and is not silently omitted.
The gateway performs a bounded newest-first scan of the URI reverse mapping and then reads the authoritative summary for each resolved sqid. Results preserve the mapping order. URI mappings are created only when the request reaches `accepted` or a later state, so `accepting` receipts are absent by construction. No mapping is a not-found result; exceeding the 100-request maximum is an error. A mapping whose authoritative summary is missing is an internal consistency error that fails the lookup; it is not returned as user-facing not-found and is not silently omitted.

### List by Queue and Receipt Time

The gateway performs one bounded range scan of the queue projection using queue, receipt-time bounds, and an optional keyset cursor. The ordering key is immutable, so later status updates cannot move an item across an issued cursor.
The gateway performs one bounded range scan of the queue projection using queue, receipt-time bounds, and an optional keyset cursor. Queue projections are created only when the request reaches `accepted` or a later state, so `accepting` receipts are absent by construction. The ordering key is immutable, so later status updates cannot move an item across an issued cursor.

## Consistency

The request log remains the append-only audit record. The authoritative summary and queue projection are eventually consistent views of its winning current state.

Because the authoritative summary and queue projection are separate writes, a short interval can exist where `Status` and `List` show different statuses. Retried materialization repairs the queue projection from the authoritative sqid summary until both converge. Neither API reconciles logs during reads to hide this interval.

Request context has a stronger guarantee than status convergence: it is persisted before the initial log and before the request is published to the orchestrator. Once a queue projection is visible, its queue, sqid, change URIs, and receipt time are complete.
Request context has a stronger guarantee than status convergence: the authoritative receipt is persisted before the request is published to the orchestrator. Public URI and queue projections are activated only by `accepted` or a later event. Once a queue projection is visible, its queue, sqid, change URIs, and receipt time are complete.

## Compatibility and Rollout

Expand Down
3 changes: 3 additions & 0 deletions submitqueue/extension/storage/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,10 @@ go_library(
"build_store.go",
"change_store.go",
"request_log_store.go",
"request_queue_summary_store.go",
"request_store.go",
"request_summary_store.go",
"request_uri_store.go",
"speculation_path_build_store.go",
"speculation_tree_store.go",
"storage.go",
Expand Down
3 changes: 3 additions & 0 deletions submitqueue/extension/storage/mock/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,10 @@ go_library(
"build_store_mock.go",
"change_store_mock.go",
"request_log_store_mock.go",
"request_queue_summary_store_mock.go",
"request_store_mock.go",
"request_summary_store_mock.go",
"request_uri_store_mock.go",
"speculation_path_build_store_mock.go",
"speculation_tree_store_mock.go",
"storage_mock.go",
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

71 changes: 71 additions & 0 deletions submitqueue/extension/storage/mock/request_uri_store_mock.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading