THOTH-GQL-OPS-01: GraphQL mutation-guard runtime operations control record and prerequisite specifications

CHANGELOG.md

@@ -6,6 +6,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 ### Added
+  - [793](https://github.com/thoth-pub/thoth/pull/793) - `THOTH-GQL-OPS-01`: implement the approved GraphQL mutation-guard **runtime-operations** control work. Adds the evidenced operational-control record and the **PROVISIONAL** mode-transition runbook required by `ADR-0006` section 8.3.5, and the three prerequisite task specifications `THOTH-GQL-OPS-02`, `THOTH-GQL-OPS-03` and `THOTH-GQL-OPS-04` — each `DRAFT` with implementation `NOT AUTHORIZED` and no branch created. Every operational statement carries a named evidence source and an evidence class (`[REPO]`, `[EXTERNAL]`, `[REPO + EXTERNAL]` or `[UNVERIFIED]`), and no deployed-state conclusion is attributed to repository evidence alone. Re-derived independently at the exact base and **confirmed, not refuted**: the mode is read **once at process start** with no reload path, signal handler, watcher or admin route anywhere in the workspace, so changing the effective mode of a running process is impossible; **no** surface exposes the effective mode of a serving instance, leaving `OFF` and `OBSERVE` externally indistinguishable; and the image's default `init` command does not register the guard argument, so — traced through pinned `clap_builder` 4.6.0, where `verify_arg` returns `UnknownArgument` only under `cfg(debug_assertions)`, and reproduced in an isolated throwaway probe outside this repository — a **guard-enabled** release build started through `init` silently ignores `THOTH_GRAPHQL_MUTATION_GUARD_MODE` and is unconditionally `OFF`, while the same path panics in a debug build. Newly established and recorded: on the `init` path an **invalid** value is also silently ignored rather than rejected, because the `value_parser` never runs; the container environment is the **sole** configuration source, with pinned `dotenv` unable to override it and no `.env` present in the `FROM scratch` image; the production GraphQL API service supplies **no** container command override and so inherits `init`, while a different Thoth service in the same definition does override its command, making the inheritance a deliberate configuration state; the serving fleet is **autoscaled**, so the expected population is a range with a live current value and never a copied number; rolling replacement makes a mixed-mode window **structurally guaranteed** rather than exceptional; a mode change is **both** a configuration change **and** a deployment, positively contradicting the withdrawn "configuration change without a deploy" claim; rollback uses the forward change's configuration/deployment mechanism and is technically executed by the same execution-capability team, while its actual latency and whether it additionally requires CTO approval both remain `[UNVERIFIED]` — no authorization equivalence is inferred from sharing the technical mechanism — and it is therefore **not** a kill switch; and runtime log retention is configured to a **finite** duration, while the approved `OBSERVE` observation-window duration, whether that retention covers it, and any remedy are all `[UNVERIFIED]` and **downstream** — a remedy cannot be selected before the duration it must cover is approved, so `THOTH-GQL-OPS-04` records the requirement and the unresolved dependency rather than choosing one. The **test** environment was established to be **pre-guard** as well as production — both run a release whose binary contains no mutation guard, so neither has a guard mode and neither may be described as `MutationGuardMode::OFF`; there is consequently no environment in which a mode could currently be changed, and the future rehearsal must first deploy a guard-enabled candidate to a non-production environment. Ownership is recorded as roles, not individuals, and **execution capability is deliberately not relabelled as accountable ownership**: the technical team holding maintain permission on the authoritative deployment source is established as technically able to execute a change, approval authority for merge and for **each** activation is the CTO, but the **accountable production runtime owner** and the **post-activation observation sign-off owner** are both recorded `[UNVERIFIED]` — the first needs a CTO designation no amount of further reading can supply, the second is a proposal awaiting the CTO's own confirmation — and whether operational rollback additionally requires CTO approval is likewise unresolved. Four acceptance criteria are therefore recorded **FAIL/BLOCKED** rather than converted into passes, which is the expected shape of a task terminating at disposition `C`. The fleet-verification mechanism is **specified and deliberately not implemented**, with its information-disclosure boundary made explicit rather than treated as a formatting choice; partial-fleet handling distinguishes the relatively benign `OFF`/`OBSERVE` observation gap from `OBSERVE`/`ENFORCE` request-acceptance inconsistency; and the timed rollback rehearsal is defined with its four measurements but **not executed** and **not owned by `THOTH-GQL-OPS-04`** — it, the service-health/activation-threshold gate and the preview/staging acceptance gate all remain **downstream** of the runtime-operations gate, so every timing field stays marked `TO BE MEASURED AT PREVIEW/STAGING GATE` and no numeric threshold or duration is invented anywhere. The `THOTH-GQL-OPS-02` remediation class remains feature-local and must **preserve all existing `init` migration and startup semantics**; a production container-command override is recorded everywhere it is mentioned as **not** an interchangeable fix, because `init` runs migrations before starting the API and an override would remove migration execution from the deployment path, and the mechanism is deliberately **not** selected here. Documentation and control only — no runtime, GraphQL, schema, migration, `Cargo`, `Dockerfile` or workflow change; no production configuration value, secret, resource identifier or private hostname recorded; all reads of the secret-bearing private deployment source were read-only, metadata-only and narrowly scoped, and the credential exposure encountered there is escalated rather than remediated. No deployment, no production action, no environment transitioned and no guard-enabled candidate activated. Terminal disposition **`C - insufficient operational capability/evidence; BLOCKED`**, with the `ADR-0006` runtime-operations gate recorded **NOT SATISFIED**: both capability gaps remain open, **CG-13 remains OPEN**, `THOTH-GQL-OPS-02` and `THOTH-GQL-OPS-03` remain blocking prerequisites and `THOTH-GQL-OPS-04` the earliest possible closure. Production `OBSERVE` and `ENFORCE` remain `NOT AUTHORIZED`, `BE-02` runtime remains `NOT AUTHORIZED`, [PR #788](https://github.com/thoth-pub/thoth/pull/788) and [issue #765](https://github.com/thoth-pub/thoth/issues/765) are unmodified, and the exact implementation head requires fresh independent exact-head review and separate explicit CTO merge authorization
   - [792](https://github.com/thoth-pub/thoth/pull/792) - `THOTH-GQL-OPS-01-SPEC`: add the `THOTH-GQL-OPS-01` specification for GraphQL mutation-guard **runtime operations** — which opens, and deliberately does **not** close, the runtime-operations gate in the approved `ADR-0006` activation sequence after the inert `THOTH-GQL-BATCH-01` foundation merged. It is a **bounded feature-specific successor** to control gap CG-13, covering only mode control for `THOTH_GRAPHQL_MUTATION_GUARD_MODE`: runtime owner, configuration authority, restart/redeploy semantics, propagation to every serving replica, fleet-wide verification of the **effective** mode, partial-fleet detection and handling, rollback, change/rollback authority, a timed preview/staging rehearsal requirement and retained audit evidence. Discovery established, at the exact base, that the mode is read **once at process start** with no reload path anywhere in the workspace, so a mode change always requires a new process; that **no** surface exposes the effective mode of a serving instance, leaving `OFF` and `OBSERVE` externally indistinguishable; and — reproduced in an isolated probe outside this repository against pinned `clap` 4.6.1 — that the image's default `init` command does **not** register the guard argument, so in a release build a **guard-enabled** process started through `init` silently ignores `THOTH_GRAPHQL_MUTATION_GUARD_MODE` and is unconditionally `OFF` (in a debug build the same path panics). It also established that merge, release and activation are three distinct states: the release currently deployed to production **predates** `THOTH-GQL-BATCH-01`, its binary contains no mutation guard at all, and it is therefore recorded as **pre-guard** rather than as `MutationGuardMode::OFF` — merging the foundation deployed nothing. That conclusion carries explicit evidence provenance and is attributed to no single source: `[REPO]` establishes only that the relevant release/`master` code contains no `MutationGuardMode`, defines no `mutation_guard_mode()` and has no guard wiring on its GraphQL startup path, while `[EXTERNAL]` — previously established scoped authoritative deployment metadata — establishes which release/image production actually runs, so current production is `PRE-GUARD` on `[REPO + EXTERNAL]` evidence together; a deployed-state conclusion may never be attributed to repository evidence alone, and where the external half is unobtainable under the scoped-read rules it is downgraded to `[UNVERIFIED]` rather than re-derived or obtained by widening access. That failure is silent but **fail-safe** and does not affect the correctness of the merged inert state; it is recorded as a bounded, separately specified, separately reviewed prerequisite of `OBSERVE`, explicitly **not** remediated here and explicitly **not** an architecture change. **These are capability gaps, not merely evidence gaps:** production configuration inherits the image's `init` command, so the current deployment path could not consume the guard-mode input once guard-enabled code is deployed — making an `OFF -> OBSERVE` transition of a guard-enabled candidate not operationally performable through that path — and no implemented mechanism could verify a change if one could be made — so `THOTH-GQL-OPS-01` is required to terminate at CG-13 disposition **`C - BLOCKED`** with the `ADR-0006` runtime-operations gate recorded **NOT SATISFIED**, and is forbidden from returning disposition `A` while either gap holds. Closing them requires separate bounded, independently reviewed and merged tasks — `THOTH-GQL-OPS-02` (mode-control path) and `THOTH-GQL-OPS-03` (fleet-verification mechanism) — after which `THOTH-GQL-OPS-04` may re-verify and decide on evidence; specifying a prerequisite is never delivery of it, and a specification for a verifier is never a verified fleet. The permitted `-02` remediation class is feature-local and must **preserve all existing `init` migration and startup semantics**; replacing the production container command with `start graphql-api` is **not** an interchangeable fix, because `init` runs migrations before starting the API and an override would remove migration execution from the deployment path, requiring separate migration/deployment-control analysis and approval. The specification requires every operational statement to carry an evidence class, forbids inventing any propagation or rollback duration, forbids inferring a runtime platform from the Docker image or release workflow, requires the smallest fleet-verification mechanism to be **specified but not implemented**, marks the runbook **PROVISIONAL**, binds all reads of the secret-bearing private deployment source to scoped, metadata-only retrieval, and keeps service-health signals and activation thresholds as the separate next gate. Documentation and control only — no runtime, GraphQL, schema, migration, `Cargo`, `Dockerfile` or workflow change, no production configuration value or secret recorded, no deployment, no environment transitioned and no guard-enabled candidate activated, and no modification of [PR #788](https://github.com/thoth-pub/thoth/pull/788), its branch or [issue #765](https://github.com/thoth-pub/thoth/issues/765). **CG-13 remains OPEN** and is only cross-referenced to the proposed successor; `THOTH-GQL-OPS-01` is `DRAFT` with implementation `NOT AUTHORIZED`, production `OBSERVE` and `ENFORCE` remain `NOT AUTHORIZED`, `BE-02` runtime remains `NOT AUTHORIZED`, and the exact specification head requires fresh independent exact-head review and explicit CTO specification approval
   - [791](https://github.com/thoth-pub/thoth/pull/791) - `THOTH-GQL-BATCH-01`: request-scoped GraphQL batching foundation implementing approved `ADR-0006`. Adds a request-scoped batch store on the GraphQL `Context` keyed by `(top-level response key, loader identity, normalized load shape, parent key)` with the three-state `NotLoaded` / `Loaded` (including `Loaded([])`) / `LoadFailed` model; a look-ahead-driven, alias-safe, synchronous set-based prefetch supporting both direct and **descendant** paths; a single isolated pinned-Juniper compatibility shim deriving the top-level response scope; and the central mutation request guard with modes `OFF` / `OBSERVE` / `ENFORCE` behind a baseline eligibility gate that leaves juniper the sole authority for parse, validation, operation-selection and input errors. **The merged state is inert**: the guard defaults to `OFF`, the store is derived-unavailable outside `ENFORCE` (so `OFF + store enabled` and `OBSERVE + store enabled` are structurally unrepresentable), no request-path overhead is added, and production request acceptance is unchanged. The foundation is adopted by **no production field**: no production child resolver, none of the 88 `MutationRoot` resolvers, and no public GraphQL contract changes — the generated SDL is byte-identical. No migration, no `schema.rs` change, no `policy.rs` change and no new dependency. Merge does **not** authorize activation: production `OBSERVE` and `ENFORCE` each require separate explicit CTO production activation approval and remain NOT AUTHORIZED, CG-13 and the monitoring/threshold blockers remain open, and `BE-02` runtime remains NOT AUTHORIZED
   - [790](https://github.com/thoth-pub/thoth/pull/790) - Record CTO approval of `ADR-0006` as merged through PR [789](https://github.com/thoth-pub/thoth/pull/789): request-scoped GraphQL batching / set-based loading, F2 response-key scoping, the central mutation request guard and its staged `OFF`/`OBSERVE`/`ENFORCE` controls are now the approved architecture decision. Documentation/control only; no architecture, runtime or production behaviour changes. `THOTH-GQL-BATCH-01` runtime implementation remains NOT AUTHORIZED, `OBSERVE` and `ENFORCE` remain NOT AUTHORIZED, CG-13 and the monitoring/threshold blockers remain open, and `BE-02` runtime remains NOT AUTHORIZED

docs/engineering/ai-delivery/README.md

@@ -30,7 +30,28 @@ These documents apply across Thoth engineering programmes unless a stricter appr
   evidence, never on repository evidence alone — and production activation
   state. It terminates at CG-13 disposition
   `C - BLOCKED`, leaves the `ADR-0006` runtime-operations gate `NOT SATISFIED`,
-  and activates nothing).
+  and activates nothing). Its delivered output is the
+  [operational-control record](../repository-map/graphql-mutation-guard-runtime-operations.md)
+  and the **provisional**
+  [mode-transition runbook](../repository-map/graphql-mutation-guard-mode-transition-runbook.md).
+- `tasks/THOTH-GQL-OPS-02.md` - mutation-guard mode-control path
+  (`DRAFT`; implementation `NOT AUTHORIZED`; branch must not exist). Makes
+  `THOTH_GRAPHQL_MUTATION_GUARD_MODE` consumable on the production-applicable
+  command path while preserving all existing `init` migration and startup
+  semantics. Closes capability gap 1. Making the mode settable is not setting it.
+- `tasks/THOTH-GQL-OPS-03.md` - effective-mode fleet-verification mechanism
+  (`DRAFT`; implementation `NOT AUTHORIZED`; branch must not exist). Implements
+  the smallest mechanism proving the effective mode of every serving instance,
+  with per-instance attribution and mixed-fleet detection, without affecting
+  request acceptance. Closes capability gap 2. A verifier is not a verified
+  fleet.
+- `tasks/THOTH-GQL-OPS-04.md` - bounded runtime-operations verification and
+  closure (`DRAFT`; implementation `NOT AUTHORIZED`; branch must not exist).
+  After `-02` and `-03` merge, re-establishes all external evidence, proves both
+  capabilities against the real runtime, finalises the runbook, and decides CG-13
+  disposition `A` or `C` on evidence. It is the earliest task that may record the
+  runtime-operations gate as `SATISFIED`, and it may return `C` again. It must
+  not activate `OBSERVE`.
 - `reviews/CTRL-FOUNDATION-01-review-brief.md` - independent review requirements.
 
 ## Core rule