Skip to content

Add registrar endurance scenario (#783) - #977

Merged
sehkone merged 14 commits into
mainfrom
sehkone/issue-783
Sep 1, 2026
Merged

Add registrar endurance scenario (#783)#977
sehkone merged 14 commits into
mainfrom
sehkone/issue-783

Conversation

@sehkone

@sehkone sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add the extended-tier registrar renewal scenario and Cargo launcher, including isolated deployment setup, six-minute leaves, renewal cadence overrides, expiry-bound certificate checks, unchanged root-anchor pinning, and external strace observations.
  • Register registrar-endurance with the existing extended-suite lifecycle and artifact summary.
  • Make the core preflight honor configured Step CA and responder host ports so isolated runs verify the stack they launch.
  • Give deadline-triggered teardown five minutes to remove and verify all run-scoped resources, pass the documented temporary AppRole mutation its watched credential paths, and keep concurrently launched endurance deployments distinct.
  • Preserve a short endurance RUN_TOKEN intact and truncate only longer tokens, retaining the PID tail that distinguishes concurrent runs.

Closes #783.

Part of #784.

Validation evidence

Test plan

  • Run scripts/preflight/ci/check.sh.
  • Run scripts/validate-e2e-run-scope.sh, including short and long endurance-token coverage.
  • Run scripts/preflight/ci/test-core.sh with isolated host-port overrides: the unit suite and monitoring check passed, but its later init correctly stopped on existing initialized OpenBao state for this worktree.
  • Run the no-hosts arm of scripts/preflight/ci/e2e-matrix.sh: local and remote lifecycle phases passed; reinit recovery stopped because Docker reports 172.17.0.1 is not assigned to a local interface.
  • Run the hosts arm locally (this machine lacks passwordless sudo; CI remains the gate).
  • Verify the temporary AppRole-routing mutation makes the daemon trace parser fail, then revert it and rerun unmodified.
  • Verify the manual extended E2E workflow for the final head.

@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Temporary AppRole trace proof

For the negative control, I temporarily replaced read_acme_inputs with an AppRole login using the run-scoped role_id and secret_id files. The temporary commit was removed from the submitted branch before the final clean run.

diff --git a/src/registrar_certs.rs b/src/registrar_certs.rs
index d0afcbe5..20370b1d 100644
--- a/src/registrar_certs.rs
+++ b/src/registrar_certs.rs
@@ -88,6 +88,7 @@ use crate::config::{
     DaemonProfileSettings, HookSettings, Paths, RegistrarEndpointSettings, Settings,
 };
 use crate::eab::EabCredentials;
+use crate::openbao::OpenBaoClient;
 use crate::registrar::internal::{
     InternalCredential, InternalPaths, active_root_fingerprint, load_internal_config,
 };
@@ -740,18 +741,23 @@ pub(crate) async fn read_acme_inputs(
     openbao_url: &str,
     kv_mount: &str,
 ) -> Result<SurfaceAcmeInputs> {
-    let active_root = active_root_fingerprint(secrets_dir).with_context(|| {
-        format!(
-            "reading the deployment's active root fingerprint below {}",
-            secrets_dir.display()
-        )
-    })?;
-    // The credential's own active-root refusal stands: a start inside a
-    // trust-rotation window refuses with its diagnostic, and finishing
-    // the rotation is the remedy. Nothing here retries around it.
-    let credential = InternalCredential::load(secrets_dir, openbao_url, &active_root)
-        .context("loading the bootroot-internal credential for the registrar surface issuance")?;
-    read_acme_inputs_with(&credential, kv_mount).await
+    let approle_dir = std::env::var_os("BOOTROOT_REGISTRAR_ENDURANCE_APPROLE_DIR")
+        .map(PathBuf::from)
+        .context("reading the registrar endurance AppRole credential directory")?;
+    let role_id_path = approle_dir.join("role_id");
+    let secret_id_path = approle_dir.join("secret_id");
+    let role_id = std::fs::read_to_string(&role_id_path)
+        .with_context(|| format!("reading {}", role_id_path.display()))?;
+    let secret_id = std::fs::read_to_string(&secret_id_path)
+        .with_context(|| format!("reading {}", secret_id_path.display()))?;
+    let mut client = OpenBaoClient::with_local_trust(openbao_url, secrets_dir)
+        .context("creating the TLS-configured OpenBao client for registrar surface issuance")?;
+    let token = client
+        .login_approle(role_id.trim(), secret_id.trim())
+        .await
+        .context("authenticating to OpenBao with the registrar endurance AppRole")?;
+    client.set_token(token);
+    read_acme_inputs_with_client(&client, kv_mount).await
 }
 
 /// Reads the two inputs through an already-loaded credential.
@@ -772,7 +778,13 @@ pub(crate) async fn read_acme_inputs_with(
         "authenticating to OpenBao with the bootroot-internal certificate for the registrar \
          surface issuance",
     )?;
+    read_acme_inputs_with_client(&client, kv_mount).await
+}
 
+async fn read_acme_inputs_with_client(
+    client: &OpenBaoClient,
+    kv_mount: &str,
+) -> Result<SurfaceAcmeInputs> {
     let eab_value = client
         .read_kv(kv_mount, PATH_AGENT_EAB)
         .await

The expected-failing workflow 33468509595 rebuilt that variant, completed the post-expiry client and root-pinned endpoint assertions, then produced these watched-path matches:

pid=75297 syscall=openat pathname=/tmp/bootroot-registrar-endurance-uoczyW/approle-control/role_id result=10
pid=75297 syscall=openat pathname=/tmp/bootroot-registrar-endurance-uoczyW/approle-control/secret_id result=10
pid=75298 syscall=openat pathname=/tmp/bootroot-registrar-endurance-uoczyW/approle-control/role_id result=10
pid=75298 syscall=openat pathname=/tmp/bootroot-registrar-endurance-uoczyW/approle-control/secret_id result=10
[fatal][renewal-window] daemon renewal trace opened an AppRole control credential path

The clean branch was then restored and force-pushed without either temporary commit. 33470380715 is the succeeding post-revert extended run.

@sehkone sehkone changed the title Add registrar endurance scenario Add registrar endurance scenario (#783) Sep 1, 2026
@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

[Reviewer Round 1]

Changes requested.

  • P2 — Cleanup failures are deliberately ignored in run-registrar-endurance.sh. A failed docker compose down, image removal, tmpfs unmount, or run-root removal still returns the successful test status. That can leave containers, volumes, a mounted tmpfs, or images behind while reporting a green scenario, violating the issue’s cleanup-on-success-and-failure requirement. Preserve the original assertion failure, but make cleanup failure fail an otherwise successful run and verify no run-scoped resources remain.

  • P3 — Three PR commits contain literal \n\n text instead of real body line breaks (60f2b8b, 1c4a298, 863bd72), contrary to the repository commit-message rules. Reword them before merge.

@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

[Review Verdict Round 1: NOT_APPROVED]

Prove registrar leaf renewal past expiry in the scheduled Docker suite
while observing that renewal never reads AppRole credentials.

Closes #783
GitHub run tokens can exceed Bootroot's deployment-name limit.
Retain a sufficiently unique prefix while preserving the full token for
artifacts and images.

Part of #783
Early failures can occur before initialization creates Compose values.
Provide interpolation-only values so the run-scoped deployment is always
removed.

Part of #783
The registrar-endurance prefix occupies twenty characters. Limit its
run-token suffix to nineteen so every generated instance name meets the
39-character deployment constraint.

Part of #783
The isolated no-EAB deployment must distinguish an explicit clear EAB
record from an absent KV entry before the registrar issues its leaves.

Part of #783
Registrar endpoint callers pin a CA anchor, but ACME can return only an
intermediate. Include configured anchors in the surface chain so renewed
candidates remain verifiable by their unchanged root pin.

Part of #783
The external red-team client must encode the registrar protocol exactly
so a post-expiry mint proves the renewed endpoint responds.

Closes #783
Permit declared host-port overrides to isolate preflight runs when a
developer host already has a default listener. Keep registrar renewal
publication on the existing bootstrap repair path until the normal
bundle merge validates the target.

Part of #783
Keep the core preflight aligned with configured Compose port overrides
so an isolated run verifies the stack it started.

Part of #783
A green endurance scenario must also prove that it removed every
resource it owned. Preserve an earlier assertion failure while surfacing
teardown and leftover failures from an otherwise successful run.

Part of #783
@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

[Author Round 1]

Addressed and pushed to PR #977 at 512448a6.

  • Fixed — Cleanup failures now fail an otherwise-successful endurance run while preserving an original assertion failure. It removes and verifies run-scoped containers, volumes, networks, image, tmpfs, and run root.
  • Fixed — Rewrote the three flagged commit bodies with real wrapped newlines; normalized the remaining malformed branch commit bodies too.

Validation:

  • scripts/preflight/ci/test-core.sh passed with isolated port overrides.
  • E2E matrix no-hosts arm passed; hosts arm is blocked by unavailable passwordless sudo.
  • validate-e2e-leftover-check.sh, bash -n, and diff checks passed.
  • Full preflight’s deploy smoke remains blocked by OrbStack holding port 9000.

@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

[Reviewer Round 2]

Follow-up review: changes requested.

Round 1 items are resolved: cleanup failures now affect a successful run, and the commit bodies no longer contain literal \n text.

  • P1 — The 20-minute wrapper can kill cleanup before it completes. run-registrar-endurance.sh allows only 30 seconds after TERM, while cleanup permits docker compose down up to 90 seconds at line 137. If the deadline fires during teardown, timeout kills the shell before its leftover checks and removals finish, violating cleanup-on-failure. Bound the assertion phase separately or give cleanup enough independent time to complete and verify removal.

  • P1 — The supplied negative-control mutation is not reproducible with the submitted harness. Its read_acme_inputs patch reads BOOTROOT_REGISTRAR_ENDURANCE_APPROLE_DIR, but the scenario only creates APPROLES_DIR and never exports that variable; the traced daemon is launched without it at line 427. Applying the documented source-only mutation would fail before opening either control file, so it cannot produce the claimed parser matches. Pass the run-scoped directory into the traced daemon environment, then rerun and update the proof.

  • P2 — The PR says workflow run 33478951396 is for the final commit, but it ran 4fdad172, not current head 512448a6. The current cleanup implementation therefore has no linked extended-workflow evidence, despite the issue explicitly requiring a manually dispatched run. Dispatch and link a successful run for the submitted head.

@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

[Review Verdict Round 2: NOT_APPROVED]

Give timeout-triggered teardown enough time to remove and verify every
run-scoped resource. Supply the temporary negative-control variant with
its watched AppRole paths so its external trace evidence is reproducible.

Part of #783
@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

[Author Round 2]

Addressed and pushed 3b71e0d7 (Protect endurance cleanup) to PR #977.

  • Fixed — Timeout-triggered cleanup now receives a five-minute grace period, exceeding the 90-second Compose teardown limit so removal and leftover verification can complete.
  • Fixed — The traced daemon now receives the run-scoped BOOTROOT_REGISTRAR_ENDURANCE_APPROLE_DIR, making the documented temporary AppRole mutation reproducible.
  • Partially addressed — Dispatched and linked the current-head extended workflow: run 33487637188. It is still in progress, so no success claim is made yet.

Validation passed: scripts/preflight/ci/check.sh, bash -n, scripts/validate-e2e-leftover-check.sh, and git diff --check.

The local no-hosts E2E matrix reached reinit recovery, then stopped because Docker reported 172.17.0.1 is not a local interface; the hosts arm remains unavailable due to missing passwordless sudo.

@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

[Reviewer Round 3]

[Reviewer Round 3]

Changes requested.

Round 2 items are resolved: cleanup has a five-minute grace period, the temporary AppRole mutation receives its control directory, and the linked extended workflow succeeded on the submitted head.

  • P1 — Concurrent extended-suite runs can use the same deployment instance. run-extended-suite.sh supplies RUN_TOKEN="registrar-endurance-$$", which becomes registrarendurance<PID> after sanitizing. run-registrar-endurance.sh then retains only its first 19 characters. Since registrarendurance is already 18 characters, the generated instance retains only the first PID digit. For example, PIDs 12345 and 19876 both produce registrar-endurance-registrarendurance1. Those runs can share containers, volumes, networks, and teardown targets, violating the required run scoping and risking one run deleting another’s deployment. Retain the discriminating token tail or pass a short unique token from the suite.

@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

[Review Verdict Round 3: NOT_APPROVED]

Keep the concurrent suite process identifier inside the bounded
instance name so one endurance run cannot clean up another.

Part of #783
@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

[Author Round 3]

Fixed — preserved the discriminating tail of long run tokens and changed the suite token to a compact PID-based value, preventing concurrent endurance runs from sharing instance resources.

Pushed commit e2e95f2b to sehkone/issue-783.

Validation passed:

  • scripts/preflight/ci/check.sh
  • scripts/preflight/ci/test-core.sh
  • scripts/validate-e2e-run-scope.sh
  • scripts/validate-e2e-leftover-check.sh

The PR description was updated. A fresh Extended E2E run for the pushed SHA is in progress.

The extended suite must pass every launcher input in one command
environment. A comment inside the continuation separated those assignments
and made the scenario fail before setup.

Part of #783
@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

[Reviewer Round 4]

[Reviewer Round 4]

Changes requested.

  • P1 — The prior run-scoping finding remains unresolved. run-registrar-endurance.sh:56 takes ${RUN_TOKEN: -19}, but Bash returns an empty string when the token is shorter than 19 characters. The suite now supplies e-$$ (run-extended-suite.sh:140), so e2542 becomes empty and every suite invocation uses INSTANCE=registrar-endurance-. The successful linked workflow confirms this: its image tag included e2542, while its containers, volumes, and network were all named registrar-endurance-… without that suffix. Concurrent runs therefore still share teardown targets and can delete each other’s resources. Preserve the whole short token, or conditionally truncate only tokens exceeding the 19-character budget.

@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

[Review Verdict Round 4: NOT_APPROVED]

A short token could collapse during truncation, causing concurrent
extended runs to target the same Docker resources. Retain it whole
when it fits and otherwise preserve the distinguishing tail.

Part of #783
@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

[Author Round 4]

Fixed — The endurance launcher now preserves short RUN_TOKEN values and truncates only tokens longer than 19 characters, retaining unique PID tails. Added run-scope validation coverage.

Committed and pushed: 8828c8a0 Preserve short endurance run tokens.

PR description updated; manual extended E2E run for this head is queued: run 33509187670.

Validation passed: quality check and validate-e2e-run-scope.sh. Local core/matrix runs were blocked by pre-existing environment state/bind-host limitations; PR CI is pending.

@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

[Reviewer Round 5]

[Reviewer Round 5]

Approved.

Round 4 is resolved: the launcher now preserves short run tokens and only truncates longer ones in run-registrar-endurance.sh. The current-head extended run succeeded and its artifacts show a distinct registrar-endurance-e<PID> deployment, successful post-expiry mint and pinned endpoint checks, correct control-trace matches, and an empty daemon-trace match file.

No remaining prior items or new findings. The PR linkage and test-plan hygiene are correct.

[Review Verdict Round 5: APPROVED]

@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

[Review Verdict Round 5: APPROVED]

@sehkone

sehkone commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Suggested squash commit

Title

Add registrar endurance scenario

Body

Prove registrar client and endpoint certificates renew past their
original expiry without reading AppRole credentials. Run the assertion in
the extended Docker suite with isolated resources, predictable renewal
cadence, unchanged root-anchor pinning, and external strace evidence.

Harden the supporting launch and preflight paths so the scenario handles
custom ports, cleanup failures, and concurrent run tokens safely.

Closes #783
Part of #784

@sehkone
sehkone merged commit 744b26d into main Sep 1, 2026
26 checks passed
@sehkone
sehkone deleted the sehkone/issue-783 branch September 1, 2026 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add the registrar-endurance scenario for the renewal properties

1 participant