AE control surface serves plaintext by default (CONTROL_TLS off) — dx bearer JWT crosses the CNI in cleartext

[ConstanzeTU]

Symptom

adaptive_export logs on every boot:

level=warning msg="control surface: TLS DISABLED — bearer JWT crosses the CNI in cleartext (set CONTROL_TLS=true)"

Root cause

The control surface (:9100 — the endpoint dx calls for /export/start, /dx/evidence_graph, /dx/evidence_manifest) requires a bearer JWT when CONTROL_REQUIRE_AUTH=true + PL_JWT_SIGNING_KEY is set (which it is in the dx-steered topology). But TLS on that same surface is gated on CONTROL_TLS=true, which is default-OFF (src/vizier/services/adaptive_export/cmd/main.go:724-737, comment: "Default-OFF for incremental rollout").

Result: auth is ON but the channel is plaintext, so the signed token (and the control payloads) travel the pod network unencrypted — anyone able to sniff the CNI on the node can capture a valid AE-control token.

Why it's currently off

• CONTROL_TLS is not set in the AE manifest (k8s/vizier/bootstrap/adaptive_export_deployment.yaml), and the service-tls-certs secret (the cert the broker/PEM already use, expected at /certs/server.{crt,key}) is not mounted into the AE DaemonSet.

Investigate / fix

1. Confirm the service-tls-certs (or equivalent) secret is available in pl and mountable into the AE DaemonSet.
2. Mount it at /certs + set CONTROL_TLS=true in adaptive_export_deployment.yaml.
3. Set dx's AE_CONTROL_ADDR to https://adaptive-export-control.pl.svc.cluster.local:9100 (the dx client already does TLS skip-verify for the in-cluster self-signed cert — see k8s/vizier/dx/dx-daemon.yaml comment).
4. Verify dx still steers + writes the evidence graph over TLS, and the warning becomes control surface: TLS ENABLED.
5. Decide whether default-ON is appropriate now that the rollout is past incremental (the cert infra already exists).

Refs

• src/vizier/services/adaptive_export/cmd/main.go:706-742
• k8s/vizier/dx/dx-daemon.yaml (AE_CONTROL_ADDR http, with the flip-to-https comment)
• observed on the sovereign-SOC lab (fix/ae-protocol-export-pxexport, aeprod63)

[ConstanzeTU]

Public-release plan — make the control surface secure-by-default (auth + TLS)

Promoting this to a release blocker and widening scope from "TLS off" to the full secure-default. Today
both gates are default-OFF (cmd/main.go:699 auth, :724 TLS): auth only turns on with
CONTROL_REQUIRE_AUTH=true, TLS only with CONTROL_TLS=true, and neither is set in the shipped manifest —
so out-of-the-box the dx→AE bearer JWT crosses the CNI in cleartext (boot logs the two DISABLED warnings on
every PG run we've done).

Targets (acceptance criteria)

• T1 — no cleartext by default: a default AE deploy serves the control surface over TLS, and requires
  the bearer JWT when a signing key is present. Boot logs TLS ENABLED + auth ENABLED, not the two
  DISABLED warnings.
• T2 — canonical dx-steered deploy still works: dx connects https://…:9100 with skip-verify + bearer
  (the dx client already supports this — see k8s/vizier/dx/dx-daemon.yaml AE_CONTROL_ADDR comment). dx
  still steers + writes dx_evidence_graph/dx_evidence_manifest over TLS.
• T3 — fail-closed, explicit opt-out: insecure (plaintext / no-auth) is only reachable via an explicit
  CONTROL_INSECURE=true with a loud warning — never the silent default. Without a signing key and without
  CONTROL_INSECURE, the control surface refuses to start (fail-closed).

Planned change (stacks on PR #92 branch fix/ae-protocol-export-pxexport)

1. cmd/main.go — flip the defaults:
   • TLS default ON. Serve TLS using /certs/server.{crt,key} when mounted; if absent, self-generate an
     ephemeral in-memory self-signed cert (mirror dx's DX_RECEIVER_TLS self-signed receiver) so TLS works
     with zero extra secrets. Plaintext ONLY if CONTROL_INSECURE=true.
   • Auth default ON when PL_JWT_SIGNING_KEY is present (drop the extra CONTROL_REQUIRE_AUTH=true
     requirement). No key + no CONTROL_INSECURE ⇒ fail-closed (don't serve the control surface).
   • Keep CONTROL_TLS_CERT/KEY as overrides; CONTROL_REQUIRE_AUTH/CONTROL_TLS become deprecated no-ops
     (secure is the default); add CONTROL_INSECURE.
2. Deploy manifests: drop the now-unnecessary manual CONTROL_TLS=true; mount service-tls-certs at
   /certs in the AE DaemonSet if using the shared cert (else self-gen). Flip dx AE_CONTROL_ADDR
http://… → https://… in k8s/vizier/dx/dx-daemon.yaml and entlein/dx deploy/dx-daemon.yaml.

Test plan

• Unit (control/…_test.go): server serves TLS by default (self-gen cert path when /certs absent);
  rejects unauthenticated requests when a key is present; CONTROL_INSECURE=true ⇒ plaintext + loud warn;
  fail-closed when no key and not insecure.
• Rig: default AE deploy on a live PG → dx (https) steers + writes the evidence graph + manifest; boot
  log shows TLS ENABLED + auth ENABLED, no cleartext warning; the §H/§I demo join still resolves.

[ConstanzeTU]

Validated live on rig 6a81a7a6 (aeprod64) — secure-by-default ✅

Deployed aeprod64 over the rig's stock AE (which had CONTROL_ADDR=:9100 + PL_JWT_SIGNING_KEY but no /certs mount):

• Boot: control surface: bearer-JWT auth ENABLED (audience=vizier) + control surface: TLS ENABLED (self-signed) — the self-generated cert fallback engaged (no cert mounted), so it's secure with zero extra secrets. (Old 0.14.20 logged auth DISABLED + TLS DISABLED.)
• Live probes (in-cluster curl to adaptive-export-control.pl.svc:9100):
  • https /healthz → 200 (TLS serving)
  • http /healthz → 400 (plaintext rejected — TLS-only, no cleartext)
  • https /export/start no bearer → 401 (auth enforced)

T1 + T3 confirmed. T2 (dx steering over https) covered by the AE_CONTROL_ADDR http→https manifest flip (dx skip-verifies the self-signed cert).