feat: automatic ACME certificate rotation and integration tests - #13
Open
phaus wants to merge 21 commits into
Open
feat: automatic ACME certificate rotation and integration tests#13phaus wants to merge 21 commits into
phaus wants to merge 21 commits into
Conversation
added 21 commits
September 5, 2026 00:57
Adds a new pkg/autocert package that orchestrates Let's Encrypt certificate provisioning and renewal via go-acme/lego v4.35.2. - Config (HTTP-01/DNS-01 challenge selection, email, CA URL) - Account creation/persistence interface - HTTP-01 challenge handler (/.well-known/acme-challenge/) - DNS-01 wiring for lego's built-in autodns provider - Obtain/Renew certificate flow with PostgreSQL storage interface - Background renewal loop Also vendors lego and its transitive dependencies. Because lego v4 requires newer x/net, x/crypto, grpc, protobuf etc., the whole graph is upgraded; go.mod now declares go 1.25.0 (required by the dep graph). The legacy Azure SDK used by blobstore/backend/azure.go is pinned back to the previously-vendored version via a replace directive so that backend continues to compile unchanged.
The CI runner installs Go 1.22.12. With go 1.25.0 in go.mod it downloaded the Go 1.25 toolchain, which then failed to find the covdata tool when running -race -cover tests for pkg/rpcplus/comborpc and fdrpc. Revert to go 1.23 so the downloaded toolchain matches the project baseline and coverage works.
The project now depends on packages (e.g. cloud.google.com/go/auth) that require Go >= 1.25.0. The CI runner was installing Go 1.22.12, causing toolchain downloads and missing the covdata coverage tool. Install Go 1.25.0 directly in both build and test jobs.
Adds a self-contained integration test (TestManagerObtainWithPebble) that starts a local Pebble ACME test server, runs the autocert HTTP-01 challenge flow, and verifies the issued certificate chains back to the Pebble root CA. Test is gated by the ACME_PEBBLE_TEST environment variable because it downloads and runs the letsencrypt/pebble module. Also adds an optional HTTPClient field to autocert.Config so tests can inject a TLS client that trusts the test CA. Includes the pebbletest helper as a separate Go module to keep Pebble out of the main module's vendor tree.
fileSystemStore.Commit's needsRemoval strips the consistent-snapshot hash prefix via strings.SplitN(...,".",2). Target paths with no dot (e.g. channels/stable) produced a length-1 slice and panicked, aborting the release commit before stale-target pruning and staged cleanup completed. Guard against a missing hash separator and fall back to checking the path as-is.
Restore the IsCgroup2UnifiedMode() guard around CheckCpushares (cpu.shares only exists on v1) and the notifyOnOOMV2 implementation (memory.events via inotify) that were dropped when upgrading vendored runc. Without the guard, every job on a cgroup v2 host crashed in procHooks with ENOENT on cpu.shares.
Register ACME cert routes under /certs/letsencrypt/domains/:domain so they do not conflict with the static /certs/letsencrypt/config routes (httprouter forbids static and wildcard siblings at the same tree level). Update the API client and tests to match.
Refresh util/ca-certs/ca-certs.pem (was a frozen 2016 Mozilla bundle, missing ISRG Root X1, so ACME/Let's Encrypt TLS verification failed from containers). Contents now the current curl.se cacert.pem (Aug 2026). Update the copy mapping in script/export-tuf (and builder/manifest.json) from /etc/ssl/certs/ca-certs.pem to /etc/ssl/certs/ca-certificates.crt, which is one of the paths Go's crypto/x509 loads on Linux; the old filename was never used by Go, so containers could not verify public CAs.
syncRouteCert reused the route's stale UpdatedAt for the event dedupe hash (unlike the normal update path, which scans back a fresh updated_at from http_route_update), and CreateEvent has no ON CONFLICT, so the ACME cert sync event collided with the route update event that bound the ACME domain. Provisioning/renewal then failed with SQLSTATE 23505 (events_unique_id_idx) and the router was never notified of the new certificate. Refresh route.UpdatedAt before creating the event so each sync emits a distinct route event and the router picks up the new certificate. Also make controller/data/acme_test.go actually runnable against PostgreSQL: migrate to the latest schema and reconnect with prepared statements (named queries previously failed with syntax errors), and truncate test timestamps to timestamptz microsecond precision. Without the fix, TestACMECertSyncsBoundRoutes fails with the exact 23505 error seen on the live cluster.
The dashboard binary reads its compiled assets from disk relative to its working directory (dashboard/bindata.go Asset() falls back to the filesystem). export-tuf never included dashboard/app/build, so the web process panicked at startup with "open app/build/assets/manifest.json: no such file or directory". Add ExtraDirs mapping dashboard/app/build to /app/build, matching the controller's schema dirs pattern.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #12 (continuation).
This PR finishes the remaining data-layer and rotation work for Let's Encrypt ACME certificate management:
acme_domaincolumn tohttp_routes. When an ACME certificate is provisioned or renewed,controller/data/route.gonow syncs the new certificate to all bound routes and fires route events so the router hot-swaps without requiringflynn route update --acme <domain>to be re-run.flynn route update <id> --acme <domain>persistsacme_domainon the route while still snapshotting the current cert; the field is cleared when the route is switched back to manual-c/-k.controller/data/acme_test.gocoversACMEStoreaccount/certificate CRUD, upsert, domain normalization, deletion, andRouteRepo.SyncACMECertroute binding + renewal rotation (requires PostgreSQL to run).acme_domain; the route JSON schema is updated.Test Results
go build ./...cleango test -vet=off ./pkg/autocert/...→ 7 passedgo test -vet=off ./controller/ -run TestACME→ 7 passedcontroller/data/acme_test.gointegration tests compile (require Postgres).Commits
router: add acme_domain field to Route and HTTPRoutecontroller/data: add acme_domain column and ACME cert route synccontroller: wire ACME store to sync route certs on renewalcli: persist acme_domain on route update --acmecontroller/data: add ACMEStore integration tests