add the rpc namespace: connect generation with buf, no docker - #17
Merged
Conversation
The twirp namespace runs protoc inside the elephant-twirptools image, which exists for one reason: protoc is a C++ binary, so pinning it meant shipping a container. Everything else in the image is Go. buf is a pure-Go compiler that drives the same plugins, so the whole toolchain becomes "go run <module>@<version>" with the versions as constants here, and generating needs no Docker, installs nothing and takes nothing off PATH. That is the same pin the image was, moved into version control, and it is what lets the fleet generate Connect code alongside Twirp during the migration off it. The generation template is passed to buf inline with --template, so no repository commits a buf.gen.yaml and no repository can drift from the pinned plugin set. A buf.yaml is written only where the workspace needs one, which is a repository that vendors a proto file: a workspace cannot reach outside its root, so the trick of handing protoc a dependency's module directory as a --proto_path has no equivalent, and rpc:vendorProto copies the file in and makes its directory a module root of its own. protoc-gen-elephant-rpc is in the plugin set but has no release to pin yet, so an empty version means the plugin is skipped, and ELEPHANT_RPC_PLUGIN runs it from a checkout while it is being written. Two things could not stay as they were. protoc-gen-openapi3 is built against a protobuf-go that reports FEATURE_SUPPORTS_EDITIONS without a minimum edition, which protoc accepts and buf rejects outright, so that plugin is driven directly from the target with the descriptors buf compiled - the same exchange buf would have made, and it produces byte-identical specifications. And a go_package written as a relative path, which the stub template produced, is not an import path, which never mattered while everything was generated into one directory and does the moment Connect generates into a subpackage; rather than making every repository edit its .proto files, the import path is derived from the module path and passed to the plugins. Checked against copies of elephant-collab and elephant-api: the OpenAPI specifications come out byte-identical, the twirp files differ only in their gzipped descriptor blob, and the .pb.go differences are the protoc-gen-go version bump the image was holding back.
protoc-gen-openapi3 documents the Twirp surface: the paths it emits are "/twirp/<pkg>.<Service>/<Method>" and the errors it declares are Twirp's. With Twirp off by default and OpenAPI on, the first rpc:generate in a new Connect-only repository committed a specification of an API that repository does not serve — and those specifications are linked from every service README and read by external customers, so the wrong document is worse than none. rpc.OpenAPI becomes an OpenAPISetting whose zero value follows Twirp, with OpenAPIOn and OpenAPIOff for the repositories that want to decide for themselves. RPC_OPENAPI still overrides it for a single run. Nothing changes for a repository that serves /twirp/: it sets rpc.Twirp and keeps its specifications.
The plugin's adapters take and return the plain service interface, so something has to declare it. protoc-gen-twirp does while a repository still generates Twirp, and nothing does when it stops: a Connect-only repository generating with the plugin got adapters that could not compile. The interface option now follows rpc.Twirp the way the OpenAPI specifications do, and rpc.ElephantRPCOptions overrides it in either direction, for a repository that wants the generated interface while it still serves the /twirp/ paths, or that declares the interface itself. The end-to-end test now generates the fixture with the real plugin in both configurations, rather than only with the fixture stub. The stub says nothing about the Go import path mappings reaching the plugin or about what it emits compiling against the Connect code generated beside it, which is what the wiring has to get right. The plugin has no released version to pin, so the test is skipped unless ELEPHANT_RPC_PLUGIN names a checkout or a module version - the same variable that generates a repository with it.
rpc:generate asked git for the last ancestor tag before it had loaded the configuration, so it failed outright in a repository with no tags. That is exactly where a new Connect-only service starts: with Twirp off the OpenAPI step is skipped, the resolved version is never used, and the only thing standing between the repository and its first generated code was a tag it has no reason to have yet. The version is now a resolver that generateAll calls behind the OpenAPI guard, so it is asked for only when a specification is actually written.
Nobody consumed the OpenAPI 3 specifications, they described the Twirp surface only, and protoc-gen-openapi3 cannot run under buf at all, so the rpc namespace generates Go and nothing else. With nothing left to stamp a version into, rpc:release goes too: a release is a plain git tag, and no target consults git describe any more. The module no longer needs google.golang.org/protobuf, which only the hand-rolled driver for the OpenAPI plugin used.
…live under rpc A repository that keeps its services in the repository root and vendors an import got an rpc/vendor directory from rpc:vendorProto, and the mere existence of rpc/ made the targets believe the sources had moved there. Service discovery then found only the excluded vendor directory and generation failed with "no rpc/*/service.proto files to generate from". The proto root is rpc only when that directory holds something other than the vendored protos. An empty rpc directory still counts, so a repository can choose the layout before it has a service to put in it. The regression test moves the vendoring fixture to the root layout.
The plugin was skipped for want of a version to pin. elephantine's feature/connect-rpc branch is pushed, so its pseudo-version is the pin until the work is tagged, and the end-to-end plugin test runs by default rather than only under ELEPHANT_RPC_PLUGIN.
Go 1.27 is the fleet's floor from this round on, and mage is the module every repository's magefile builds against, so it goes first. The generation fixtures move with it; rpc/testdata/greeter is left on the old floor deliberately, as the fixture for a repository that has not moved.
The generated Twirp code depended on the machine: protoc-gen-twirp has no go.mod, so "go run <module>@<version>" resolved its dependencies afresh on every run and compiled it with whatever toolchain was around, and the gzipped descriptor it embeds comes out of compress/flate, which changed between Go 1.26 and Go 1.27. The same fixture gave 180 and 176 bytes of descriptor on two toolchains. It now runs out of a module carried here with a complete go.sum, and GOTOOLCHAIN is set to a pin for every generator, which is what makes the output the same on both. A toolchain directive would not have done it, since that is a floor the go command overrides with a newer local toolchain. The same invocations drop a -mod flag out of GOFLAGS, which a repository that vendors its dependencies would otherwise use to send the generators looking for themselves in its vendor directory. Two configurations are refused rather than generated, both of which produced code that does not compile: Twirp together with the plugin's interface=true, since they write the same declaration, and a go_package naming an import path other than the one the generated code lands under. An unpinned protoc-gen-elephant-rpc is an error too, where it used to skip the plugin and report success with the adapters left as they were, and whichever of service.rpc.go and service.twirp.go is not generated is removed if an earlier configuration left it behind. Service discovery takes the versioned layout as well, and rpc:stub scaffolds it, so a repository can move to buf's convention a service at a time. A hand-written buf.yaml that does not declare the vendored proto root is reported rather than left as a workspace buf cannot resolve.
The plugin's own fixes for the review landed on feature/connect-rpc after the previous pin was taken, and the corrected release order tags mage first with this pseudo-version, then elephantine, then repoints this constant at the tag.
The module moved to go 1.27.1 and the v2.12 binary the lint job used is built with Go 1.26, which refuses a target newer than itself. v2.13 is built with Go 1.27; the configuration verifies and the code lints clean under it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The platform is moving from Twirp to Connect and will serve both for a long dual-stack period. This adds the
rpcmage namespace that generates the Connect code alongside the messages, and replaces theelephant-twirptoolsDocker image as the way the fleet runs its protobuf generators.What
rpc:generatedoes. buf is the compiler and every plugin runs out of a module pinned in therpcpackage, so generating needs no Docker, installs nothing and takes nothing offPATH. Per service it writesservice.pb.go,<pkg>connect/service.connect.go(connect-go),<pkg>connect/service.elephant.go(the adapters fromprotoc-gen-elephant-rpcin elephantine that keep the plain service interface as the contract),service.rpc.go(that interface, when Twirp is off) andservice.twirp.gowhenrpc.Twirp = true. Services are discovered in both layouts,<root>/*/service.protoand<root>/*/v*/service.proto, andrpc:stubscaffolds the versioned one.rpc:vendorProtocopies a cross-module.proto(the fleet's one case isnewsdoc/newsdoc.protofrom elephant-api) intorpc/vendorand maintains thebuf.yamlbuf needs for it, since a buf workspace cannot reach into the module cache the way protoc was handed a--proto_path.Reproducibility.
protoc-gen-twirpis a+incompatiblemodule with nogo.mod, and its gzipped descriptor differs between Go 1.26 and 1.27, so it runs from an embedded module with a completego.sumunder a pinned toolchain (rpc.GeneratorToolchain), and a test generates under both toolchains and compares.GOFLAGSandGOTOOLCHAINare set for the generator invocations rather than inherited. Two configurations that would produce uncompilable code are refused:rpc.Twirptogether with the plugin'sinterface=true, and ago_packagenaming an import path other than the one the code lands under. An unpinnedprotoc-gen-elephant-rpcis an error, not a silent skip.Removed. OpenAPI generation does not carry over: nobody consumed the specifications, they described the Twirp surface only, and the generator cannot run under buf. There is no
rpc:release; a release is a plain tag. Thetwirpnamespace stays, deprecated, until the image is retired.Go 1.27.1 is the module's floor, recorded as a breaking lead-in since every magefile importing these targets needs that toolchain.
Release coupling.
rpc.ElephantRPCVersioncurrently pinsprotoc-gen-elephant-rpcto a pseudo-version of elephantine'sfeature/connect-rpcbranch. The order is: tag this module with that pin, tag elephantine, re-pin here to the tag and tag again, then the declaration modules. The CHANGELOG and README say so.Adopted on feature branches by elephant-api, elephant-tt-api, elephant-public-api, bifrost-api and elephantine, all of which regenerate to a zero-byte diff against their committed output.