Modularize orchestrator wiring: extract queue profiles, topic registry builder, and controller registration by JamyDev · Pull Request #290 · uber/submitqueue

JamyDev · 2026-06-30T15:50:52Z

Summary

Adds an RFC at doc/rfc/submitqueue/modular-queue-wiring.md proposing 5 extractions to make the orchestrator's queue setup more modular:

Promote queueExtensions/queueRegistry into submitqueue/core/queueprofile
Extract declarative topic-registry builder into submitqueue/core/topicregistry
Extract controller registration helpers into submitqueue/orchestrator/controller/wire
Add data-driven profile hints to QueueConfig (YAML-driven extension selection)
Simplify example orchestrator main.go from ~950 to ~300 lines

Also adds the RFC to doc/rfc/index.md. Doc-only change, no code modifications.

Test Plan

CI/Build validation

Revert Plan

git revert

API Changes

No API changes

Monitoring and Alerts

Standard monitoring via existing dashboards

🤖 Generated by minion
Co-Authored-By: minion-bot noreply@minion.uber.com
Co-Authored-By: jamy@uber.com
Agent: claude-code | Task: b069bea1-12a5-478a-b517-61ce0abd6749

View your task: https://minion.uberinternal.com/task/b069bea1-12a5-478a-b517-61ce0abd6749

AI Verification

Verified at 3d8d47f on Jul 13 22:32 · 2 files analyzed · 1s

Verifier	Status	Issues
ureview	completed	0

0 issues detected

_{Skipped verifiers: uber-one, web, diff-template, java-lint, go-lint, go-coverage, go-glint, go-proto-lint, go-thrift-lint, android-lint, java-coverage, ios-lint, android-coverage, ios-test, claude · EngWiki}

Prior runs

Run at fcb7447 on Jun 30 15:50 · 2 files · 1s · 0 issues detected

Issues

Propose extracting per-queue extension profiles, topic-registry builder, and controller registration from the orchestrator example main.go into reusable domain packages, plus data-driven profile hints in QueueConfig. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

JamyDev · 2026-06-30T15:53:11Z

+
+This keeps the controller list in the domain layer (testable, importable) and reduces the wiring main.go to: build dependencies → call `wire.RegisterPrimary` → call `wire.RegisterDLQ`. Adding a new pipeline stage becomes a single-file edit in this package.
+
+### 4. Extend `QueueConfig` with optional profile hints


This may be a bit controversial @behinddwalls :)

…onfig removal Analyze three options for per-queue extension selection in light of potential QueueConfig removal: hints in the entity, a separate profile config file, or keeping profiles in Go code only. Recommends landing the mechanical extractions (steps 1-3) unconditionally and deferring step 4 until the QueueConfig question is resolved. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

JamyDev · 2026-07-13T22:36:22Z

+|---|---|---|---|
+| **A: Profile hints in QueueConfig (step 4 as proposed)** | Add `QueueProfile` fields to the entity; deployers declare scorer/conflict/etc. in `queues.yaml`; the wiring layer maps hint strings → instances. | Single source of truth for queue identity + behavior. YAML-only queue addition for known extension types. | Expands `QueueConfig` from a pure name registry into a config carrier — if QueueConfig is later removed, these fields need a new home. The entity gains fields the gateway doesn't use (profile hints are orchestrator-only). |
+| **B: Separate profile config file** | Leave `QueueConfig` as-is (name-only). Create a separate `queue-profiles.yaml` (or a `profiles:` section in a new file) consumed only by the orchestrator wiring. The orchestrator loads both queue names and profiles; the gateway loads only names. | Clean separation: gateway validates names, orchestrator resolves profiles. `QueueConfig` stays minimal and removable. No entity-level coupling. | Two config files to keep in sync (queue names must match). More moving parts in the wiring layer. |
+| **C: No data-driven profiles — keep profiles in Go code** | Drop step 4 entirely. Steps 1–3 (queueprofile, topicregistry, wire) still land as mechanical extractions. Per-queue profiles stay in Go, just using the promoted `queueprofile.Registry` instead of the current inline types. | Simplest change. No new config surface. Full type safety — a misspelled scorer name is a compile error, not a runtime lookup miss. Consistent with the existing philosophy ("all behavioral and VCS configuration lives in the extension factory implementations"). | Adding a new queue still requires a recompile. Doesn't address the "three-place edit" problem for deployers who don't write custom extensions. |


Think I'm leaning option C here @behinddwalls

behinddwalls · 2026-07-14T16:37:19Z

+    PublishOnly []consumer.TopicConfig
+}
+
+func Build(p BuildParams) (consumer.TopicRegistry, error)


maybe we call it Create? conflates with Build (CI)

behinddwalls · 2026-07-14T18:45:32Z

@@ -0,0 +1,272 @@
+# Modular Queue Wiring


would it possible to run through an example for this...might make things a bit more clear..

I am chatting with Claude to see if we can land on some good sample

behinddwalls · 2026-07-14T19:13:26Z

Design counter-proposal: declare, don't assemble

Full end-to-end walkthrough (every construct defined, every file shown step by step): gist — declare, don't assemble. Summary below.

The test that drove it: could another repo import submitqueue as a library and build an fx / plain-main / test-harness service from it?

The core move: the RFC's extractions (and our first drafts of this feedback) still leave each service implementing assembly — four copies kept aligned by convention and review. The counter-proposal: each service declares itself as data, one shared engine assembles.

piece	owner	form
engine — construction, lifecycle, DLQ pairing, start/stop ordering	`platform/pipeline.Construct` + `platform/lifecycle` — written once, tested once	code
service definition — `Deps` (seams) · `Stages` (queue controllers, one row = controller + topic + DLQ reconciler) · `Controllers` (RPC, unbound to any wire contract)	each domain pkg	data + plain funcs
deployment policy — impl selection (`profiles.go`), creds, topic names, classifiers, transport, signals	each `main` (our `service/` mains become reference hosts)	host code

Highlights, each shown concretely in the gist:

Adding a stage = one table row. Today it takes 3 hand-coordinated edits (topic config, registration, registerDLQControllers); "every consumed stage has a DLQ pair + reconciler" becomes true by construction instead of by review — just in time for the speculation stages.
One lifecycle.Component handle; the library exposes no server, reads no env, traps no signals. An fx host maps 1:1 (each block an fx.Provide, Construct one fx.Invoke, lifecycle two fx.Hook lines).
Per-queue impl variation is the existing Factory contract ("identity in, resolve internally") — profiles stay host-private in profiles.go, so core/queueprofile stays trigger-gated exactly per the in-code TODO.
Controllers are the portable Go surface (proto messages, no server interface — binding to a proto service/transport is ~10 lines of host glue), and the committed api/ protos are the cross-language surface: other languages regenerate stubs from the same sources, one contract, two codegens.
Not the DI framework this RFC rightly rejects: no runtime graph, no reflection, no topo-sort — a typed engine over declarative data; one-offs enter via Options.

The problem statement and the QueueConfig A/B/C analysis survive nearly verbatim — this mostly changes the extraction plan's shape. Happy to pair on the rewrite.

JamyDev changed the title ~~docs(submitqueue): add modular queue wiring RFC~~ Modularize orchestrator wiring: extract queue profiles, topic registry builder, and controller registration Jun 30, 2026

JamyDev commented Jun 30, 2026

View reviewed changes

JamyDev marked this pull request as ready for review July 10, 2026 17:39

JamyDev requested review from a team, behinddwalls and sbalabanov as code owners July 10, 2026 17:39

JamyDev force-pushed the jamy/rfc-modular-queue-wiring branch from 8801dd0 to 3d8d47f Compare July 13, 2026 22:32

JamyDev commented Jul 13, 2026

View reviewed changes

behinddwalls reviewed Jul 14, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Modularize orchestrator wiring: extract queue profiles, topic registry builder, and controller registration#290

Modularize orchestrator wiring: extract queue profiles, topic registry builder, and controller registration#290
JamyDev wants to merge 2 commits into
mainfrom
jamy/rfc-modular-queue-wiring

JamyDev commented Jun 30, 2026 •

edited

Loading

Uh oh!

JamyDev Jun 30, 2026

Uh oh!

JamyDev Jul 13, 2026

Uh oh!

behinddwalls Jul 14, 2026

Uh oh!

behinddwalls Jul 14, 2026

Uh oh!

behinddwalls commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		This keeps the controller list in the domain layer (testable, importable) and reduces the wiring main.go to: build dependencies → call `wire.RegisterPrimary` → call `wire.RegisterDLQ`. Adding a new pipeline stage becomes a single-file edit in this package.

		### 4. Extend `QueueConfig` with optional profile hints

Uh oh!

Conversation

JamyDev commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Revert Plan

API Changes

Monitoring and Alerts

AI Verification

Issues

Uh oh!

JamyDev Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

JamyDev Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

behinddwalls Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

behinddwalls Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

behinddwalls commented Jul 14, 2026

Design counter-proposal: declare, don't assemble

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JamyDev commented Jun 30, 2026 •

edited

Loading