v4 major upgrade: scoping audit umbrella

[thomhurst]

Tracking issue for the v4 scoping audit: multi-agent review of engine internals, core API surface, source generators/analyzers, options generator, docs, and CI job output across ~25 workflow runs. 100 issues raised; breaking API-reshaping issues carry the breaking label and the v4.0.0 milestone.

Related pre-existing open issues: #3228, #3226, #2996, #2623.

Priorities

Every issue carries a priority label — filter with label:P0 etc.:

• P0 — critical: silent corruption, hangs, or regressions landing unnoticed. Work first.
• P1 — high: real user impact, or foundation the v4 branch needs early. Work second.
• P2 — medium: improvements and polish. After P1.
• P3 — low: hygiene. Whenever.

Recommended working order

Phase 0 — stop the bleeding (P0, releasable on 3.x):
#3337 (main has had no CI since April — fix first so everything else is guarded) → #3244 (silently skipped modules) → #3253 (AlwaysRun cleanup can no-op) → #3240 (TCS/lock recursion) → #3251 (startup hang on diamond graphs) → #3267 (orphaned process trees) → #3279 (distributed value-type crash; breaking fix, may need a 3.x mitigation + full fix in v4).

Phase 1 — 3.x hardening (P1, non-breaking):

• Engine bugs: Engine: unobserved faulted tasks from ModuleState.CompletionSource and SubModule.CallbackTask #3241 Scheduler falsely reports deadlock when the only in-flight module is deferred back to Pending #3242 ModuleScheduler.Dispose races in-flight workers; ObjectDisposedException masks real failures #3243 Only the first concurrent module failure is surfaced; sibling exceptions are dropped #3245 ModuleLoggerProvider dispose-order can drop a module's entire buffered log output #3247 Retry back-off is not cancellable: Polly ExecuteAsync called without the cancellation token #3249
• Perf/robustness: Startup: estimated-time lookups are rate-limited to 100 modules/second #3261 (one-line win) SecretObfuscator rebuilds and re-sorts the full secret set on every log line #3254 CoordinatedTextWriter: per-character global locking and O(n^2) buffer scans on the redirected Console.Out #3255 Scheduler loop: make event-driven; O(n^2) allocations per 100ms cycle under the exclusive write lock #3256 ProgressSession: per-module ticker tasks, uncancellable delays, fire-and-forget render loop #3258 Cache module attribute reflection across the engine (repeated per event, per call, with instance-identity bugs) #3260 Run conditions can be evaluated twice per run and are re-reflected per module #3264 Stream command output live to the module logger and cap in-memory buffering #3266 Deferred progress-mode output is unbounded in RAM and lost entirely on crash #3269
• Tooling: Analyzers: AsyncModule and AwaitThis dead since ModuleBase removal; StatefulModule has a dead branch #3309+Consumer analyzer test suite never runs in CI; test ref assemblies mislabeled #3310 (dead analyzers + the CI gap that hid them) MissingDependsOn analyzer never fires on context.GetModule<T>() and ignores generated accessors #3311 Analyzers and StyleCop are disabled in CI - add an analyzer lane #3316 SourceGenerator: incremental caching broken (Location and ImmutableArray in model records) #3303 Packaging: generator/analyzer DLLs packed from hardcoded bin paths without build ordering #3308 OptionsGenerator: strip control characters in XML doc comments (raw NUL bytes in Pulumi output) #3317 Delete legacy Generated/ sub-trees duplicating options and service types #3319 OptionsGenerator: emit the new keyword when a generated property shadows a base member (CS0108 recurrence) #3327
• CI: CI: move the HangDump package into test/Directory.Build.props #3332 CI: Can_Send_Email authenticates against a real SMTP provider and is flaky #3334 CI: guard the ubuntu pipeline job against runner starvation (52-minute lost-runner run) #3335 CI: add a fast-fail lane before the 20-minute pipeline job; investigate macOS duration variance #3338
• Pre-v4 docs prep: Docs site: add search and cut a versioned 3.x doc set before v4 #3330 (cut 3.x doc version before v4 lands), Docs: curated examples for the top tool packages (36 of 56 pages end in filler) #3323

Phase 2 — v4 branch foundation (breaking, in dependency order):
#3272 (remove obsolete API — clears the ground) → #3273 (context hierarchy) → #3274 (registry → source-gen) → #3290 (single builder entry point) → #3292 + #3279 (result model) → #3283 (hooks) → then the remaining milestone items in any order; #3320 (generated renames) last so tool packages regenerate once.

Phase 3 — P2 improvements, opportunistically alongside phases 1-2. Phase 4 — P3 whenever.

Engine correctness bugs

• Engine: TaskCompletionSources lack RunContinuationsAsynchronously and are completed inside the scheduler write lock #3240 TCSes lack RunContinuationsAsynchronously, completed inside scheduler write lock
• Engine: unobserved faulted tasks from ModuleState.CompletionSource and SubModule.CallbackTask #3241 Unobserved faulted tasks (ModuleState.CompletionSource, SubModule.CallbackTask)
• Scheduler falsely reports deadlock when the only in-flight module is deferred back to Pending #3242 False scheduler-deadlock report when only in-flight module is deferred
• ModuleScheduler.Dispose races in-flight workers; ObjectDisposedException masks real failures #3243 ModuleScheduler.Dispose races in-flight workers
• WaitForAllModules mode: exceptions escaping the worker loop are swallowed at Debug level, silently skipping remaining modules #3244 WaitForAllModules swallows worker-loop exceptions at Debug level
• Only the first concurrent module failure is surfaced; sibling exceptions are dropped #3245 Only first concurrent module failure surfaced
• RequirementChecker ignores IPipelineRequirement.Order (GroupBy without OrderBy) #3246 RequirementChecker ignores Order
• ModuleLoggerProvider dispose-order can drop a module's entire buffered log output #3247 ModuleLoggerProvider dispose-order drops buffered logs
• Cancellation plumbing leaks: undisposed linked CTS per module, discarded CancellationTokenRegistration, never-unsubscribed global handlers #3248 Cancellation plumbing leaks (linked CTS, registrations, global handlers)
• Retry back-off is not cancellable: Polly ExecuteAsync called without the cancellation token #3249 Retry back-off not cancellable
• Cycle detection uses unbounded recursion; long dependency chains overflow the stack #3250 Cycle detection stack overflow on long chains
• AllDescendantDependencies has no visited set: exponential blow-up on diamond dependency graphs #3251 Exponential AllDescendantDependencies on diamond graphs
• ModuleState.Dependencies dictionary is mutated under lock but enumerated without it #3252 ModuleState.Dependencies unsynchronized enumeration
• AlwaysRun cleanup modules can silently no-op during failure teardown and run strictly sequentially #3253 AlwaysRun cleanup modules can silently no-op during teardown

Engine performance & robustness

• SecretObfuscator rebuilds and re-sorts the full secret set on every log line #3254 SecretObfuscator rebuilds secret set per log line
• CoordinatedTextWriter: per-character global locking and O(n^2) buffer scans on the redirected Console.Out #3255 CoordinatedTextWriter per-char locking, O(n²) scans
• Scheduler loop: make event-driven; O(n^2) allocations per 100ms cycle under the exclusive write lock #3256 Scheduler loop: event-driven + allocation fixes
• ModuleScheduler.AddModule rebuilds the entire dependency state for every dynamically added module #3257 AddModule rebuilds entire dependency state per dynamic module
• ProgressSession: per-module ticker tasks, uncancellable delays, fire-and-forget render loop #3258 ProgressSession ticker/pause/loop lifecycle
• ModuleLogger allocates per log call and builds a new AnsiConsole per renderable #3259 ModuleLogger per-call allocations, per-renderable AnsiConsole
• Cache module attribute reflection across the engine (repeated per event, per call, with instance-identity bugs) #3260 Cache module attribute reflection (incl. instance-identity bug)
• Startup: estimated-time lookups are rate-limited to 100 modules/second #3261 Startup: estimated-time lookups rate-limited to 100/s
• EstimatedTimes cache: per-module full-directory enumeration and unbounded growth #3262 EstimatedTimes cache: per-module directory enumeration, unbounded growth
• Startup: whole-AppDomain type scans and eager assembly loading (UnusedModuleDetector, DependencyChainProvider O(n^2)) #3263 Startup: whole-AppDomain type scans, eager assembly loading
• Run conditions can be evaluated twice per run and are re-reflected per module #3264 Run conditions evaluated twice per run
• PipelineBuilder.Environment builds and disposes a full generic host on every read #3265 PipelineBuilder.Environment builds a host per read
• Stream command output live to the module logger and cap in-memory buffering #3266 Stream command output live + cap buffering
• Kill the whole child process tree on cancellation (job objects / process groups) #3267 Kill whole child process tree on cancellation
• Disposer.RegisterOnShutdown roots one ProcessExit handler per module logger #3268 Disposer roots one ProcessExit handler per module logger
• Deferred progress-mode output is unbounded in RAM and lost entirely on crash #3269 Deferred progress-mode output unbounded in RAM, lost on crash
• ModuleOutputBuffer spins on a global lock with 50ms Monitor.TryEnter and drops output on timeout #3270 ModuleOutputBuffer lock spin drops output on timeout
• BuildSystemDetector: cache detection, explicit precedence, and full BuildSystem coverage in IsRunningIn #3271 BuildSystemDetector caching, precedence, full enum coverage

v4 API reshaping (breaking, milestone v4.0.0)

• v4: remove obsolete APIs and back-compat aliases #3272 Remove obsolete APIs and back-compat aliases
• v4: collapse the duplicated context interface hierarchy (19 empty markers + IPipelineHookContext) #3273 Collapse duplicated context interface hierarchy
• v4: replace the ModuleInitializer/static-registry integration wiring with compile-time source generation #3274 Replace ModuleInitializer/static-registry wiring with source generation
• v4: tighten the public surface — internalize engine plumbing #3275 Tighten public surface — internalize engine plumbing
• Matrix support: wire up MatrixModuleExpander or mark the public API experimental #3276 Matrix: wire up expander or mark experimental
• TryGetService<T> throws instead of returning null #3277 TryGetService throws instead of returning null
• v4: CommandResult — required members instead of CS8618 suppression #3278 CommandResult: required members, no CS8618 suppression
• v4: unify ModuleResult<T?> vs ModuleResult<T> — InvalidCastException for value-type results in distributed mode #3279 Unify ModuleResult<T?>/ModuleResult<T> (distributed InvalidCastException)
• v4: Git package — lazy GitInformation, honest nullability, Commits on the interface #3280 Git package: lazy GitInformation, honest nullability
• Module.Configuration race and Tags allocating per access #3281 Module.Configuration race, Tags allocation
• GetModule<T> is an O(n) DI scan with SingleOrDefault #3282 GetModule O(n) DI scan
• v4: consolidate the four overlapping module hook mechanisms #3283 Consolidate four overlapping hook mechanisms
• v4: remove no-op DependsOnLazy and eagerly-evaluated DependsOnIf(Func<bool>) #3284 Remove no-op DependsOnLazy and eager DependsOnIf(Func)
• v4: single dependency-declaration surface #3285 Single dependency-declaration surface
• v4: composable skip/before/after configuration (8 last-wins WithSkipWhen overloads) #3286 Composable skip/before/after configuration
• v4: consistent execute-method naming across Module, Module.NonGeneric and SyncModule #3287 Consistent execute-method naming
• v4: retry API — WithRetry(count, baseDelay, shouldRetry); stop leaking Polly #3288 Retry API redesign; stop leaking Polly
• v4: SkipDecision — drop surprising implicit conversions, init-only, document #3289 SkipDecision: drop surprising implicit conversions
• v4: one registration/builder entry point; delete sync-over-async Build() #3290 One registration/builder entry point; delete sync-over-async Build()
• v4: PipelineOptions — init-only record, options pattern, no Spectre global mutation in setters #3291 PipelineOptions: init-only, options pattern, no Spectre globals
• v4: single result model — drop ModuleResultType and Is* triplication; PipelineSummary cleanup #3292 Single result model; PipelineSummary cleanup
• v4: CommandException should carry the CommandResult, not inline stdout/stderr into Message #3293 CommandException carries CommandResult
• v4: generate interfaces for tool sub-domain services (concrete unmockable types on IDotNet/IDocker facades) #3294 Generate interfaces for tool sub-domain services
• v4: one command-builder abstraction (delete non-generic ICommandBuilder; tool builders implement the CRTP interface) #3295 One command-builder abstraction (CRTP)
• v4: discoverable tool entry points — context.Tools.* instead of extension methods requiring usings #3296 Discoverable tool entry points (context.Tools.*)
• v4: Async suffix on generated command methods; split the 78-member IGitCommands #3297 Async suffix on generated methods; split IGitCommands
• v4: readonly collections in public API surface #3298 Readonly collections in public API
• v4: remove global-namespace extension pollution (T.AsTask(), bool.AsSkipDecisionIfTrue) #3299 Remove global-namespace extension pollution
• v4: IEnvironmentDomainContext.WorkingDirectory setter mutates process-wide CWD under parallel modules #3300 WorkingDirectory setter mutates process-wide CWD
• v4: IHttp should not hand out raw HttpClients (bypasses secret masking, resilience and logging) #3301 IHttp: no raw HttpClients
• Docs/XML sweep: v2 APIs still taught (PipelineHostBuilder in README and docs, non-compiling examples, broken crefs) #3302 Docs/XML sweep: v2 APIs still taught

Source generator & analyzers

• SourceGenerator: incremental caching broken (Location and ImmutableArray in model records) #3303 Incremental caching broken (Location, ImmutableArray equality)
• SourceGenerator: predicates run semantic analysis on every type in every project #3304 Predicates run semantic analysis on every type
• SourceGenerator: duplicate module class names silently collapse into one extension method #3305 Duplicate module names silently collapse
• SourceGenerator: add MPG#### diagnostics instead of silent skips #3306 MPG#### diagnostics instead of silent skips
• Add source-generator snapshot and incremental-cache tests #3307 Source-generator snapshot + incremental-cache tests
• Packaging: generator/analyzer DLLs packed from hardcoded bin paths without build ordering #3308 Generator/analyzer DLLs packed from hardcoded bin paths
• Analyzers: AsyncModule and AwaitThis dead since ModuleBase removal; StatefulModule has a dead branch #3309 AsyncModule/AwaitThis dead since ModuleBase removal
• Consumer analyzer test suite never runs in CI; test ref assemblies mislabeled #3310 Consumer analyzer tests never run in CI; ref assemblies mislabeled
• MissingDependsOn analyzer never fires on context.GetModule<T>() and ignores generated accessors #3311 MissingDependsOn never fires on real usage
• Analyzers: renumber to a single MP#### ID family, add HelpLinkUri, generate per-rule docs #3312 MP#### IDs, HelpLinkUri, generated per-rule docs
• MPDEP002 claims circular-dependency detection but only finds direct 2-cycles #3313 MPDEP002 only finds direct 2-cycles
• New analyzer batch: unregistered module, async void, blocking calls, unflowed CancellationToken, duplicate DependsOn #3314 New analyzer batch (unregistered module, async void, blocking, CT flow)
• Add code fixes for the six analyzer rules without one #3315 Code fixes for the six rules without one
• Analyzers and StyleCop are disabled in CI - add an analyzer lane #3316 Analyzers/StyleCop disabled in CI — add analyzer lane

Options generator & tool packages

• OptionsGenerator: strip control characters in XML doc comments (raw NUL bytes in Pulumi output) #3317 Strip control characters in XML doc comments (NUL bytes)
• OptionsGenerator: CI check for stale generator-version stamps in committed output #3318 CI check for stale generator-version stamps
• Delete legacy Generated/ sub-trees duplicating options and service types #3319 Delete legacy Generated/ sub-trees (ambiguous duplicate types)
• OptionsGenerator: compound-word PascalCase produces bad public names (DotNetBuildserver, AzAppconfig, Nologo) #3320 Compound-word PascalCase (DotNetBuildserver, Nologo…) — breaking
• OptionsGenerator: dedupe aliased command trees (docker builder vs buildx) #3321 Dedupe aliased command trees (docker builder/buildx)
• OptionsGenerator: commit command-coverage manifests for every tool; missing manifest should be a hard error #3322 Command-coverage manifests for every tool
• generate-cli-options workflow: git and brew scrapers are registered but never run #3324 git/brew scrapers registered but never run
• generate-cli-options workflow: derive the tool map from the generator instead of 1,000-line YAML duplication #3325 Derive tool map from generator, not 1,000-line YAML
• OptionsGenerator: write a README for the generation subsystem #3326 OptionsGenerator README
• OptionsGenerator: emit the new keyword when a generated property shadows a base member (CS0108 recurrence) #3327 Emit new on shadowed generated properties (CS0108)
• Node package: add an npm scraper; npm options are hand-written while pnpm is generated #3328 npm scraper; Node package consistency
• Shared smoke-test harness for generated tool surfaces (most packages have no tests) #3329 Shared smoke-test harness for generated tool surfaces

Docs & first-run experience

• Docs: curated examples for the top tool packages (36 of 56 pages end in filler) #3323 Curated examples for top tool packages (36 filler pages)
• Docs site: add search and cut a versioned 3.x doc set before v4 #3330 Docs search + versioned 3.x doc set before v4
• Ship dotnet new templates and a realistic example pipeline #3331 dotnet new templates + realistic example pipeline

CI

• CI: move the HangDump package into test/Directory.Build.props #3332 HangDump package → test/Directory.Build.props
• CI: FormatMarkdownModule should list offending files or auto-fix instead of failing with a run-it-locally message #3333 FormatMarkdownModule: list files / auto-fix
• CI: Can_Send_Email authenticates against a real SMTP provider and is flaky #3334 Can_Send_Email flaky (real SMTP auth)
• CI: guard the ubuntu pipeline job against runner starvation (52-minute lost-runner run) #3335 Guard ubuntu pipeline job against runner starvation
• CI: 'Can't add secret mask for empty string' warning on every run #3336 Empty add-mask warning every run
• CI: no build or tests run on pushes to main #3337 No build/tests on pushes to main
• CI: add a fast-fail lane before the 20-minute pipeline job; investigate macOS duration variance #3338 Fast-fail lane; macOS duration variance
• CI: CodeQL job fails on transient network timeouts #3339 CodeQL transient network retry