Support incremental builds with probe and enumeration fingerprinting - #162
Open
David Federman (dfederm) wants to merge 9 commits into
Open
Support incremental builds with probe and enumeration fingerprinting#162David Federman (dfederm) wants to merge 9 commits into
David Federman (dfederm) wants to merge 9 commits into
Conversation
…ting File probes and directory enumerations are filtered out at the application layer, so neither contributes to a fingerprint. A cache entry therefore records no dependency on whether a file existed or on what a directory contained, which is why a lookup can hit after that state has changed. Closing that gap needs a fingerprint entry able to express what was observed rather than only which files were read. These are the types that express it, added ahead of any producer or consumer so the schema can be read on its own. ObservationType is a byte enum with pinned values because it is part of the on-disk PathSet payload. QuickBuild is growing the same feature against the same schema, so the values and the precedence order are fixed on both sides. ObservedPathEntry carries the path, the type, and -- for directory enumerations -- the search pattern and the member lists. Comparison is case-insensitive on the path and on member names to match Windows, and ordinal on the pattern, since a build that asked for `*.CS` made a different request than one that asked for `*.cs` whatever the filesystem then did with it. The constructor normalizes the enumeration-only fields to null on other types so entries that mean the same thing compare equal regardless of what a caller threaded through. ObservationTypePrecedence resolves a path observed several ways in one build: a read tells us more than an enumeration, which tells us more than a probe, so the strongest observation wins. ObservedAccess is the sandbox-side counterpart, holding an absolute path before normalization.
PathSet holds a list of normalized paths, which cannot express anything beyond "this file was read". Replacing it with the ObservedPathEntry list is the structural change the feature needs, and doing it before anything new is observed keeps it reviewable on its own. Nothing new is captured here. The sandbox still filters probes and enumerations out, so every entry produced is a FileContentRead. This schema transition intentionally invalidates pre-feature cache entries: PathSet changes from FilesRead to Entries, the feature setting joins the weak fingerprint, and file reads gain a typed identity in the strong fingerprint. Existing entries degrade to cache misses and age out through normal eviction. The rules that only bind once probes and enumerations do arrive come with it, because they are what the entry list is for and splitting them out would leave a data structure with no semantics: * FilterObservations decides what reaches the PathSet. Predicted inputs and IgnoredInputPatterns matches are dropped as before, and file reads are still gated on the input hasher recognizing the path. Probes and enumerations must additionally normalize under the repo or package root: a path under neither is not something the cache can reason about, and that is where most of the volume and all of the machine-specific noise sits. * FoldPathSetEntries canonicalizes several observations of one path by the precedence order, while keeping enumerations of the same directory under different search patterns as separate entries rather than collapsing them. * The strong fingerprint gains a payload per type. A directory enumeration hashes the member list carried on the entry rather than reading the filesystem, which is what makes the value reproducible at lookup from the cached entry alone. EnableProbeAndEnumerationFingerprinting arrives off. It contributes to the weak fingerprint so entries produced with and without it never mix. PathSet also normalizes a null entry list to empty. It is used as a dictionary key during lookup, so a payload lacking the property would be hashed -- and throw -- before any caller could check it. Degrading to a miss is what the null handling downstream already assumed.
MSBuild grew an EnumeratePattern field on FileAccessData, which directory-enumeration observations need: without the pattern a filtered enumeration such as `*.cs` is recorded as if it were unfiltered, so any unrelated file appearing in the directory invalidates the entry and the common case essentially never hits. Its absence therefore has to disable the feature outright rather than degrade it. MSBuildCache compiles against a reference assembly but runs against whatever Microsoft.Build.dll the host supplies, so a direct property access would not compile today and would throw on older hosts if the compile reference were raised. The property is bound once to a delegate instead, after which each read costs about a virtual call -- worth caring about, since this sits on the file-access reporting path. FileAccessData is a struct. Instance members on a value type take their receiver by reference, so the delegate does too; the alternative of calling PropertyInfo.GetValue per access would box the struct every time. The running MSBuild version is captured alongside it for diagnostics only. The version that first carries the field is not knowable while it is unreleased, so probing for the field tests the exact dependency and stays correct if it is ever serviced back to an older branch. Nothing consumes this yet.
The sandbox has always reported probes and enumerations; the repository logged them and threw them away. They are now classified and kept, which is what makes a cache entry able to depend on a file having been absent or a directory having held a particular set of names. Classification is asymmetric on purpose. A probe becomes AbsentPathProbe only for error codes that mean the path definitively did not exist -- not found, path not found, bad net path, invalid name -- and everything else, including success, becomes ExistingProbe. Treating a transient failure such as ERROR_SHARING_VIOLATION or ERROR_ACCESS_DENIED as absence would let a flaky machine fingerprint the same sources differently between builds. QuickBuild classifies the same way. Enumerate and EnumerationProbe are not the same thing despite the names. Enumerate is reported against the directory and carries the search pattern. EnumerationProbe is reported against each matched child -- FindFirstFileEx's first result and every FindNextFile after it -- so it is an existence probe on that child. Recording it as an enumeration would key a directory-enumeration entry on a file path, which lookup-time validation can never re-validate because it requires the path to still be a directory; every project enumerating this way would then miss permanently. The directory's own Enumerate observation still catches member-list changes. Classification runs before the generic `error != 0` skip, since a probe's whole value is often that it failed. Accesses carrying Read or Write are content accesses and keep flowing to the file table untouched. `*` is normalized to null so an unfiltered enumeration folds to one entry however the caller spelled it, and the observation list is only allocated when the feature is enabled, since probes dominate event volume. Self-outputs are not filtered yet and enumerations carry no member lists, so enumeration entries currently over-invalidate. The next commit addresses both. The feature remains off by default.
A project observes its own build in progress. It probes an output before generating it, probes it again afterwards, and enumerates directories it is writing into. Those observations describe intra-build state, not the pre-build state a cache lookup is answering against, and keeping them means the entry can never match: the probe recorded absence, and by the next build the output is there. Two forms of this are handled. Probes and enumerations of any path this project ever wrote are dropped, along with every ancestor directory of such a path. The ancestor walk matters because MSBuild routinely probes an output directory before creating it. "Ever written" is deliberately broader than the output set, which is filtered to existing files and would leak transient temporaries and build-created directories back in. This is project-local, so probes of another project's outputs still count. Enumerations of a directory the project writes into cannot simply be dropped -- the directory usually holds real inputs too. Instead its contents are split into members the project wrote and members it did not. Only the latter go into the fingerprint; the former are recorded separately so lookup can subtract them from what it finds on disk. A project that enumerates a staging directory it also populates then matches whether or not the previous build's outputs are still there, which is the case that makes clean-to-incremental cycles work. Members are captured at this layer rather than at fingerprint time because the self-output set is only known here. The search pattern is compiled through a shared helper. Patterns come from whatever the build passed to the enumeration API, and Windows substitutes wildcards that have no glob equivalent -- DOS_STAR, DOS_QM and DOS_DOT are spelled `<`, `>` and `"` -- which the glob parser rejects, as it does braces and a closing bracket. An unparseable pattern falls back to matching every member. That can only over-invalidate, since the recorded member list becomes a superset; dropping the observation instead would lose the dependency and risk exactly the incorrect hit this feature exists to prevent. The helper is shared so that the capture and validation sides always reach the same conclusion about a pattern.
Everything so far records observations. Nothing yet checks them, and the strong fingerprint cannot: a probe entry contributes only its identity and an enumeration entry hashes the member list stored on itself, neither of which touches the disk. Recomputing the fingerprint from a cached PathSet therefore always reproduces the fingerprint stored in the selector no matter what the filesystem now holds. This check is what enforces those entries, and cache lookup gains nothing by reaching it any earlier than it must. That is the opposite of how it reads. For file content reads the same check is purely an optimization, because their bytes are hashed into the fingerprint and a change is caught by the comparison regardless. Someone could reasonably conclude the whole thing is redundant and delete it, expecting to pay in cache hits; they would instead start silently producing incorrect ones. Both call sites say so. An existing probe requires the path to still exist and an absent probe requires it to still be missing -- the latter is the clean-to-dirty case that motivated the feature, where a file the build only checked for has since appeared. Directory enumerations re-enumerate, subtract the member names the project wrote during the populate build, and compare what remains against the recorded members. Subtracting the self-outputs is what lets the comparison hold whether or not the previous build's outputs are still on disk. A recorded member list of null is not an empty directory: it is how the capture side records a directory it could not enumerate at all, and the fingerprint already encodes the two differently. So it is validated as absence rather than by re-enumeration. Re-enumerating would report a miss precisely because nothing had changed -- the directory is still missing -- making any project that enumerates a path that does not exist permanently uncacheable, which is an ordinary consequence of a wildcard over an absent directory. In the other direction the member comparison treats null and empty alike, so falling through would let a directory that did not exist and now exists empty validate as unchanged. An enumeration that fails mid-way returns no observation at all and forces a miss, so an IO error can never be mistaken for an empty directory.
Everything the feature needs is in place, so the setting defaults on and the documented limitation goes away: MSBuildCache no longer assumes the repo is clean before building. Entries produced before this sit under a different weak fingerprint and age out through normal eviction, so nothing needs migrating. Turning it on unconditionally would be wrong on a host that does not report enumeration patterns. There the capture side records every filtered enumeration as if it were unfiltered, which costs hits broadly rather than producing incorrect ones -- but the weak fingerprint would still claim the feature was on. The setting is therefore forced off on such a host, and forced rather than merely ignored, so the property stays the single source of truth for behavior, for logging and for the weak fingerprint instead of reporting a value that disagrees with what the build did. An explicit opt-out on a capable host is still honored. The clamp names the running MSBuild version when it logs, since being told a feature is off is not actionable without knowing what you are running or that upgrading would fix it. The new MSBuild property is added to GlobalPropertiesToIgnore so toggling it with -p: does not also perturb the global-property hash.
The unit tests exercise the pieces in isolation against synthetic PathSets. What they cannot show is that a real build, through MSBuild and the sandbox, hits and misses where it should. Each scenario builds a small generated repo two or three times, perturbs exactly one thing between builds, and asserts the hit and miss counts. The set is chosen so a failure localizes: * A file added under a target-time dynamic glob misses, and an unchanged source tree hits -- the enumeration channel in both directions. * A marker file appearing where the build only probed for it misses, and a marker whose state does not change hits -- the probe channel, and the clean-to-dirty case the feature exists for. * The same marker outside the repo and package roots hits, as does one matched by IgnoredInputPatterns, which is what proves the scope and pattern filters are load-bearing rather than incidental. * A project that enumerates a directory it also writes into hits across a clean, incremental and clean cycle. * Editing a source file misses and rebuilding hits, deliberately routed through neither new channel, so a failure there points at cache fundamentals rather than this feature. * Three consecutive no-change builds all hit. The third replays over the second build's replayed outputs, which catches an observation that only re-validates against outputs a real build produced -- something a two-build test passes. The suite skips with a warning when the MSBuild under test does not report enumeration patterns, since the feature disables itself there and the scenarios would assert misses that cannot happen. Failing instead would block every PR until a suitable MSBuild ships. The warning is raised as a pipeline log issue so a skip is visible in the build summary rather than buried in a log. A skip is otherwise indistinguishable from a pass, so the pipeline job that runs against a bootstrap of dotnet/msbuild main -- the only one carrying the field today -- requires the capability and fails without it. If the bootstrap layout or the API changes, that job fails rather than quietly running nothing. The production Visual Studio jobs keep skipping, and will start running the scenarios on their own once the field ships. Builds now tolerate post-completion accesses under the Windows directory: Code Integrity touching a finished worker otherwise throws from a BuildXL IO-completion thread and terminates the build. Those paths are outside the repo and package roots and so are already excluded from observations. test.ps1 becomes smoke.ps1 now that there is more than one suite.
Andy Gerlicher (AndyGerlicher)
approved these changes
Aug 6, 2026
Render the Azure Pipelines boolean template parameter as a PowerShell boolean literal so scenarios.ps1 can bind it to its typed parameter. Copilot-Session: 50484828-3f84-40b9-9775-c576cc297508
David Federman (dfederm)
enabled auto-merge (squash)
August 6, 2026 03:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Improves incremental/local build caching by fingerprinting file existence probes and directory enumerations previously filtered from Detours sandbox observations. Cache lookup now validates probe state and directory membership, allowing correct hits for unchanged local builds and misses when relevant filesystem state changes.
Compatibility and safety
net472andnet9.0.IgnoredInputPatternsto direct observations and enumerated directory members.The README now documents the setting and removes the clean-repository-only limitation.
Validation
Common.Tests(net9.0)Common.Tests(net472)dotnet build MSBuildCache.sln --no-restore --configuration DebugEnd-to-end coverage includes glob membership changes, present/absent probes, ignored and out-of-scope observations, self-output directories, source edits, incremental hits, and repeated warm-cache hits. Older MSBuild versions skip capability-dependent scenarios; the MSBuild-main bootstrap job requires them.