fix(scan): surface run warnings in structured scan results - #220
Open
rohanpoudel2 wants to merge 2 commits into
Open
fix(scan): surface run warnings in structured scan results#220rohanpoudel2 wants to merge 2 commits into
rohanpoudel2 wants to merge 2 commits into
Conversation
A scan whose target changes mid-run completes against the original snapshot and reports "Repository HEAD changed while the scan was running; results were saved for the original revision." That warning only ever reached the `onWarning` observer, and the CLI observer only writes it to stderr. `ScanResult.toJSON()` had no `warnings` key, so `codex-security scan --json` exited 0 with a document indistinguishable from a scan of the commit CI actually checked out. Nothing automated could tell that the results were for a stale tree. Collect the warnings a run reports and carry them on `ScanResult`, so they appear in `toJSON()` and therefore in `--json` output. Warnings are recorded where they are reported, including the ones cleanup produces after the result is collected, and are redacted the same way the CLI already redacts them before printing, because the result is written to CI logs and artifacts while the observer stream is not. The key is always present and empty when a run reported nothing, matching `toJSON()`, which emits every key, and `scans show --json`, which always emits a `warnings` array. Observer behavior, exit codes, and every existing key are unchanged. `scans list --json` and `scans show --json` already carry the stored warnings and need no change. Refs openai#195
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #195
Problem
A scan whose target changes mid-run completes against the original snapshot and produces a warning:
That warning reached the
onWarningobserver and nothing else. The CLI's observer writes it to stderr, whilescan --jsonprintsresult.toJSON()to stdout — andScanResult.toJSON()had nowarningskey at all. A CI job therefore received exit 0 and a JSON document indistinguishable from a clean scan of the commit it actually checked out. The one signal that the results were for a stale tree was unavailable to any machine consumer.The same gap affected the other two warning producers, not just target drift:
onFinalizewarnCleanupFailedChange
Warnings now travel with the result.
ScanResultgains awarnings: readonly string[]field, always emitted bytoJSON()and therefore present in--jsonoutput.toJSON()already emits every key unconditionally (sarifPath: null,cost: null), and the workbench's ownscans show --jsonalways emits awarningsarray, so an always-present array is consistent here. It also lets a CI job write.warnings.length > 0without having to distinguish "no warnings" from "an older SDK".Two details worth calling out for review:
run(), and the warnings are attached after#runreturns.ScanResultis constructed incollectResult()beforecomplete-scanruns, so the drift warning does not exist yet at construction time, and cleanup warnings arrive later still, from#run'sfinally.run()owns the array and passes awarnreporter into#run; the three emission sites call it instead ofnotifyObserverdirectly. Recording is synchronous at the emission site rather than inside the observer callback, becausenotifyObserverdispatches on a microtask and depending on that ordering would be fragile.ScanResult.warnstoresredactedErrorMessage(warning)while forwarding the raw string toonWarning, so existing observer behavior is byte-for-byte unchanged. This mirrors thefail-scanpath, which redacts before storing because the stored text is read back byscans show.ScanResult.withWarnings()returnsthiswhen there is nothing to add, so the common path allocates nothing.Deliberately not included
--fail-on-warningflag or by promoting drift to exit 1 — is a breaking change for existing pipelines and is yours to decide. This PR only makes the condition detectable; A scan whose target changed mid-run exits 0 and reports nothing in--json, so CI cannot detect that the results are for a stale tree #195 asks for both, and I would rather not decide the second half unilaterally._bundled_plugin/change.coverage.completenessstayscompleteon a drifted run. Degrading it would ripple into the CLI's existingcompleteness !== "complete"→ exit 2 path, which is the same behavior change wearing a different hat.exportchange. Warnings live in the workbench'scompletion_warnings_jsoncolumn, not in scan-directory artifacts, so surfacing them there would require changing the bundled Python exporter and its output schemas.scans show --jsonandscans list --jsonalready includewarnings; no change was needed there.Verification
The load-bearing test is in
cli.test.ts: it asserts the exact drift string lands in parsed--jsonstdout, not merely stderr, and that the exit code stays 0. Before the change it failed withReceived: undefined;result.test.tsandapi.test.tsgained matching assertions.Full suite: 719 pass / 5 skip / 0 fail (717 baseline plus 2 new tests).
pnpm run typesandpnpm run formatare clean.