Honor SkillSpector analysis completeness

[Tomauskasz]

Summary

• Select the SkillSpector completeness contract from a validated metadata.skillspector_version.
• Support the captured SkillSpector 2.9.6 status-less report shape and require the full 2.10+ completeness schema.
• Reject unsuccessful, malformed, or contradictory reports before processing findings.
• Preserve trustworthy findings from coherent partial reports while keeping the SkillEvaluator result incomplete.
• Preserve distinct modern findings when match_fingerprint is null and legacy findings when display text is null.
• Require complete, unique component inventories and reconcile every complete-report issue with exact source-scoped component evidence, except artifact-integrity findings for skipped bytecode.
• Require non-empty versioned finding paths and reconcile known universal analyzer work with the component count on complete scans.
• Preserve only independently reconstructable scoring evidence across compacted or hidden occurrences, including explicit rule floors.
• Reject occurrence-expanded rows that disagree on the severity, confidence, or finding ID copied from one compacted identity.
• Keep shipped bytecode visible to SkillSpector while generated output directories are excluded from staged scans.

Root cause

SkillEvaluator previously treated completeness-field presence as the schema boundary. Real SkillSpector 2.9.6 --no-llm output already contains analysis_completeness, but it does not contain the 2.10 status field. Conversely, a report labeled 2.10+ could omit the new fields and take the legacy path.

The previous validation also trusted summary flags without reconciling component counts, coverage, analyzer work, and limitations. Valid findings from partial scans were discarded because validation returned before finding conversion.

Contract

The validator now:

• requires a semantic metadata.skillspector_version and exact metadata.llm_requested agreement with the scan stage;
• applies the captured status-less contract to 2.9.6 and the status-bearing contract to 2.10+;
• requires both top-level and nested execution-success markers;
• validates non-negative integer component counters, excluding booleans;
• requires non-negative pre/post-filter finding counts, rejects before < after, and reconciles those counts with the serialized active and suppressed findings under the version-specific report behavior;
• requires scanned_components == fully_inspected_files;
• requires total_components == fully_inspected_files + partially_inspected_files + entirely_uninspected_files;
• requires versioned component inventories with unique source/path identities, non-empty paths, boolean executable evidence, exact count reconciliation, and symmetric agreement with metadata.has_executable_scripts;
• requires every versioned finding to identify a non-empty producer path;
• requires each non-SC8 issue in a complete report to resolve to exact source/path component evidence;
• recomputes coverage_percent from the component counts;
• requires the captured 2.9.6 analyzer roster, the stable 2.10+ roster, and all semantic analyzers when LLM availability is confirmed, while permitting future 2.10+ analyzers;
• validates version-specific analyzer work partitions, state/counter agreement, and limitation reconciliation;
• requires known universally applicable analyzers to complete exactly one work item per component on complete nonempty scans, while preserving partial-scan and future-analyzer compatibility;
• accepts canonical partial reference-coverage and transitive-truncation limitations, including repeated analyzer IDs from root and child summaries;
• requires the risk recommendation to match severity and completeness, including LOW/CAUTION for partial scans;
• computes a conservative risk-score floor without trusting occurrence-level confidence, executable status, or ordering discarded by 2.10 report compaction;
• requires non-empty 2.10 finding_id values, scopes executable-file matching and report deduplication by producer provenance, uses serialized match fingerprints before finding IDs, and keeps legacy null-text rows distinct;
• derives executable multipliers from component evidence, allocates the global count of compacted or hidden raw findings once across the most score-reducing rule slots, preserves visible base contributions and explicit rule floors, and retains a producer-score lower bound after filtering only when the raw finding count is fully serialized;
• preserves SkillSpector's blocking SC8 shipped-bytecode finding and rule-specific score floor, and keeps shipped bytecode in generated-output-free staged scans;
• processes valid partial findings, marks the result incomplete, and suppresses clean or successful summaries;
• rejects failed or structurally invalid reports before finding conversion.

Reviewer follow-up

This revision addresses all three P1 findings:

• version-gated completeness selection now has a captured SkillSpector 2.9.6 --no-llm fixture;
• coherent partial reports retain valid HIGH findings while remaining incomplete;
• required total, scanned, and fully-inspected counters are type-checked and reconciled with the producer equations.

Self-review also caught and corrected the partial LOW/CAUTION invariant, huge-integer coverage ordering, missing execution/version/recommendation evidence, incomplete analyzer rosters, analyzer status/work contradictions, duplicate transitive analyzer IDs, released 2.10/2.11 transitive-truncation output, unvalidated finding counts and identities, a false rejection caused by rebuilding a pre-compaction score from compacted findings, source-provenance loss in score reconstruction, null-fingerprint findings collapsed by display prose, legacy null-text findings collapsed into one row, executable component evidence lost during post-filter scoring, lossy replacement of the producer score, fractional score truncation at the policy threshold, and shipped bytecode removed before the artifact-integrity analyzer ran. Exact SkillSpector 2.10.0 producer probes confirm the report-deduplication, source-scoped scoring, null-fingerprint, and SC8 score-floor behavior covered by the public-seam regressions.

Independent immutable-head review then reproduced three further defects: a missing component inventory could understate executable risk, report compaction could attach executable metadata from the wrong occurrence, and hidden pre-compaction multiplicity could make the reported-score bound unsafe. Those cases now fail closed or use independently reconstructed evidence. Final self-review also removed inconsistent duplicate source-scope logic and rejected duplicate component identities that could overwrite executable evidence.

Final immutable-head review reproduced three more P1 defects: empty or absent issue paths bypassed executable-score reconciliation; complete reports could claim zero work from every required analyzer; and representative confidence copied during compaction could reject a genuine producer score. Versioned paths now fail closed, complete universal-analyzer work is tied to the component inventory, and compacted identities use a conservative global unknown-finding budget.

A subsequent immutable-head review found that the first compaction fix erased all visible numeric risk whenever one raw finding was hidden or one identity expanded into multiple rows. The final score floor now counts one proven representative per compact identity, assigns the finite global unknown-finding budget to the largest possible diminishing-weight reductions, and preserves explicit producer rule floors. It also rejects reports that claim fewer raw findings than their distinct serialized compact identities.

The last immutable-head review found that rows under one modern compact identity could disagree on the producer-copied severity, confidence, or finding ID. That allowed a forged 15/LOW/SAFE report to pass even though the corresponding raw findings score 69/HIGH/DO_NOT_INSTALL. The validator now rejects inconsistent representative fields before score reconciliation while still allowing occurrence-specific paths and source details to differ.

Verification

• Full suite after rebasing onto current main: 5955 passed, 17 skipped, 4 deselected

• Focused validator suite: 353 passed

• make lint: passed

• make build: source distribution and wheel built successfully

• Captured 2.9.6 fixture: valid JSON and accepted by the validator

• Installed SkillSpector 2.10.0: clean and 14-advisory reference-skill probes passed against clean-checkout content; report-deduplicated and source-scoped score probes match the validator contract

• Mutation check: the SC8 score-floor regression failed when the floor was removed and passed after restoration

• Red/green checks: fractional score-floor and staged-bytecode regressions failed before their fixes and passed afterward

• Red/green check: the conflicting compacted-identity report passed before the fix and now fails closed; a conflicting-finding-ID variant also fails closed

• Local DCO reproduction: all five PR commits have parsed Signed-off-by trailers

• git diff --check: passed

• Independent pre-publication review: analyzer-roster, component-inventory, path, universal-work, compacted-score, hidden-multiplicity, and visible-score-floor findings were reproduced and corrected

• Final exact-head producer-contract and repository-standards reviews: no P0-P2 findings

• I am familiar with the Contributing Guidelines

• Added or updated focused tests

• Updated documentation for user-visible changes

• Ran make lint

• Ran the full test suite

• Ran make build

• Did not add credentials, private datasets, or proprietary benchmark content

Release Impact

• No user-visible release note needed
• Updated CHANGELOG.md

Compatibility

SkillSpector 2.9.6 reports use the strict captured compatibility path. Reports from 2.10 and newer must provide the complete status-bearing schema. Older versioned reports without analysis_completeness retain the legacy path; unsupported pre-2.10 completeness objects fail closed.

AI assistance disclosure

Codex inspected the SkillEvaluator and SkillSpector producer contracts, reproduced the reviewer cases, drafted the implementation and tests, captured the 2.9.6 fixture, and ran the verification commands. I reviewed the compatibility boundary, partial-finding behavior, fail-closed invariants, diff, and final pull-request text.

[Tomauskasz]

End-to-end follow-up:

• Installed this fix with SkillEvaluator PR fix(tier3): decode native Codex exec calls #111, the pinned Harbor provenance backport, and SkillSpector PR #451.
• Direct SkillSpector scan: LOW/SAFE, 100% coverage, complete analysis, zero ledger exceptions.
• Installed Tier 1 security validation passed and exited 0.
• One four-case paired Codex/Terra full run completed with 8 expected and 8 scored attempts, no execution errors, and a complete report.
• All three tiers passed; with-skill score was 0.9471 versus 0.6613 baseline.

The full source suite for this PR also passed: 5160 passed, 17 skipped, 4 deselected. Lint and build passed.

[rng1995]

The new contract checks still accept contradictory incomplete evidence.

[rng1995]

These three flags do not cover the full completeness contract. A report with is_complete=true, status=complete, and both execution flags true is accepted as SAFE even when coverage_percent=0, entirely_uninspected_files=1, or ledger_exceptions contains a fatal entry. Validate those authoritative fields and reject contradictions so an incomplete SkillSpector report cannot fail open; add regressions for the contradictory cases.

[Tomauskasz]

Addressed in 6fd3a5d and updated onto current main in merge commit 4ffb55c. A claimed-complete report now requires present authoritative details with numeric 100% coverage, zero integer partial/uninspected counts, and empty ledger-exception/limitation lists. Missing or contradictory details fail closed; reports that omit analysis_completeness entirely retain the legacy path.

Verification after merging current main: 1,053 validator tests passed; the full suite passed with 5,482 passed, 17 skipped, and 4 deselected; Ruff passed; source and wheel builds passed; git diff --check passed. The merge also incorporates main's Gitleaks workflow repair.

Fresh CI, DCO, and Security runs were created, but GitHub marked them action_required before running. The contributor account cannot approve fork workflows, so they are waiting for maintainer approval.

[chrisknvidia]

[P1] Completeness accounting still has an unchecked contradiction

The follow-up addresses coverage, partial/uninspected counts, exceptions, and limitations, but leaves required counters unchecked. A report with total_components=2, scanned_components=1, fully_inspected_files=1, coverage_percent=100, and otherwise-complete flags is accepted. Please require nonnegative integer total/scanned/fully fields; for a complete report require scanned_components == total_components, validate fully_inspected_files against the producer's documented file-count invariant, and add missing/mismatched regressions.

[rng1995]

@Tomauskasz Gentle ping when you have a chance: the remaining completeness-contract review thread is still waiting for an update or response. Thank you for your patience throughout the review and for your contribution.

[wuzhenhua24]

Independent reproduction — this matches the root cause described here exactly, and I bisected the SkillSpector side.

Minimal reproducer. A skill with no findings at all is enough; the only trigger is a / in a heading, which the reference resolver records as an unresolved local reference:

---
name: repro
description: Minimal reproducer for the SkillSpector completeness contract mismatch. Use when reproducing the reported validation failure.
---

# Repro

## Overview

Prose only. No executable content.

### Input/Output Separation

Keep inputs and outputs distinct.

Behaviour across SkillSpector versions (same skill, same SkillEvaluator commit, --no-llm):

SkillSpector	score	severity	recommendation	is_complete	skillevaluator security-scan
2.9.6	0	LOW	SAFE	false	PASS
2.10.0	0	LOW	CAUTION	false (status: partial)	INCOMPLETE
2.11.0	0	LOW	CAUTION	false (status: partial)	INCOMPLETE

Failure message on 2.10.0+:

skillspector JSON field 'risk_assessment.recommendation' does not match the risk severity;
security scan did not complete

Three things worth noting:

1. The skill has zero findings and score: 0, yet the entire security scan result is discarded. The rejection is not scoped to one questionable claim in the report.
2. 2.9.6 already reports is_complete: false for this skill — it just doesn't escalate the recommendation. So the behavioural change in 2.10.0 is the fail-closed escalation, not completeness detection. That supports reading analysis_completeness before the recommendation/severity invariant, as this PR does.
3. The trigger surface is wide. ledger_exceptions is a single reference_unresolved from the reference_resolution phase, raised by a slash inside a Markdown heading. Prose-heavy skills hit this readily — the first real skill where I saw it had two such exceptions, one from a heading and one from a table cell.

Environment: SkillEvaluator 3bfba44 (main, reports 0.2.1), SkillSpector installed via uv tool install, macOS / Python 3.13.

Until this lands, we are pinning SkillSpector to v2.9.6 downstream, with a regression test that fails on 2.10.0+ so we can tell when the pin is safe to drop.

[Tomauskasz]

Follow-up from the complete three-lane review is in bdd69c1.

The ten contradictory/missing completeness-detail cases now share one parameter matrix and one scanner/assertion body. Runtime behavior and case coverage are unchanged; the test removes 23 net lines of duplicated mechanics.

Verification:

• focused completeness and legacy contract set: 12 passed;
• full serial suite: 5,482 passed, 17 skipped, 4 deselected;
• repository Ruff check passed;
• source and wheel builds passed;
• git diff --check passed.

[Tomauskasz]

Final test-only polish is in 1f25501.

The partial-report regression now overrides only fields that differ from the shared fixture. Current and legacy clean-report coverage now shares one parameterized test path. Production behavior and contract coverage are unchanged; the test file is 20 lines smaller.

Final verification: 5,482 passed, 17 skipped, 4 deselected; Ruff passed; source and wheel builds passed; git diff --check passed. The independent follow-up review found no issue.

Fresh CI, DCO, and Security runs for this head are action_required and are waiting for maintainer approval of the fork workflows.

[Tomauskasz]

The DCO-only failure is corrected in the rebased head 18b8fc9669670c9dc7a90f4095b5a0db35da2533.

The branch was rebased onto current main (b882b16a0ff604c1cc1590b9b72d0719906c65fd) with sign-off. All four non-merge PR commits now have parsed Signed-off-by trailers; the formerly unsigned bdd69c18 was rewritten as 54e51c8dcd8ae1cc291f4b643017ca42a16fb110.

The only rebase conflict was additive in CHANGELOG.md; the resolution preserves every current-main entry and the existing SkillSpector completeness entry. Production code and tests merged automatically.

Post-rebase verification: 5,488 passed, 17 skipped, 4 deselected; Ruff passed; source and wheel builds passed; git diff --check passed.

Fresh DCO, CI, and Security runs were created but are currently action_required. NVIDIA maintainer approval of the fork workflows is needed before they can execute.

[chrisknvidia]

[P1] Version-gate the completeness schema

This uses analysis_completeness field presence as the schema boundary, but real SkillSpector 2.9.6 --no-llm reports include that object without the 2.10 status field, so valid legacy scans fail here. Conversely, a report still labeled 2.10.0 can omit both new fields and take the legacy path; I reproduced that report being accepted. Please select the contract from a validated metadata.skillspector_version (with an explicit compatibility policy), require the 2.10+ fields, and add a captured real 2.9.6 fixture.

[chrisknvidia]

[P1] Preserve findings from partial scans

This returns to the caller before _process_skillspector_cli_result runs, so structurally valid issues from a partial report are discarded. I reproduced this with released SkillSpector 2.10.0 and 2.11.0: both partial JSON reports retained HIGH findings, while SkillEvaluator showed only the incompleteness error. Please separate structural validity from completeness, process trustworthy issues, then mark the scan incomplete; add a regression containing a valid HIGH issue.

[Tomauskasz]

The three September 3 P1 review findings and the subsequent contract findings are addressed in rebased head debc8436c41c1daf874bc80203e8e11bcf51d9a8 on current main (73b27dad60d3927e202ea6099ce79bb25053fd2b).

• Contract selection now uses a validated metadata.skillspector_version. The strict 2.9.6 path is covered by captured --no-llm JSON, while 2.10+ requires the full status-bearing schema.
• Coherent partial reports now retain validated findings, stay incomplete, and do not emit clean or successful summaries.
• Total, scanned, fully inspected, partially inspected, and uninspected counts must be non-negative integers and satisfy the producer equations; coverage is recomputed from those counts.
• Pre/post-filter finding counts must be non-negative integers, satisfy the producer ordering, and remain coherent with serialized active and suppressed findings under the 2.9.6 and 2.10 report behaviors.
• Versioned reports must contain a unique component inventory with non-empty paths and boolean executable evidence. Complete reports reconcile the inventory count and exact source/path issue coverage, with SC8 exempt because shipped bytecode is intentionally absent from public component metadata.
• Every versioned finding must identify a non-empty producer path. Known universal analyzers must complete exactly one work item per component on complete nonempty scans; partial reports and unknown future analyzers retain their existing compatibility paths.
• Occurrence-expanded rows under one modern compact identity must agree on the producer-copied severity, confidence, and finding ID; occurrence-specific paths and source details may still differ.

The follow-up also requires the captured 2.9.6 analyzer roster, the stable 2.10+ roster, and all semantic analyzers when LLM availability is confirmed. It validates analyzer work partitions and state agreement, and checks both execution markers, stage-specific LLM metadata, limitations, recommendation semantics, and non-empty 2.10 finding IDs. Self-review caught and fixed the partial LOW/CAUTION path, huge-integer coverage ordering, incomplete analyzer-roster acceptance, duplicate transitive analyzer IDs, released transitive-truncation output, unvalidated finding totals and identities, a real 2.10 pre-compaction score compatibility failure, source-provenance loss in score reconstruction, modern null-fingerprint findings collapsed by display prose, legacy null-text findings collapsed into one row, executable component evidence lost during post-filter scoring, lossy replacement of the producer score, fractional score truncation at the policy threshold, shipped bytecode removed before the artifact-integrity analyzer ran, inconsistent source-scope priority, and duplicate component identities. Exact SkillSpector 2.10.0 probes confirm that report-time deduplication can reduce two post-filter findings to one serialized issue, a non-executable source remains at score 40 even when another source uses the same executable path, three source scopes score 70 rather than collapsing to 40, and distinct null-fingerprint finding IDs remain score-distinct.

Independent review of the prior immutable head reproduced three additional counterexamples. Missing components allowed executable-risk understatement; compacted mixed executable/non-executable occurrences could select the wrong multiplier; and hidden raw duplicates made a serialized-only producer-score bound unsafe. The final contract rejects missing evidence and preserves only independently provable score evidence across compaction.

Final immutable-head review found three more P1s. Empty or absent issue paths bypassed executable-score matching; a complete report could claim zero universal analyzer work; and representative confidence copied during compaction could reject a genuine producer score. All six exact public-seam cases failed against the reviewed head and pass after the fixes. Captured 2.9.6, future-analyzer, and partial-report compatibility cases also pass.

A subsequent immutable-head review found one more P1 in that correction: one hidden raw finding or one occurrence-expanded identity erased independently provable risk from every visible rule. The final implementation deduplicates modern report identities, computes the finite global count of unknown raw findings, and allocates those findings once to the largest possible diminishing-weight reductions. The six-rule counterexamples now retain conservative score floors of 55 with one hidden raw finding and 30 with six expanded duplicates. Reports that claim fewer raw findings than their distinct compact identities also fail closed.

The last immutable-head review found that contradictory representative fields under one compact identity could forge a passing score. A six-rule report claiming 15/LOW/SAFE passed even though its raw findings score 69/HIGH/DO_NOT_INSTALL. The exact regression now fails closed, and a separate conflicting-finding-ID case is covered.

Post-rebase verification: 5,955 passed, 17 skipped, 4 deselected; Ruff passed; source and wheel builds passed; the focused validator suite passed 353 tests; live SkillSpector 2.10.0 clean and 14-advisory reference-skill probes passed against clean-checkout content; report-deduplication, source-scoped score, null-fingerprint, and SC8 floor probes passed; the fractional score-floor, staged-bytecode, and conflicting-compacted-identity regressions went red before their fixes and green afterward; the exact independent-review regressions failed against their prior heads and passed after the fixes; 2.9.6 and future-analyzer compatibility cases pass; the SC8 floor regression killed a no-floor mutation; git diff --check and captured-fixture JSON parsing passed. All five PR commits retain parsed Signed-off-by trailers.

Final exact-head producer-contract and repository-standards reviews found no P0-P2 issues.