Skip to content

Add an overall deadline for the complete compliance audit sequence #466

Description

@viraatc

Context

When audit: is enabled, the CLI runs the normal benchmark first (including finalize_benchmark() and accuracy scoring), then calls run_audit(). audit.only: true skips the preceding main benchmark.

PR #409 gives each AuditRunSpec subrun its own setup-inclusive run_timeout_s deadline so audit subruns match normal benchmark-run semantics. That is not an overall audit budget: a reference phase and fixed-sample phase can each consume the full value, and AuditTest.verify() runs afterward outside those per-subrun deadlines.

origin/main has no active run watchdog for audits; it only applies the existing response/metrics drain budgets inside each subrun.

Scope

Add a first-class overall audit deadline in a follow-up MR rather than overloading the per-benchmark run_timeout_s contract in PR #409. Define its ownership alongside phase configuration work in #449 and scoring/finalization cancellation work in #464.

Acceptance criteria

  • One deadline begins before the first audit subrun setup and is shared across every planned audit subrun.
  • Remaining budget, rather than a fresh full budget, is passed to each subrun.
  • The contract explicitly covers or boundary-checks AuditTest.verify() and result writing.
  • audit.only: true uses the same audit deadline semantics.
  • Expiry produces a non-zero exit, honest incomplete/interrupted artifacts, and no leaked children.
  • Existing per-subrun response drain, metrics drain, and interrupted teardown-grace semantics remain independently configurable.
  • Integration coverage blocks deterministically during multiple audit lifecycle stages and proves the overall budget is not reset.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions