Skip to content

Namespace evidence by stage and make evidence publication complete and inspectable #9

Description

@douglasjarquin

Problem

Test and Lint still write the same stdout.log and stderr.log names under one run location, so later Lint evidence can overwrite Test evidence. Other stages also lack a consistent evidence namespace and durable association between stage status and the evidence that supports it.

Required implementation

Use independent stage paths:

<run-id>/intent/
<run-id>/rebase/
<run-id>/review/
<run-id>/test/stdout.log
<run-id>/test/stderr.log
<run-id>/document/
<run-id>/lint/stdout.log
<run-id>/lint/stderr.log
<run-id>/push/
<run-id>/pr/
<run-id>/ci/
  1. Introduce a stage-aware evidence API rather than making each caller invent path prefixes.
  2. Test and Lint must never overwrite each other.
  3. Record, where applicable:
    • command/argv or a safely redacted representation
    • timeout
    • exit code
    • bounded stdout/stderr
    • stage completion state
  4. Review evidence must include:
    • prompt version
    • output-schema version
    • base and candidate identity
    • structured findings response
    • bounded diagnostics
  5. CI evidence must associate each check with:
    • check name/state
    • details link
    • workflow run ID when applicable
    • rerun rounds
    • bounded failure-log excerpts
  6. Preserve partial evidence when a stage fails or is canceled, marking incomplete records explicitly.
  7. Redact secrets before durable publication.
  8. Reject path traversal, symlink escape, and unsafe filenames.
  9. Ensure status stage records contain their evidence references.
  10. Ensure PR evidence links are actually accessible to the user; do not publish local-only refs as though GitHub can resolve them.
  11. Make evidence publication failure block Push/PR when evidence is required.
  12. Apply bounded retention and document cleanup behavior.
  13. Keep per-file and per-run size limits.

Acceptance criteria

  • Test output remains intact after Lint completes.
  • Every stage has an independent evidence namespace.
  • Review and CI evidence contains enough identity to reproduce the decision.
  • Partial evidence survives failure/cancellation.
  • Stage status links to the correct evidence.
  • Evidence references in a PR resolve for a human.
  • Secret-bearing output is redacted before persistence/publication.
  • Traversal and symlink attacks are rejected.
  • Retention is bounded and tested.

Required tests

  • Test then Lint with distinct output
  • concurrent/partial stage writes
  • stage evidence references in status
  • review prompt/schema evidence
  • CI multi-check evidence
  • required publication failure before Push
  • secret redaction
  • path traversal and symlink escape
  • per-file/per-run limits
  • retention cleanup
  • accessible publication verification

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions