diff --git a/CHANGELOG.md b/CHANGELOG.md index bdc6e72..55bc90c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,24 @@ All notable changes are recorded here. The project remains alpha before `1.0.0`, and minor releases may change experimental interfaces. +## 0.1.1 — 2026-08-03 + +### 新增 / Added + +- 新增 `verify-receipt` CLI，用于重新验证回执结构、阶段语义和内容哈希。 + Added the `verify-receipt` CLI for revalidating receipt structure, phase semantics, and content hashes. +- 新增无需 Docker 和第三方依赖的一键可信本地 Demo，并以回归测试保证可重复运行。 + Added a one-command trusted-local demo with no Docker or third-party dependency, covered by a repeatability regression test. +- 新增基于真实夹具输出的 README 终端演示图。 + Added a README terminal visual derived from real fixture output. + +### 变更 / Changed + +- 重构项目首页，明确目标用户、相对普通 CI 的差异、三分钟快速开始、适用范围、成熟度和常见问题。 + Reworked the project front page around target users, differentiation from typical CI, a three-minute quick start, fit, maturity, and FAQ. +- 收窄产品声明：当前是面向 Coding Agent 基础设施与评测工程师的 Alpha 协议实现，而不是通用测试平台或生产多租户沙箱。 + Narrowed the product claim: this is an alpha protocol implementation for coding-agent infrastructure and evaluation engineers, not a general test platform or production multi-tenant sandbox. + ## 0.1.0 — 2026-08-03 ### 新增 / Added diff --git a/Makefile b/Makefile index 1bcf3e8..495b449 100644 --- a/Makefile +++ b/Makefile @@ -2,11 +2,14 @@ PYTHON ?= python3 RUN_DIR ?= /tmp/patchproof-demo IMAGE ?= python:3.12.10-slim@sha256:fd95fa221297a88e1cf49c55ec1828edd7c5a428187e67b5d1805692d11588db -.PHONY: test demo clean +.PHONY: test demo-local demo clean test: PYTHONPATH=src $(PYTHON) -m unittest discover -s tests -v +demo-local: + PYTHONPATH=src $(PYTHON) scripts/run_demo.py + demo: PYTHONPATH=src $(PYTHON) -m patchproof propose \ --repo fixtures/calculator \ diff --git a/README.md b/README.md index f4ec981..b234c83 100644 --- a/README.md +++ b/README.md @@ -1,27 +1,132 @@ -# PatchProof +
+ +# PatchProof — AI Coding Agent 补丁验证与审计工具 + +**不要只相信 Agent 说“测试通过”；证明它修复了正确的问题。** + +**Do not merely trust an agent saying “tests passed.” Prove it fixed the right problem.** [![CI](https://github.com/eatdrop/patchproof/actions/workflows/ci.yml/badge.svg)](https://github.com/eatdrop/patchproof/actions/workflows/ci.yml) [![Python](https://img.shields.io/badge/Python-3.11%20%7C%203.12-3776AB)](https://www.python.org/) [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) [![Status: Alpha](https://img.shields.io/badge/status-alpha-orange.svg)](CHANGELOG.md) -PatchProof 是面向 AI 生成补丁的证据级验证工具。它把“补丁看起来合理”转换为可审计协议：仓库快照绑定、精确哈希审批、补丁前失败复现、补丁后复现、完整回归、外置隐藏测试和内容寻址回执。 +面向 Coding Agent 平台、评测与 CI/安全工程师的证据级补丁验证器：绑定准确 Diff 与仓库快照，执行补丁前后测试、完整回归和外置隐藏测试，再生成可重验回执。 -PatchProof is an evidence-grade validator for AI-generated patches. It turns “the patch looks plausible” into an auditable protocol: repository-snapshot binding, exact-hash approval, fail-before reproduction, pass-after reproduction, full regression, external hidden tests, and a content-addressed receipt. +An evidence-grade patch validator for coding-agent platforms, evaluations, and CI/security engineers: bind the exact diff to a repository snapshot, run fail-before/pass-after checks, full regression, and external hidden tests, then emit a re-verifiable receipt. -> PatchProof 验证外部候选补丁；它本身不是补丁生成 Agent。 +
+ +[![PatchProof 真实四阶段演示 / Real four-phase demo](docs/assets/patchproof-demo.svg)](https://github.com/eatdrop/patchproof/actions/workflows/ci.yml) + +> 图中四阶段结果来自仓库内真实夹具和 `make demo-local`；`proof_grade=true` 由断网、只读、非 root 的 Docker CI 作业验证。 > -> PatchProof validates externally supplied candidate patches; it is not a patch-generating Agent. +> The four phase results come from the repository's real fixture and `make demo-local`; `proof_grade=true` is established by the network-disabled, read-only, non-root Docker CI job. + +## 3 秒看懂 / Understand It in 3 Seconds + +| | 中文 | English | +|---|---|---| +| **是什么 / What** | 位于 Coding Agent 与合并操作之间的确定性验证层 | A deterministic validation layer between a coding agent and merge | +| **给谁 / Who** | Agent 平台、AI 评测、CI 与安全工程师 | Agent-platform, AI-evaluation, CI, and security engineers | +| **解决什么 / Why** | 防止“测试绿了，但没复现原问题、改了评分器或执行的不是审批过的补丁” | Prevent green checks that never reproduced the bug, changed the grader, or ran a patch different from the approved one | + +PatchProof 验证外部候选补丁，本身不调用模型、不生成补丁，也不替代人工代码审查。 + +PatchProof validates externally supplied candidate patches. It does not call a model, generate patches, or replace human code review. + +## 为什么普通 CI 不够 / Why Typical CI Is Not Enough + +普通单次 CI 可以证明“当前工作树通过了某些命令”，却不一定证明“候选补丁修复了声明的问题”。PatchProof 把容易被忽略的条件变成强制协议。 + +A typical one-pass CI job can show that a current worktree passed commands, but it may not prove that the candidate patch fixed the claimed problem. PatchProof makes the missing conditions explicit and mandatory. + +| 需要回答的问题 / Question | 普通单次 CI / Typical one-pass CI | PatchProof | +|---|---|---| +| 补丁前是否真实复现？ / Did the bug reproduce before the patch? | 通常未知 / Usually unknown | 基线复现必须失败 / Baseline reproduction must fail | +| 执行对象是否就是审批对象？ / Was the approved object executed? | 分支和文件名可变 / Branches and filenames can move | Diff + 仓库快照 + 完整哈希绑定 / Diff, snapshot, and full-hash binding | +| 候选补丁能否修改评分器？ / Can the candidate modify its grader? | 仓库内测试可能同时变化 / In-repo tests may change | Reproduction 与 Hidden Tests 外置复制 / Reproduction and hidden tests are copied externally | +| 真实仓库是否被副作用修改？ / Did validation mutate the real repository? | 取决于 Job 实现 / Depends on the job | 一次性副本执行并复查快照 / Disposable copies plus a post-run snapshot check | +| 结果能否离线重验？ / Can the result be rechecked offline? | 常见产物是日志 / Usually logs | 内容寻址 JSON 回执 / Content-addressed JSON receipt | + +这一方向不是单纯追逐热点：SWE-bench Verified 使用 `FAIL_TO_PASS` 测试筛选可信案例，Codex 官方方案也把隔离容器与执行期断网作为重要边界。PatchProof 将这些原则缩小为可嵌入其他 Agent 系统的本地协议。 + +This direction is more than trend-chasing: SWE-bench Verified uses `FAIL_TO_PASS` tests to filter credible cases, while the official Codex design treats isolated containers and disabled runtime internet as important boundaries. PatchProof narrows those principles into a local protocol that other agent systems can embed. + +- [SWE-bench Verified：FAIL_TO_PASS / SWE-bench Verified: FAIL_TO_PASS](https://openai.com/index/introducing-swe-bench-verified/) +- [Codex：隔离容器与执行期断网 / Codex: isolated container and disabled runtime internet](https://openai.com/index/introducing-codex/) + +## 3 分钟快速开始 / 3-Minute Quick Start + +本地可信 Demo 只需要 Python 3.11+，不需要 Docker，也不会修改真实夹具仓库。 + +The trusted-local demo needs only Python 3.11+. It does not require Docker or modify the real fixture repository. + +```bash +git clone https://github.com/eatdrop/patchproof.git +cd patchproof +make demo-local +``` + +没有 `make` 时可直接运行脚本： -## 为什么需要 / Why It Exists +If `make` is unavailable, run the script directly: -普通测试结果无法回答这些问题：测试是否真的执行？补丁前是否能复现故障？审批对象是否就是后来执行的补丁？测试是否被候选补丁修改？真实仓库是否发生了副作用？ +```bash +# macOS / Linux +PYTHONPATH=src python3 scripts/run_demo.py -A normal test result does not answer whether tests actually ran, the bug reproduced before the patch, the approved object is the executed patch, the candidate altered its grader, or the real repository changed as a side effect. +# Windows PowerShell +$env:PYTHONPATH="src"; python scripts/run_demo.py +``` -PatchProof 的闭环如下： +你会看到真实执行的四阶段结果： -PatchProof uses this proof loop: +You will see four phases that actually executed: + +```text +[PASS] fail before patch (1 test) +[PASS] pass after patch (1 test) +[PASS] full regression (1 test) +[PASS] external hidden tests (2 tests) + +validation: PASSED +repository unchanged: YES +receipt integrity: VERIFIED +proof grade: NO (trusted-local; Docker required) +``` + +本地 Demo 故意显示 `proof grade: NO`：它证明判定逻辑可运行，但没有借用 Docker 隔离路径的安全声明。需要隔离证据时使用下方 Docker 流程，或查看持续通过的 [Shadow Validation CI](https://github.com/eatdrop/patchproof/actions/workflows/ci.yml)。 + +The local demo deliberately reports `proof grade: NO`: it proves the verdict logic runs without borrowing the Docker path's isolation claim. Use the Docker workflow below—or inspect the continuously passing [shadow-validation CI](https://github.com/eatdrop/patchproof/actions/workflows/ci.yml)—when isolated evidence is required. + +## 适合谁 / Who It Is For + +适合： + +Good fit: + +- 构建 Coding Agent、自动修复或 Agent PR 流水线的工程师。 + Engineers building coding agents, automated repair, or agent-to-PR pipelines. +- 需要 fail-to-pass、pass-to-pass 与独立隐藏评分的评测作者。 + Evaluation authors who need fail-to-pass, pass-to-pass, and independent hidden grading. +- 想在合并前把候选 Diff、审批和执行证据绑定起来的 CI/安全团队。 + CI and security teams that need candidate diffs, approvals, and execution evidence bound before merge. +- 研究 Agent 权限分离、正确拒答和可审计执行的学习者。 + Practitioners studying agent authority separation, correct abstention, and auditable execution. + +暂不适合： + +Not yet a fit: + +- 用作通用测试框架或普通 CI 的替代品。 + Replacing general-purpose test frameworks or normal CI. +- 直接运行任意 shell、安装任意依赖或验证所有语言生态。 + Running arbitrary shell commands, installing arbitrary dependencies, or covering every language ecosystem. +- 在强对抗、多租户生产环境中执行恶意代码。 + Executing hostile code in a strongly adversarial multi-tenant production environment. + +## 工作原理 / How It Works ```text Read-only repository / 只读仓库 @@ -47,46 +152,47 @@ Content-addressed receipt + bilingual report 内容寻址回执 + 双语报告 ``` -## 安装 / Installation +任一阶段出现零测试、ImportError、超时、输出截断或基础设施失败，整体结果都会关闭失败。 + +Zero tests, ImportError, timeout, output truncation, or infrastructure failure in any phase fails closed. + +## 完整 Docker 流程 / Full Docker Workflow -当前版本为 Alpha，推荐从固定 Release 或源码安装。 +
+展开生产形态命令 / Expand production-shaped commands -The current release is alpha. Install from a pinned release or source revision. +### 1. 安装 / Install ```bash python -m pip install . patchproof --help ``` -PatchProof 运行时仅使用 Python 标准库。默认隔离验证需要本机 Docker CLI 与可用守护进程。 +PatchProof 运行时只使用 Python 标准库。隔离验证需要 Docker CLI 与可用守护进程。 -PatchProof has no third-party runtime dependency. Default isolated validation requires a local Docker CLI and daemon. +PatchProof uses only the Python standard library at runtime. Isolated validation requires a Docker CLI and daemon. -## 快速开始 / Quick Start +### 2. 绑定补丁 / Bind the Patch -### 1. 绑定候选补丁 / Bind a Candidate Patch - -输出产物必须位于目标仓库外部。 +输出产物必须放在目标仓库之外。 Output artifacts must remain outside the target repository. ```bash -PYTHONPATH=src python -m patchproof propose \ +mkdir -p /tmp/patchproof-demo + +patchproof propose \ --repo fixtures/calculator \ --diff fixtures/division-by-zero.diff \ --out /tmp/patchproof-demo/proposal.json ``` -### 2. 精确审批 / Approve the Exact Proposal - -审批必须重新提供完整 Proposal Hash；前缀、过期快照和其他提案都不会通过。 - -Approval requires the complete Proposal Hash. Prefixes, stale snapshots, and different proposals are rejected. +### 3. 审批准确哈希 / Approve the Exact Hash ```bash PROPOSAL_HASH="$(python -c 'import json; print(json.load(open("/tmp/patchproof-demo/proposal.json"))["proposal_hash"])')" -PYTHONPATH=src python -m patchproof approve \ +patchproof approve \ --proposal /tmp/patchproof-demo/proposal.json \ --proposal-hash "$PROPOSAL_HASH" \ --run-id demo \ @@ -94,21 +200,21 @@ PYTHONPATH=src python -m patchproof approve \ --out /tmp/patchproof-demo/approval.json ``` -`approved_by` 当前是声明式审计元数据，不是经过认证的身份。 +`approved_by` 是声明式审计元数据，不是认证身份。 -`approved_by` is declared audit metadata, not an authenticated identity. +`approved_by` is declared audit metadata, not authenticated identity. -### 3. 在 Docker 中验证 / Validate in Docker +### 4. 隔离验证 / Validate in Isolation -镜像必须包含精确 SHA-256 摘要；PatchProof 不会在验证时拉取镜像。 +镜像必须包含准确 SHA-256 摘要；PatchProof 不会在验证时自动拉取镜像。 -The image must include an exact SHA-256 digest. PatchProof never pulls an image during validation. +The image must include an exact SHA-256 digest. PatchProof never pulls it during validation. ```bash IMAGE='python:3.12.10-slim@sha256:fd95fa221297a88e1cf49c55ec1828edd7c5a428187e67b5d1805692d11588db' docker pull "$IMAGE" -PYTHONPATH=src python -m patchproof validate \ +patchproof validate \ --repo fixtures/calculator \ --proposal /tmp/patchproof-demo/proposal.json \ --approval /tmp/patchproof-demo/approval.json \ @@ -118,27 +224,94 @@ PYTHONPATH=src python -m patchproof validate \ --docker-image "$IMAGE" ``` -Docker Runner 默认使用： - -The Docker runner uses: +Docker Runner 使用固定摘要镜像、`--pull=never`、`--network=none`、只读根与仓库挂载、非 root 用户、移除 Capabilities、`no-new-privileges`、PID/内存/CPU/输出限制和超时清理。 -- 固定摘要镜像、`--pull=never` / digest-pinned image and `--pull=never`; -- `--network=none`、只读根和只读仓库挂载 / disabled network, read-only root and repository mount; -- UID/GID `65534`、`--cap-drop=ALL`、`no-new-privileges` / non-root user, dropped capabilities, and no-new-privileges; -- PID、内存、CPU、输出和超时限制 / PID, memory, CPU, output, and timeout limits; -- 超时后按随机容器名强制清理 / forced cleanup by randomized container name after timeout. +The Docker runner uses a digest-pinned image, `--pull=never`, `--network=none`, read-only root and repository mount, a non-root user, dropped capabilities, `no-new-privileges`, PID/memory/CPU/output limits, and timeout cleanup. -## 本地可信模式 / Trusted Local Mode +
-`--unsafe-local` 仅用于可信夹具和开发调试。即使四阶段测试通过，其回执也会明确标记 `isolated=false`、`proof_grade=false`。 - -`--unsafe-local` is only for trusted fixtures and development. Even when all four phases pass, the receipt explicitly records `isolated=false` and `proof_grade=false`. +## 重验回执 / Re-Verify a Receipt ```bash -make test -make demo IMAGE="$IMAGE" +RECEIPT="$(find /tmp/patchproof-demo/audit -name 'validation-*.json' -print -quit)" +patchproof verify-receipt --receipt "$RECEIPT" ``` +该命令重新验证严格结构、阶段语义和内容哈希，并输出 `integrity_valid`、`validation_success` 与 `proof_grade`。它证明内容内部一致且未被静默修改，不认证是谁执行或签署了验证。 + +The command revalidates strict structure, phase semantics, and the content hash, then reports `integrity_valid`, `validation_success`, and `proof_grade`. It proves internal consistency and detects silent modification; it does not authenticate who executed or signed the validation. + +## 可验证证据 / Verifiable Evidence + +- **71 项自动化测试 / 71 automated tests**：覆盖 Diff、路径、符号链接、快照、审批、补丁应用、Runner、CLI、Demo、回执与 Skill 脚本。 + They cover diffs, paths, symlinks, snapshots, approvals, patch application, runners, CLI, demo behavior, receipts, and Skill scripts. +- **Python 3.11/3.12 CI**：每次 PR 与 `main` 推送均运行。 + It runs on every pull request and `main` push. +- **真实 Docker Shadow Validation / Real Docker shadow validation**：在 GitHub Actions 中执行完整四阶段闭环并上传审计产物。 + It executes the full four-phase loop in GitHub Actions and uploads audit artifacts. +- **只读前向案例 / Read-only forward case**：[IssueLens 复用记录](docs/issuelens-case-study.md)扫描 63 个真实项目文件并验证前后快照一致。 + The [IssueLens reuse record](docs/issuelens-case-study.md) scans 63 real project files and verifies identical before/after snapshots. +- **可安装 Release / Installable release**：[GitHub Releases](https://github.com/eatdrop/patchproof/releases)包含经过构建与隔离安装验证的 wheel。 + The [GitHub Releases](https://github.com/eatdrop/patchproof/releases) include a wheel verified through build and isolated installation. + +## 当前成熟度 / Current Maturity + +| 已实现并测试 / Implemented and tested | 明确不声称 / Explicitly not claimed | +|---|---| +| 有界 UTF-8 文本与 Unified Diff / Bounded UTF-8 text and unified diffs | 任意二进制、重命名或所有 Diff 方言 / Arbitrary binaries, renames, or every diff dialect | +| 固定 Python `unittest discover` / Fixed Python `unittest discover` | pytest/tox/nox、依赖安装或任意命令 / pytest/tox/nox, dependency installation, or arbitrary commands | +| 单租户可信宿主上的 Docker 加固 / Docker hardening on a trusted single-tenant host | 生产级恶意多租户沙箱 / Production hostile multi-tenant sandbox | +| 哈希完整性与内容寻址 / Hash integrity and content addressing | 数字签名、可信时间戳或身份认证 / Digital signatures, trusted timestamps, or identity authentication | +| 独立协议、测试与真实前向审计 / Independent protocol, tests, and real forward audit | 已被大量团队采用或证明适用于所有 Agent / Broad adoption or proof for every agent system | + +v0.1 的优先级是把边界做窄、做真、做可拒绝。扩大 Runner 与语言生态之前，项目更需要第二个真实系统集成和外部用户反馈。 + +The v0.1 priority is to keep the boundary narrow, truthful, and rejectable. Before broadening runner and language support, the project needs a second real-system integration and external user feedback. + +## 常见问题 / FAQ + +### PatchProof 会生成或审查代码吗？ / Does PatchProof generate or review code? + +不会。它是模型无关的候选补丁验证层，可以接在 Codex、Claude Code、Copilot、自研 Agent 或人工补丁之后。 + +No. It is a model-agnostic candidate-patch validation layer that can follow Codex, Claude Code, Copilot, an in-house agent, or a human-authored patch. + +### 为什么补丁前测试必须失败？ / Why must the test fail before the patch? + +如果测试在补丁前已经通过，补丁后通过不能证明它修复了目标问题。失败原因仍必须由独立测试准确描述；ImportError 和零测试不会被当成有效复现。 + +If a test already passed before the patch, passing afterward proves no repair. The independent test must still describe the intended failure; ImportError and zero-test runs are not accepted as reproduction. + +### 为什么本地 Demo 通过却不是 `proof_grade`？ / Why is a passing local demo not `proof_grade`? + +本地模式执行可信夹具但不隔离。只有声明 `isolated=true` 且四阶段全部通过时，`proof_grade` 才为真。 + +Local mode runs trusted fixtures without isolation. `proof_grade` becomes true only when a runner declares `isolated=true` and all four phases pass. + +### Docker 提示找不到镜像怎么办？ / What if Docker says the image is missing? + +PatchProof 使用 `--pull=never` 防止验证时镜像漂移。先手动 `docker pull` README 中的完整摘要镜像，再运行验证。 + +PatchProof uses `--pull=never` to prevent image drift during validation. Pull the full digest-pinned image shown above before running validation. + +### 为什么不支持任意测试命令？ / Why not support arbitrary test commands? + +任意命令会显著扩大命令注入、依赖下载和宿主副作用面。v0.1 选择固定 `unittest` 协议；扩展 Runner 必须先定义同等严格的权限与证据边界。 + +Arbitrary commands greatly expand command-injection, dependency-download, and host-side-effect surfaces. v0.1 intentionally fixes the `unittest` protocol; future runner adapters must first define equally strict authority and evidence boundaries. + +### 回执哈希等于数字签名吗？ / Is the receipt hash a digital signature? + +不等于。哈希能发现内容变化，但不能认证审批人或验证执行者。需要真实性或不可抵赖性时，应在外层加入签名、身份提供方和可信时间戳。 + +No. A hash detects content changes but does not authenticate the approver or validation runner. Add signatures, an identity provider, and trusted timestamps when authenticity or non-repudiation is required. + +### 可以用于私有代码吗？ / Can it be used with private code? + +工具本身不调用云端 API；Docker Runner 默认断网并只读挂载一次性副本。但宿主、Docker 守护进程、固定镜像和外置测试仍属于信任边界，请先阅读 [SECURITY.md](SECURITY.md)。 + +The tool itself calls no cloud API; the Docker runner disables networking and mounts disposable copies read-only. The host, Docker daemon, pinned image, and external tests remain trust boundaries, so read [SECURITY.md](SECURITY.md) first. + ## Python API / Python API ```python @@ -172,43 +345,28 @@ assert receipt.proof_grade ## 可复用 Codex Skills / Reusable Codex Skills -仓库包含两个经过官方结构校验并可独立使用的 Skill： - -The repository includes two independently usable, structurally validated Skills: - -- [`verifiable-agent-audit`](skills/verifiable-agent-audit/SKILL.md)：审计 Agent 权限、信任边界、审批、隔离、恢复、预算、评测和副作用。 - Audits Agent authority, trust boundaries, approvals, isolation, recovery, budgets, evaluations, and side effects. -- [`agent-eval-builder`](skills/agent-eval-builder/SKILL.md)：构建固定 Manifest、独立 Grader、正确拒答、失败分类、有效分母和 CI。 - Builds pinned manifests, independent graders, correct-abstention cases, failure taxonomies, valid denominators, and CI gates. - -## 验证现状 / Verification Status - -- 70 项标准库自动化测试覆盖 Diff、路径、符号链接、快照、审批、补丁应用、Runner、回执、CLI 和 Skill 脚本。 - 70 standard-library tests cover diffs, paths, symlinks, snapshots, approvals, patch application, runners, receipts, CLI behavior, and Skill scripts. -- Python 3.11/3.12 CI。 - Python 3.11/3.12 CI. -- 独立 Docker 作业运行真实四阶段闭环并上传审计产物。 - An independent Docker job runs the real four-phase loop and uploads its audit artifacts. - -## 安全边界 / Security Boundary - -PatchProof 不提供生产级恶意多租户沙箱，也不认证审批人身份，不安装目标项目依赖，不支持任意测试命令。v0.1 仅支持有界 UTF-8 文本文件、Unified Diff 和 Python `unittest` 发现模式。 - -PatchProof is not a production hostile multi-tenant sandbox, does not authenticate approver identity, does not install target dependencies, and does not accept arbitrary test commands. v0.1 supports bounded UTF-8 text files, unified diffs, and Python `unittest` discovery only. +仓库还包含两个经过官方结构校验、可独立使用的方法型 Skill： -详见 [SECURITY.md](SECURITY.md) 和 [架构文档](docs/architecture.md)。 +The repository also includes two independently usable, structurally validated workflow Skills: -See [SECURITY.md](SECURITY.md) and the [architecture document](docs/architecture.md). +- [`verifiable-agent-audit`](skills/verifiable-agent-audit/SKILL.md)：审计 Agent 权限、信任边界、审批、隔离、恢复、预算、评测与副作用。 + Audits agent authority, trust boundaries, approvals, isolation, recovery, budgets, evaluation, and side effects. +- [`agent-eval-builder`](skills/agent-eval-builder/SKILL.md)：构建固定 Manifest、独立 Grader、正确拒答、有效分母、失败分类与 CI。 + Builds pinned manifests, independent graders, correct-abstention cases, valid denominators, failure taxonomies, and CI gates. -真实复用记录见 [IssueLens 案例](docs/issuelens-case-study.md)。 +## 深入阅读 / Further Reading -See the [IssueLens case study](docs/issuelens-case-study.md) for a real reuse record. +- [架构与信任边界 / Architecture and trust boundaries](docs/architecture.md) +- [安全模型与限制 / Security model and limitations](SECURITY.md) +- [工程文章：把不确定补丁放进确定性边界 / Engineering article: putting uncertain patches inside deterministic boundaries](docs/engineering-boundaries.md) +- [IssueLens 真实复用案例 / Real IssueLens reuse case](docs/issuelens-case-study.md) +- [版本记录 / Changelog](CHANGELOG.md) ## 来源与独立性 / Origin and Independence -PatchProof 从 IssueLens 项目的验证协议中抽取，但拥有独立命名空间、哈希域、公共 API、CLI、测试、CI、文档和 Release 生命周期。IssueLens 是其设计来源和首个公开参考场景；当前版本不要求 IssueLens 依赖 PatchProof。 +PatchProof 从 IssueLens 的验证协议中抽取，但拥有独立命名空间、哈希域、公共 API、CLI、测试、CI、文档和 Release 生命周期。它没有复制 AppExpo 或任何实习公司的内部代码、数据与业务实现。 -PatchProof was extracted from the IssueLens validation protocol but owns an independent namespace, hash domains, public API, CLI, tests, CI, documentation, and release lifecycle. IssueLens is its design origin and first public reference scenario; the current release does not require IssueLens to depend on PatchProof. +PatchProof was extracted from the IssueLens validation protocol but owns an independent namespace, hash domains, public API, CLI, tests, CI, documentation, and release lifecycle. It contains no copied AppExpo or internship-company internal code, data, or business implementation. ## 许可证 / License diff --git a/docs/assets/patchproof-demo.svg b/docs/assets/patchproof-demo.svg new file mode 100644 index 0000000..06d056b --- /dev/null +++ b/docs/assets/patchproof-demo.svg @@ -0,0 +1,82 @@ + + PatchProof four-phase validation demo + Real trusted-local fixture output with four passing phases, unchanged repository, verified receipt integrity, and an isolated Docker proof available in CI. + + + + + + + + + + + + + + + + + + + PatchProof + AI coding-agent patch validation with evidence, not promises. + + + + + + + + + real fixture output / 真实夹具输出 + + + $ + make demo-local + PASS + fail before patch + 1 test + PASS + pass after patch + 1 test + PASS + full regression + 1 test + PASS + external hidden tests + 2 tests + + repository unchanged + YES + receipt integrity + VERIFIED + trusted-local proof grade + NO + Docker CI proof grade + TRUE + + + VALIDATION PASSED + 4 independent gates + + + + + + TESTS / 自动化测试 + 71 passing + + + RUNTIME / 运行依赖 + stdlib only + + + NETWORK / 验证网络 + disabled + + + ARTIFACT / 验证产物 + auditable receipt + + diff --git a/pyproject.toml b/pyproject.toml index 8bd6e31..6ae52cd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,22 +4,37 @@ build-backend = "setuptools.build_meta" [project] name = "patchproof" -version = "0.1.0" -description = "Evidence-grade validation for AI-generated patches." +version = "0.1.1" +description = "Verify AI coding-agent patches with fail-before/pass-after tests, isolated execution, and auditable receipts." readme = "README.md" requires-python = ">=3.11" license = { text = "MIT" } authors = [{ name = "eatdrop" }] -keywords = ["coding-agent", "patch-validation", "agent-safety", "docker"] +keywords = [ + "ai-coding-agent", + "agent-evaluation", + "agent-safety", + "docker", + "patch-validation", + "software-supply-chain", +] classifiers = [ "Development Status :: 3 - Alpha", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", + "Topic :: Software Development :: Testing", + "Topic :: Software Development :: Quality Assurance", ] dependencies = [] +[project.urls] +Homepage = "https://github.com/eatdrop/patchproof" +Repository = "https://github.com/eatdrop/patchproof" +Issues = "https://github.com/eatdrop/patchproof/issues" +Changelog = "https://github.com/eatdrop/patchproof/blob/main/CHANGELOG.md" + [project.scripts] patchproof = "patchproof.cli:main" diff --git a/scripts/run_demo.py b/scripts/run_demo.py new file mode 100755 index 0000000..c15c298 --- /dev/null +++ b/scripts/run_demo.py @@ -0,0 +1,76 @@ +#!/usr/bin/env python3 +"""Run PatchProof's trusted local fixture without external dependencies.""" + +from __future__ import annotations + +import tempfile +from pathlib import Path + +from patchproof.approval import PatchApproval +from patchproof.proposal import PatchProposal +from patchproof.repository import ReadOnlyRepository +from patchproof.runner import UnsafeLocalRunner +from patchproof.validator import ValidationReceipt, store_validation, validate_patch + + +ROOT = Path(__file__).resolve().parents[1] +PHASE_LABELS = { + "baseline_reproduction": "补丁前复现 / fail before patch", + "patched_reproduction": "补丁后复现 / pass after patch", + "full_regression": "完整回归 / full regression", + "hidden_tests": "隐藏测试 / external hidden tests", +} + + +def main() -> int: + repository = ReadOnlyRepository(ROOT / "fixtures" / "calculator") + before = repository.snapshot() + proposal = PatchProposal.create( + unified_diff=(ROOT / "fixtures" / "division-by-zero.diff").read_text( + encoding="utf-8" + ), + base_snapshot=before.digest, + ) + approval = PatchApproval.create( + run_id="quickstart-demo", + proposal=proposal, + approved_by="local-demo", + supplied_proposal_hash=proposal.proposal_hash, + ) + receipt = validate_patch( + repository=repository, + proposal=proposal, + approval=approval, + reproduction_tests=ROOT / "fixtures" / "reproduction", + hidden_tests=ROOT / "fixtures" / "hidden", + runner=UnsafeLocalRunner(timeout_seconds=10), + ) + unchanged = repository.snapshot() == before + with tempfile.TemporaryDirectory(prefix="patchproof-demo-") as directory: + stored = store_validation(receipt, Path(directory)) + loaded = ValidationReceipt.from_json( + stored.receipt_path.read_text(encoding="utf-8") + ) + integrity_verified = loaded == receipt + + print("PatchProof 可信本地演示 / trusted-local demo") + print("------------------------------------------------") + for phase in receipt.phases: + status = "PASS" if phase.passed else "FAIL" + print( + f"[{status}] {PHASE_LABELS[phase.name]} " + f"({phase.tests_run} test{'s' if phase.tests_run != 1 else ''})" + ) + print("------------------------------------------------") + print(f"验证结果 / validation: {'PASSED' if receipt.success else 'FAILED'}") + print(f"真实仓库未变 / repository unchanged: {'YES' if unchanged else 'NO'}") + print( + "回执完整性 / receipt integrity: " + f"{'VERIFIED' if integrity_verified else 'INVALID'}" + ) + print("证据等级 / proof grade: NO (trusted-local; Docker required)") + return 0 if receipt.success and unchanged and integrity_verified else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/patchproof/__init__.py b/src/patchproof/__init__.py index 381d6a1..968d773 100644 --- a/src/patchproof/__init__.py +++ b/src/patchproof/__init__.py @@ -16,4 +16,4 @@ "validate_patch", ] -__version__ = "0.1.0" +__version__ = "0.1.1" diff --git a/src/patchproof/cli.py b/src/patchproof/cli.py index d8eccff..cec375a 100644 --- a/src/patchproof/cli.py +++ b/src/patchproof/cli.py @@ -11,13 +11,13 @@ from .proposal import PatchProposal from .repository import ReadOnlyRepository from .runner import DockerRunner, UnsafeLocalRunner -from .validator import store_validation, validate_patch +from .validator import ValidationReceipt, store_validation, validate_patch def build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( prog="patchproof", - description="Evidence-grade validation for AI-generated patches.", + description="Evidence-grade patch validation for AI coding agents.", ) subparsers = parser.add_subparsers(dest="command", required=True) @@ -49,6 +49,12 @@ def build_parser() -> argparse.ArgumentParser: action="store_true", help="Run trusted fixtures without isolation; never use for untrusted code.", ) + + verify_receipt = subparsers.add_parser( + "verify-receipt", + help="Recompute and verify a receipt's internal integrity.", + ) + verify_receipt.add_argument("--receipt", required=True) return parser @@ -61,6 +67,8 @@ def main(argv: Sequence[str] | None = None) -> int: return _approve(args) if args.command == "validate": return _validate(args) + if args.command == "verify-receipt": + return _verify_receipt(args) raise ValueError("unsupported command") except (OSError, RuntimeError, ValueError) as exc: print( @@ -174,6 +182,36 @@ def _validate(args: argparse.Namespace) -> int: return 0 if receipt.success else 2 +def _verify_receipt(args: argparse.Namespace) -> int: + receipt_path = Path(args.receipt).expanduser().resolve(strict=True) + receipt = ValidationReceipt.from_json( + read_bounded_regular(receipt_path).decode("utf-8") + ) + print( + json.dumps( + { + "status": "receipt_integrity_verified", + "integrity_valid": True, + "validation_success": receipt.success, + "proof_grade": receipt.proof_grade, + "isolated": receipt.isolated, + "receipt_hash": receipt.receipt_hash, + "phases": [ + { + "name": phase.name, + "passed": phase.passed, + "tests_run": phase.tests_run, + } + for phase in receipt.phases + ], + }, + ensure_ascii=False, + sort_keys=True, + ) + ) + return 0 + + def _outside_repository(repository_root: Path, value: Path) -> Path: target = absolute_no_resolve(value) comparison = target.resolve(strict=False) diff --git a/tests/test_cli.py b/tests/test_cli.py index 6d837d8..daf7baa 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -154,6 +154,26 @@ def test_end_to_end_unsafe_local_is_explicitly_not_proof_grade(self) -> None: self.assertFalse(result["proof_grade"]) self.assertEqual(len(list(audit.glob("*.json"))), 1) self.assertEqual(len(list(audit.glob("*.md"))), 1) + receipt = next(audit.glob("*.json")) + verify_output = io.StringIO() + with redirect_stdout(verify_output), redirect_stderr(io.StringIO()): + verify_status = main( + ["verify-receipt", "--receipt", str(receipt)] + ) + verified = json.loads(verify_output.getvalue()) + self.assertEqual(verify_status, 0) + self.assertTrue(verified["integrity_valid"]) + self.assertTrue(verified["validation_success"]) + self.assertFalse(verified["proof_grade"]) + + tampered = parent / "tampered-receipt.json" + tampered_payload = json.loads(receipt.read_text()) + tampered_payload["success"] = False + tampered.write_text(json.dumps(tampered_payload), encoding="utf-8") + self.assertEqual( + self._main(["verify-receipt", "--receipt", str(tampered)]), + 1, + ) @staticmethod def _main(arguments: list[str]) -> int: diff --git a/tests/test_demo.py b/tests/test_demo.py new file mode 100644 index 0000000..beb5f0a --- /dev/null +++ b/tests/test_demo.py @@ -0,0 +1,36 @@ +from __future__ import annotations + +import os +import subprocess +import sys +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +class DemoTests(unittest.TestCase): + def test_trusted_local_demo_is_truthful_and_repeatable(self) -> None: + environment = os.environ.copy() + environment["PYTHONPATH"] = str(ROOT / "src") + for _ in range(2): + completed = subprocess.run( + [sys.executable, str(ROOT / "scripts" / "run_demo.py")], + cwd=ROOT, + env=environment, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=30, + check=False, + ) + self.assertEqual(completed.returncode, 0, completed.stderr) + self.assertIn("validation: PASSED", completed.stdout) + self.assertIn("repository unchanged: YES", completed.stdout) + self.assertIn("receipt integrity: VERIFIED", completed.stdout) + self.assertIn("proof grade: NO", completed.stdout) + + +if __name__ == "__main__": + unittest.main()