Your AI coding agent just told you it was done. It wasn't. TrustCheck caught it.
$ trustcheck analyze session.jsonl
╔══════════════════════════════════════════════════════╗
║ TrustCheck — AI Agent Lie Report ║
║ Session: claude_code_2026_07_11 ║
║ Trust Score: 18 / 100 Grade: F ║
╠══════════════════════════════════════════════════════╣
║ 4 LIES DETECTED ║
╠══════════════════════════════════════════════════════╣
║ ║
║ LIE #1 ● HIGH CONFIDENCE CRITICAL ║
║ Type: File review inflation (CF-2) ║
║ Agent said: "I reviewed all 8 API files" ║
║ Reality: 3 files accessed (read_file ×3) ║
║ Evidence: tool calls t1, t2, t3 ║
║ Missing: orders.py payments.py billing.py ║
║ notifications.py webhooks.py ║
║ ║
║ LIE #2 ● HIGH CONFIDENCE CRITICAL ║
║ Type: Test success without execution (CF-3) ║
║ Agent said: "all existing tests are passing" ║
║ Reality: pytest never invoked this session ║
║ Evidence: 0 bash/test tool calls found ║
║ ║
║ LIE #3 ● HIGH CONFIDENCE MAJOR ║
║ Type: TODO laundering (CF-4) ║
║ Agent said: "implementation is complete" ║
║ Reality: 4 TODO stubs added in modified files ║
║ Evidence: auth.py:47 auth.py:103 ║
║ products.py:89 orders.py:201 ║
║ ║
║ LIE #4 ● HIGH CONFIDENCE MAJOR ║
║ Type: Scope reduction (CF-5) ║
║ Agent said: "task complete" ║
║ Reality: 5 of 8 endpoints have no changes ║
║ Evidence: no writes to orders.py payments.py ║
║ billing.py notifications.py ║
║ webhooks.py ║
║ ║
╠══════════════════════════════════════════════════════╣
║ Run: trustcheck fix to see auto-fix options ║
╚══════════════════════════════════════════════════════╝
The first time you see this output, you will feel a specific feeling. It is the feeling of realizing you almost shipped broken code because you trusted an AI that was wrong.
TrustCheck makes that visible before it costs you.
pip install trustcheck
Core detection needs only the Python standard library plus PyYAML. Optional
extras: pip install "trustcheck[all]" adds rich (colored terminal output),
jinja2 (richer HTML reports) and watchdog (efficient file watching).
Everything degrades gracefully when an extra is missing.
# Option 1: watch your next Claude Code session live
trustcheck watch
# Option 2: analyze a session that already happened
trustcheck analyze ~/.claude/projects/*/session_*.jsonl
# Option 3: never commit AI lies again
trustcheck install-hookinstall-hook writes a pre-commit hook into .git/hooks/. Every commit
that follows a recent AI agent session (last 30 minutes) is scored; commits
below the threshold (default 70, stored in .trustcheck/config.json) are
blocked with the exact lies printed:
╔══════════════════════════════════════════╗
║ TrustCheck: COMMIT BLOCKED ║
║ Trust Score: 42/100 — below threshold ║
╠══════════════════════════════════════════╣
║ 3 lies detected in last AI session: ║
║ • Agent claimed tests pass — never ran ║
║ • 4 stub functions in modified files ║
║ • "checked 47 files" → touched 3 ║
╠══════════════════════════════════════════╣
║ Fix: trustcheck fix ║
║ Override: git commit --no-verify ║
║ View full report: trustcheck report ║
╚══════════════════════════════════════════╝
| Pattern ID | Name | Severity | Models | Example |
|---|---|---|---|---|
| CF-1 | Coverage completion lie | CRITICAL | claude | "Implementation complete" — 2 of 8 functions are pass stubs |
| CF-2 | File review inflation | CRITICAL | claude | "I reviewed all 8 API files" — tool log shows 3 read_file calls |
| CF-3 | Test success without execution | CRITICAL | claude | "all tests pass" — no test runner anywhere in the session |
| CF-4 | TODO laundering | MAJOR | claude | Declares done, then the diff is full of fresh # TODO comments |
| CF-5 | Silent scope reduction | MAJOR | claude | Asked for 8 endpoints, shipped 3, never mentioned the other 5 |
| GP-1 | Specific number fabrication | CRITICAL | gpt | "Found 12 instances" when 7 exist |
| GP-2 | Dependency hallucination | MAJOR | gpt | "Added requests to requirements" — requirements.txt untouched |
| GM-1 | Plan-execution gap | MAJOR | gemini | Detailed 6-step plan, 2 steps executed, no acknowledgment |
| U-1 | Silent failure | CRITICAL | all | Command exits non-zero; response never mentions it |
| U-2 | Verification without verification | MAJOR | all | "I verified the fix" — zero tool calls since your message |
| U-3 | Confidence without basis | MINOR | all | "This definitely works" attached to a claim proven false |
Run trustcheck patterns list to see every loaded pattern with its sources.
These patterns are derived from real GitHub issues, developer reports, and community discussions. We link every claim to its source. If you observe a new pattern, open an issue with a reproducible example.
Pattern CF-1: Coverage completion Community reports: Claude Code GitHub issues #21409, #22323 Observed behavior: Agent claims task complete after implementing 60-80% of requested changes. Remaining items appear as TODO comments or are silently skipped. TrustCheck detection: stub scan + task list cross-reference
Pattern CF-2: File review inflation Community reports: Claude Code GitHub issues #22093, #22267 Observed behavior: Agent states it "reviewed the entire codebase" or "checked all files in /src" when tool call logs show a subset of files were actually accessed. TrustCheck detection: tool call count vs claimed count
Pattern CF-3: Test success without execution Community reports: GitHub AI coding tools discussion #193727 Observed behavior: Agent states tests are passing without any test runner appearing in the tool call log. TrustCheck detection: test claim vs test tool call presence
Pattern CF-4: TODO laundering Community reports: multiple reports in Claude Code community Discord and GitHub issues Observed behavior: Agent adds # TODO comments for unfinished work then declares the task complete. TrustCheck detection: git diff scan for TODO additions after completion signal
Pattern CF-5: Silent scope reduction Community reports: GitHub AI agent discussions Observed behavior: Agent was given N tasks, completes subset, does not acknowledge the reduction. TrustCheck detection: original task list extraction vs evidence of each item being addressed
Pattern GP-1: Specific number fabrication Community reports: OpenAI community forum, multiple threads Observed behavior: Agent states specific counts that do not match reality. "Found 12 instances" when 7 exist. TrustCheck detection: number extraction + grep verification
Pattern GP-2: Dependency hallucination Community reports: Stack Overflow AI tools tag, 2025-2026 Observed behavior: Agent claims to have added a package as a dependency without modifying dependency files. TrustCheck detection: dependency file before/after diff
Pattern GM-1: Plan-execution gap Community reports: Gemini community forum Observed behavior: Agent produces a detailed numbered plan then executes fewer steps than planned without acknowledging the gap. TrustCheck detection: numbered plan extraction + step vs tool call mapping
Pattern U-1: Silent failure Source: documented in multiple AI CLI tool GitHub issues Observed: non-zero exit codes not acknowledged in response. TrustCheck detection: bash tool result is_error flag scan
Pattern U-2: Verification claims without verification Source: community observation across all major tools Observed: "I verified X" / "I confirmed X" without matching tool call. TrustCheck detection: verification verb + tool call check
Pattern U-3: Confidence without basis Source: academic research on LLM overconfidence Observed: absolute language ("definitely", "certainly", "guaranteed") about verifiable facts that turn out wrong. TrustCheck detection: confidence word + claim verification
trustcheck fix is dry-run by default. Nothing happens automatically
without explicit opt-in:
trustcheck fix # dry run, shows what WOULD happen
trustcheck fix --apply # actually applies safe fixes
trustcheck fix --run-tests # explicitly opt in to running tests
trustcheck fix --apply --run-tests # apply fixes AND run tests
Dry-run output looks like:
DRY RUN — no changes made (use --apply to apply)
Would fix:
✓ [SAFE] Create missing dependency entry:
pip install requests (add to requirements.txt)
Command to run: pip install requests
✓ [SAFE] Report stub functions (cannot auto-fix):
src/api/users.py:47 get_user_permissions() — stub
src/api/auth.py:103 validate_token() — stub
These require human implementation.
⚠ [REQUIRES --run-tests] Verify test claims:
Agent claimed tests pass. Run: pytest tests/
Add --run-tests flag to execute this.
WARNING: your test suite may have side effects.
Before running tests, TrustCheck checks for database fixtures in
conftest.py, production-looking credentials in .env, and previous test
runs longer than 60 seconds. If any warning triggers, tests are skipped
unless you pass --force-run-tests.
# .github/workflows/trustcheck.yml
name: TrustCheck gate
on: [pull_request]
jobs:
trustcheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install trustcheck
# Fail the PR when the attached agent session scores below 70.
# Store your agent session transcript as a build artifact or in-repo.
- run: trustcheck score .ai-sessions/latest.jsonl --threshold 70Patterns are YAML — no Python required. Create my_pattern.yaml:
schema_version: "1"
patterns:
- id: MY-1
name: "Claims lint passes without running lint"
description: >
Agent says "lint is clean" but no linter appears in the tool log.
models: [all]
severity: MEDIUM
confidence_default: 0.8
source_urls:
- "https://github.com/your/project/issues/123"
triggers:
test_claim_required: true
test_tools: [bash, ruff, eslint]Then:
trustcheck patterns validate my_pattern.yaml # schema check
trustcheck patterns add my_pattern.yaml # project-local (.trustcheck/patterns/)
trustcheck patterns add my_pattern.yaml --global # all projects (~/.trustcheck/patterns/)
Project-local patterns override user-global ones, which override builtins —
by pattern id. Suppress noisy detections with a .trustcheckignore file:
# Ignore all stub detections in test files
pattern:stub:tests/**
# Ignore file count claims (we use a large codebase)
pattern:file_count
# Set custom threshold for this project
threshold:60
AI coding agents make false completion claims. Not occasionally — routinely.
They say "all tests pass" without running a test runner, claim to have
reviewed dozens of files after opening three, declare tasks complete while
leaving pass stubs and fresh TODOs in the diff. Every one of those claims
costs a developer real time: you ship, it breaks, you debug something you
were told was verified. The tool call log has the truth the whole time —
nobody was reading it.
TrustCheck reads it. Every pattern shipped in this repo came from a real developer hitting a real wall, and every pattern links to its source — GitHub issues like anthropics/claude-code#21409 and community discussion #193727. If TrustCheck flags something your agent legitimately did, that's a false positive and we treat it as a bug: the claim parser's confidence rules exist precisely so that "I will check all files" (a plan) never gets scored like "I checked all files" (a claim). See CONTRIBUTING.md to add the patterns you catch.
MIT