validate-schema produces false-positive 'critical' findings against already-fixed schema

[AndriiPasternak31]

Summary

The dev-methodology /validate-schema skill produces critical-severity false positives against schema state that is already correct in src/backend/db/schema.py.

Incident

• Triggering report: Schema Validation: 25 drift issues (11 critical tables + 14 critical columns missing from schema.py) trinity#770 (auto-filed 2026-05-11 by scheduled /validate-schema run, filer = vybe remote Trinity agent).
• Issue claimed 11 tables and 14 columns were missing from src/backend/db/schema.py (priority-p1, type-bug).
• Reality: every named table/column was already present in schema.py since PR fix(schema): backfill schema.py to match migrations.py reality (#691) trinity#712 (commit d56012b9, merged 2026-05-08).
• tests/unit/test_schema_parity.py (4 tests) passes 4/4 in origin/dev, confirming schema.py is in parity with migrations.py.

Problem

The validator skill apparently inspects something other than schema.py (perhaps architecture.md, or a stale snapshot) when flagging "Critical: missing table/column", but applies a critical severity that implies a real schema gap. Recommended behavior:

1. Consult tests/unit/test_schema_parity.py (or any project-provided parity oracle) before flagging columns/tables as missing.
2. Distinguish doc-drift from schema-drift: drift in architecture.md should NEVER be critical — it's informational/medium at most.
3. Avoid re-filing: the validator currently auto-files a new GitHub issue on a weekly schedule, so the same false-positive issue gets refiled every week until the validator is fixed. Either dedupe by content hash or stop opening issues when output matches a recently-closed/open one.

Why release-blocking

The skill is part of dev-methodology, which is installed via /plugin install dev-methodology@abilityai. It is scheduled on remote Trinity agents and auto-opens GitHub issues against downstream projects. A false-positive labelled priority-p1 / type-bug creates downstream alert fatigue and damages contributor trust in the toolchain. The skill currently makes a project that has zero schema drift look like it has 25 critical issues.

Proposed owner / SLA

• Owner: dev-methodology maintainer (please self-assign).
• SLA: 2 weeks from filing date (target resolution 2026-05-25).
• Interim mitigation: pause the scheduled /validate-schema on remote Trinity agents (courtesy ping to vybe is going out in a comment on Schema Validation: 25 drift issues (11 critical tables + 14 critical columns missing from schema.py) trinity#770).

References

• Filed incident: Schema Validation: 25 drift issues (11 critical tables + 14 critical columns missing from schema.py) trinity#770
• Structural fix already shipped: fix(schema): backfill schema.py to match migrations.py reality (#691) trinity#712
• Doc-drift cleanup PR (named in incident, in-progress): Abilityai/trinity branch AndriiPasternak31/issue-770-plan
• Parity test: tests/unit/test_schema_parity.py::TestSchemaParity (4 tests, all green)

[AndriiPasternak31]

Additional failure mode in the same skill family

While running /dev-methodology:validate-pr against Abilityai/trinity#788 (the doc-only follow-up to Abilityai/trinity#770 that triggered this issue), I hit the same false-positive class in a sibling skill:

validate-pr requires docs/memory/changelog.md — Trinity intentionally has none

The validate-pr skill (in dev-methodology/skills/validate-pr/SKILL.md) does:

```

2.1 Changelog Entry (REQUIRED)

Check if `docs/memory/changelog.md` is in the changed files list.
…
If missing: Flag as FAIL - "Changelog not updated"
```

Trinity's CLAUDE.md (root rule #4 "Tiered Documentation Updates") explicitly says:

Documentation requirements scale with change type (change history is tracked via git commits):

• Bug fix: Descriptive commit message only
• Feature / API change: architecture.md or feature-flows/ as needed
• New capability: requirements.md + feature-flows/

There is no docs/memory/changelog.md in the Trinity repo, by design.

validate-pr therefore auto-FAILs every PR Trinity has ever merged, despite the project following a different (and explicitly documented) convention.

Same root cause as the original report

Same failure class as the validate-schema bug in this issue: a dev-methodology skill encodes one convention as a hard requirement, then produces false-positive FAILs against projects that opted into a different (documented) convention. The fix shape is also the same — the skill should respect project-level convention markers (CLAUDE.md rules) before enforcing generic methodology gates.

Suggested scope for this issue

Either:

• A) Expand this issue's scope to cover the whole dev-methodology skill family (validate-schema + validate-pr + likely others). Single fix would be: a "project convention detector" that reads CLAUDE.md / project config and tailors the required-doc checklist accordingly.
• B) File a sibling issue for validate-pr and keep this one focused on validate-schema. The two skills currently fail in the same way for the same root reason, so I'd lean A.

Owner / 2-week SLA from the original issue still applies. Happy to file a sibling issue against this repo if you prefer (B); just let me know.

Other skills worth auditing in the same family

While at it, the other dev-methodology skills that prescribe project conventions and could be subject to the same mismatch:

• validate-config (checks docker-compose vs .env.example — Trinity-specific paths)
• validate-architecture (checks naming/structural rules from architecture.md)
• release (assumes a CHANGELOG.md or release-notes flow)
• commit (assumes specific commit-link convention to GitHub Issues)

Worth a quick spot-check pass on each.