Check that the skills still match the documentation

[rebekaburnett]

Stacked on #5. Base is plugin-skills, so the diff here is only the checking machinery. GitHub retargets this to main
when #5 merges.

Why

The skills restate facts from vippsas/vipps-developer-docs. That documentation changes without touching this
repository, and nothing was watching for it. There is no file to sync — the risk is that a fact stops being true.

Everything here runs on GitHub Actions runners. Nothing needs a developer's machine.

One repository drives both

The review job lives here and checks the documentation repository out read-only. Nothing is installed, duplicated, or
mirrored on that side, and the documentation repository needs no workflow of its own.

What is here

plugins/vipps/sources.json declares which documentation pages each skill is derived from — 68 pages — plus the
documentation repository, its branch, and reviewed_commit, the commit last audited. Site paths convert to repository
paths mechanically: /docs/x.md is docs/x.mdx.

scripts/check_sources.py asserts every declared and cited page still resolves, and that no skill cites a page it
has not declared, so the declarations cannot quietly go stale. Standard library only.

scripts/changed_sources.py maps a documentation diff onto the skills that summarize the changed pages, so the
review looks at what moved instead of re-deriving every claim.

.github/workflows/check-sources.yml runs the source check on pull requests touching plugins/, on push to main,
and every Monday. The scheduled run is the one that earns its keep, since a page can be renamed with no commit here.

.github/workflows/review-skill-accuracy.yml is the weekly audit. It diffs reviewed_commit against the
documentation main, reviews only the pages that changed, and compares the numbers, enums, endpoint paths, and error
codes against what each skill claims. It opens a draft pull request when it finds a difference, does nothing when it
does not, and never pushes to main.

Reading the repository rather than the site

The published Markdown is generated from .mdx at build time. The repository is ahead of the deploy, keeps content the
raw generator strips, and includes generated-specs/ and the _common/ partials where a lot of the truth actually
lives. Most importantly it can be diffed, which is what makes this cheap and precise. The published site stays the
target for link checking, since that is what an agent fetches at runtime.

Secrets, all optional

Secret	Effect if absent
ANTHROPIC_API_KEY	Review job reports a notice and passes. Nothing red
APP_ID + APP_PRIVATE_KEY	Falls back to DOCS_REPO_TOKEN
DOCS_REPO_TOKEN	Falls back to the public documentation site: no diffing, every page in scope, higher cost per run

A GitHub App with Contents: read on vipps-developer-docs is the better of the two, being organization owned with
short-lived tokens. Both repositories are internal, so GITHUB_TOKEN alone cannot reach across.

Behaviour worth knowing

• reviewed_commit advances only when a review opens a pull request. A clean run leaves it, so the next run re-reads the
  same window. Deliberate: it can waste tokens, never skip a page. Bump it by hand after a confirmed clean run.
• When nothing changed, the model is never invoked. That is what makes weekly affordable.
• workflow_dispatch takes force_full_review to audit everything regardless of the marker.

Checks

• python scripts/check_sources.py passes: 68 URLs, all 200.
• All 62 derived .mdx paths exist in the documentation repository.
• changed_sources.py was run against real history: across the last 60 documentation commits it correctly identified 15
  changed declared pages and attributed them to skills. Against the seeded reviewed_commit it reports nothing to
  review, which is right, since the skills were written from that state.
• Both workflow files parse as YAML.

Worth one manual workflow_dispatch run once ANTHROPIC_API_KEY exists, to confirm the action inputs behave as expected
before trusting the schedule.

🤖 Generated with Claude Code