Skip to content

fix(eval): score unknown principals consistently - #20

Open
vsolano9 wants to merge 1 commit into
AgentPostmortem:mainfrom
vsolano9:fix-8-unresolvable-principal-recall
Open

fix(eval): score unknown principals consistently#20
vsolano9 wants to merge 1 commit into
AgentPostmortem:mainfrom
vsolano9:fix-8-unresolvable-principal-recall

Conversation

@vsolano9

Copy link
Copy Markdown

Closes #8.

An unresolvable principal now receives recall 1.0 only when no documents were expected and 0.0 when expected documents are missing. mean_recall also drops the impossible is not None filter, keeping the declared float type and aggregation behavior aligned.

Regression coverage exercises both unknown-principal boundaries against the real PostgreSQL/pgvector fixture.

Validated locally:

  • 60 tests on Python 3.12 and Python 3.11 with real pgvector
  • both focused regressions repeated 20 times
  • strict seeded access-control evaluation: 11 cases, 0 leaks, mean recall 1.0, pass rate 1.0
  • wheel and sdist build; clean wheel install, dependency check, import/API smoke, and CLI help
  • compileall and git diff --check

Ruff reports no patch-introduced lint or format delta: the single SIM103 finding and existing formatting drift are identical on upstream main.

Score an unresolvable principal as perfect recall only when no documents were expected, while preserving a miss when expected documents exist. Remove the dead optionality filter and cover both boundaries with real-database regression tests.

Closes AgentPostmortem#8
@github-actions

Copy link
Copy Markdown

Welcome, @vsolano9, and thanks for your first pull request to VaultRAG.

A quick look at CONTRIBUTING.md covers how things are set up and run here.

I'll review this shortly. Ask anything in the thread if something is unclear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cases with an unresolvable principal score recall 0.0 and drag down mean_recall

1 participant