Skip to content

Phase 2 oracle: 20-query recall@10 with real SPECTER2 + BGE-M3 #3

Description

@deOliveira-R

Manual acceptance test required before tagging v1.0.

Per plan §6 Phase 2: the 20-query recall@10 oracle and <500ms cold-start oracle need the real library + real models loaded. `tools/phase2_search_oracle.py` already exists; it needs a curated queries file.

Steps

  1. Full Zotero migration first: `grimoire migrate zotero` (~8 min for 15k items).
  2. `grimoire index` to populate SPECTER2 item embeddings (~8 min) and BGE-M3 chunks (overnight, CPU-only).
  3. Draft 20 queries + gold answers (item IDs or DOIs) from your research notes — the ones you'd actually ask.
  4. Wire the queries file path into `tools/phase2_search_oracle.py` if not already supported.
  5. Run the oracle against the full corpus; compare keyword / semantic / hybrid.

Acceptance

  • 20 curated queries + gold-answer item IDs captured (commit `tools/phase2_queries.jsonl` or similar).
  • Hybrid recall@10 ≥ 0.90 per plan target.
  • Cold-start semantic query latency < 500ms on the TrueNAS host.
  • Regression baseline stored in `project_oracle_results.md` memory or repo docs for future regressions.

Notes

  • Small queries file (20 lines) is fine; format as `{query, gold_item_ids: [..], notes: "..."}`.
  • BGE-M3 indexing is the slow part — schedule overnight. SPECTER2 alone is ~8 min.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions