Skip to content

feat(sleep): per-night evidence chain + live prompt registry#151

Open
Alphaxalchemy wants to merge 1 commit into
microsoft:mainfrom
Alphaxalchemy:feat/sleep-evidence-chain
Open

feat(sleep): per-night evidence chain + live prompt registry#151
Alphaxalchemy wants to merge 1 commit into
microsoft:mainfrom
Alphaxalchemy:feat/sleep-evidence-chain

Conversation

@Alphaxalchemy

@Alphaxalchemy Alphaxalchemy commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Per-night evidence chain + live prompt registry for SkillOpt-Sleep

Why (origin of this change)

This started from a user-side audit of what a completed sleep night actually lets you verify. Running real nights (DeepSeek via the openai_compatible backend) against real transcripts, the report tells you that the miner processed N conversations and what rules the reflect step added — but the evidential chain in between is invisible:

your transcripts → these specific mined checks → these specific replay failures → these specific edits → this gate decision

None of the intermediate artifacts (the miner's verbatim exchange, which checks each task carried, which replay attempt failed which check, the reflect model's raw reply before parsing, the gate's arithmetic) were recorded anywhere. Given how strict/fragile the acceptance gate is by design (strict improvement or nothing), a rejected night was essentially unexplainable after the fact.

The motivation goes beyond auditability. Once every miner/reflect exchange is persisted per night, three things become possible that aren't today:

  1. Check validation — programmatically ask "did the miner's checks actually correspond to what the user accepted?" and score the miner itself.
  2. Prompt regression testing — when someone edits the miner prompt, replay archived sessions through both versions and diff the mined tasks.
  3. A meta-training loop — the miner and reflect prompts are themselves just documents; SkillOpt's own gate machinery can evolve them against the archived exchanges. The system optimizing its own optimizer is exactly what this architecture supports, and this PR lays the substrate for it.

What

  • skillopt_sleep/evidence.py — append-only, thread-safe, secret-redacted evidence.jsonl written into each night's staging dir. The full chain is logged: harvested sessions → miner exchanges (verbatim prompt/reply) → mined tasks with their checks → train/val split assignment → every replay model call (phase-tagged; cache hits marked key-only) → per-task scores with failing checks named → reflect exchanges + parsed edits → gate baseline/trials and the final decision with its score arithmetic spelled out → staged artifacts. Config-gated: evidence_log (default on), evidence_max_chars truncation cap.
  • skillopt_sleep/prompts.py — central registry of the four LLM prompt templates (miner / attempt / judge / reflect). Defaults are byte-identical to the previously inlined strings (covered by a test). User overrides in prompts.json take effect on the next model call (mtime-checked) — no restart. This is what makes the prompts first-class, versionable, and eventually gate-optimizable objects.
  • cycle.py — builds dual backends from config (optimizer_* / target_*), pre-creates the staging dir so evidence lands beside the report; staging.new_staging_dir() de-collides same-second runs; latest_staging() skips non-adoptable (evidence-only) folders so a no-tasks night can never be accidentally adopted.

Tests

tests/test_sleep_evidence.py — 8 pure-stdlib, deterministic, no-network tests: chain completeness end-to-end on the mock backend, redaction/truncation/ordering, corrupt-line tolerance, evidence_log=False disable path, prompt-default byte-parity, override round-trip + live effect without restart, and the no-tasks-night adoption guard.

Full suite: no new failures vs pristine main (same 11 pre-existing research-suite loader errors on both; this branch adds 8 tests, all passing).

Live smoke: a real DeepSeek night through the openai_compatible backend produced a 95-event chain in which the gate's rejection is fully reconstructable from the log, including its formula (baseline = 0.5·0.000 + 0.5·0.333 = 0.167; candidate = 0.167 → no strict improvement → reject).

Series

This is part 1 of a 3-PR series: this PR (evidence chain + prompt registry, no UI) → #152 (a stdlib-only local dashboard that renders the chain and edits the prompts) → #153 (a new harvest source, per the "more sources" direction of #97).

* skillopt_sleep/evidence.py — append-only, thread-safe, redacted
  evidence.jsonl per night: harvest sessions -> miner exchanges (verbatim
  prompt/reply) -> mined tasks with checks -> split assignment -> every
  replay attempt (phase-tagged, cache hits marked) -> per-task scores with
  failing checks named -> reflect exchanges + parsed edits -> gate trials
  and the final decision with its score arithmetic -> staged artifacts.
  Config-gated (evidence_log, default on; evidence_max_chars cap).

* skillopt_sleep/prompts.py — central registry of the four LLM prompt
  templates (miner/attempt/judge/reflect), byte-identical defaults to the
  previously inlined strings; user overrides in prompts.json take effect
  on the next model call (mtime-checked), no restart needed.

* cycle.py builds dual backends from config (optimizer_*/target_*),
  pre-creates the staging dir so evidence lands beside the report;
  staging.new_staging_dir() de-collides same-second runs;
  latest_staging() now skips non-adoptable (evidence-only) folders.

* tests/test_sleep_evidence.py — 8 no-network stdlib tests: chain
  completeness, redaction/truncation/ordering, disable flag, prompt
  override round-trip + live effect, no-tasks-night adoption guard.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant