feat(sleep): per-night evidence chain + live prompt registry#151
Open
Alphaxalchemy wants to merge 1 commit into
Open
feat(sleep): per-night evidence chain + live prompt registry#151Alphaxalchemy wants to merge 1 commit into
Alphaxalchemy wants to merge 1 commit into
Conversation
* skillopt_sleep/evidence.py — append-only, thread-safe, redacted evidence.jsonl per night: harvest sessions -> miner exchanges (verbatim prompt/reply) -> mined tasks with checks -> split assignment -> every replay attempt (phase-tagged, cache hits marked) -> per-task scores with failing checks named -> reflect exchanges + parsed edits -> gate trials and the final decision with its score arithmetic -> staged artifacts. Config-gated (evidence_log, default on; evidence_max_chars cap). * skillopt_sleep/prompts.py — central registry of the four LLM prompt templates (miner/attempt/judge/reflect), byte-identical defaults to the previously inlined strings; user overrides in prompts.json take effect on the next model call (mtime-checked), no restart needed. * cycle.py builds dual backends from config (optimizer_*/target_*), pre-creates the staging dir so evidence lands beside the report; staging.new_staging_dir() de-collides same-second runs; latest_staging() now skips non-adoptable (evidence-only) folders. * tests/test_sleep_evidence.py — 8 no-network stdlib tests: chain completeness, redaction/truncation/ordering, disable flag, prompt override round-trip + live effect, no-tasks-night adoption guard.
This was referenced Jul 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Per-night evidence chain + live prompt registry for SkillOpt-Sleep
Why (origin of this change)
This started from a user-side audit of what a completed sleep night actually lets you verify. Running real nights (DeepSeek via the
openai_compatiblebackend) against real transcripts, the report tells you that the miner processed N conversations and what rules the reflect step added — but the evidential chain in between is invisible:None of the intermediate artifacts (the miner's verbatim exchange, which checks each task carried, which replay attempt failed which check, the reflect model's raw reply before parsing, the gate's arithmetic) were recorded anywhere. Given how strict/fragile the acceptance gate is by design (strict improvement or nothing), a rejected night was essentially unexplainable after the fact.
The motivation goes beyond auditability. Once every miner/reflect exchange is persisted per night, three things become possible that aren't today:
What
skillopt_sleep/evidence.py— append-only, thread-safe, secret-redactedevidence.jsonlwritten into each night's staging dir. The full chain is logged: harvested sessions → miner exchanges (verbatim prompt/reply) → mined tasks with their checks → train/val split assignment → every replay model call (phase-tagged; cache hits marked key-only) → per-task scores with failing checks named → reflect exchanges + parsed edits → gate baseline/trials and the final decision with its score arithmetic spelled out → staged artifacts. Config-gated:evidence_log(default on),evidence_max_charstruncation cap.skillopt_sleep/prompts.py— central registry of the four LLM prompt templates (miner / attempt / judge / reflect). Defaults are byte-identical to the previously inlined strings (covered by a test). User overrides inprompts.jsontake effect on the next model call (mtime-checked) — no restart. This is what makes the prompts first-class, versionable, and eventually gate-optimizable objects.cycle.py— builds dual backends from config (optimizer_*/target_*), pre-creates the staging dir so evidence lands beside the report;staging.new_staging_dir()de-collides same-second runs;latest_staging()skips non-adoptable (evidence-only) folders so a no-tasks night can never be accidentally adopted.Tests
tests/test_sleep_evidence.py— 8 pure-stdlib, deterministic, no-network tests: chain completeness end-to-end on the mock backend, redaction/truncation/ordering, corrupt-line tolerance,evidence_log=Falsedisable path, prompt-default byte-parity, override round-trip + live effect without restart, and the no-tasks-night adoption guard.Full suite: no new failures vs pristine
main(same 11 pre-existing research-suite loader errors on both; this branch adds 8 tests, all passing).Live smoke: a real DeepSeek night through the
openai_compatiblebackend produced a 95-event chain in which the gate's rejection is fully reconstructable from the log, including its formula (baseline = 0.5·0.000 + 0.5·0.333 = 0.167; candidate = 0.167 → no strict improvement → reject).Series
This is part 1 of a 3-PR series: this PR (evidence chain + prompt registry, no UI) → #152 (a stdlib-only local dashboard that renders the chain and edits the prompts) → #153 (a new harvest source, per the "more sources" direction of #97).