ci: add Dreamer eval lane and harden evidence - #88
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
ahrav
changed the title
|
You have reached your Codex usage limits for security reviews. Please try again later. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: efac388f72
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const variance = aggregateDreamerEvalVariance(groupReports, args.repeat); | ||
| writeFileSync(join(groupDir, "variance.json"), `${JSON.stringify(variance, null, 2)}\n`); | ||
| } | ||
| if (!canStartDreamerEvalRun(startedAtMs, args.deadlineMinutes, Date.now(), reports.length)) break; |
There was a problem hiding this comment.
When the deadline expires after any completed run, this outer-loop break prevents every remaining selected scenario/task group from producing a directory or variance.json. The archived artifact therefore cannot distinguish a group that was selected but entirely skipped by the deadline from one excluded by CLI filters, and the new missingRuns evidence only covers the partially completed group. Record the selected groups or emit a red variance artifact for each wholly skipped group before stopping.
Useful? React with 👍 / 👎.
| const itemLabel = `${label}.receiptOutcomes[${index}]`; | ||
| const item = record(entry, itemLabel); | ||
| exact(item, ["claimId", "operation", "outcome"], itemLabel); | ||
| exact(item, ["requestDigest", "operationKey", "outcome", "affectedClaimIds"], itemLabel); |
There was a problem hiding this comment.
When parsing an archived dreamer-eval-report/v1 emitted before this commit, each receipt still has the previous {claimId, operation, outcome} shape, so this new exact-field check rejects it even though DREAMER_EVAL_REPORT_SCHEMA remains dreamer-eval-report/v1. This breaks offline loading of already-versioned run evidence; either accept the old v1 receipt representation or bump the report schema and provide explicit version handling.
Useful? React with 👍 / 👎.
Code Review SummaryStatus: 2 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
Files Reviewed (24 files)
Fix these issues in Kilo Cloud Reviewed by gemini-3.7-flash · Input: 139.4K · Output: 13.9K · Cached: 534.4K |
1 participant
Summary
Adds credential-free PR checks and a scheduled/manual live Dreamer evaluation workflow. It also hardens scorer proofs, corpus pressure, receipt attribution, deterministic invocation ordering, and report persistence when cleanup fails.
Stack
Validation
Stack created with GitHub Stacks CLI • Give Feedback 💬