Add four native readers, covering five more harnesses - #16
Conversation
Opencode (and Z.ai's Z Code, whose CLI is an Opencode fork sharing its schema exactly as OpenClaude shares Claude Code's), Goose, Copilot CLI, and Qwen. `ingest` now reads nine harnesses with no manual export step. A store-backed reader has a different shape from a file-per-session one, and three things follow. `read` takes the session id as well as the path, because every session shares one file. The store is opened SQLITE_OPEN_READ_ONLY — it belongs to a program that may be running right now, and a read-only handle makes "engram never writes to another tool's database" structural rather than a promise. And `--max-bytes` has to measure the session rather than the file. For a harness that writes one file per session those are the same thing, which is why the check reads as a file-size check in the existing readers. Opencode's store is 550 MB of every session at once, so measuring the file refused all 34 sessions of a project with "transcript is 550273024 bytes" — a transcript nobody had asked for. The guard exists to stop one runaway conversation exhausting memory, so the store readers accumulate text and check against that. SessionRef.bytes is likewise the session's own size now, summed from its rows, rather than the store's size repeated for every session in it. Per-reader, the things that would otherwise be rediscovered: Copilot CLI's row is two turns. The prompt and the reply share one row and one timestamp — the harness records no separate time for the reply — so the halves need distinct source_uuid suffixes, or the second collides with the first through turn_id and is dropped by INSERT OR IGNORE. Goose's message_id is nullable, so identity falls back to the row's position, the same reasoning codex uses for records with no id of their own. Qwen reuses the Claude Code reader. Its records carry Claude Code's envelope but put the body at message.parts[], whose entries have no `type` field at all. collect_text now treats an untyped block carrying text as text; without that it counted an entire conversation as non_message while reporting a perfectly healthy filter histogram. The readerless-harness test lists only harnesses that genuinely still have none. It must never be padded with one engram can now read, or it would assert a refusal that should no longer happen. Verified on real stores: opencode 34 sessions/547 turns for one project, zcode 20 turns, goose 12 sessions, copilot-cli 16 turns, qwen 3 turns. Gates: fmt, clippy -D warnings, 283 tests, REUSE 3.3, makeinfo clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016i16R4GhdSffsboRYq97Fs
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0c1d442726
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let Some(raw_time) = timestamp else { | ||
| // A turn with no time cannot be ordered, and inventing one would | ||
| // silently reorder the conversation. Counted, not guessed. | ||
| stats.missing_uuid += 1; | ||
| continue; |
There was a problem hiding this comment.
When a Copilot row has a NULL timestamp, this branch silently skips both nonempty halves and reports missing_uuid, even though the row has a stable turn_index identity. The capture therefore succeeds while omitting real conversation text; treat an absent timestamp as BadTimestamp, just like an invalid timestamp, rather than converting it into an unrelated filter count.
AGENTS.md reference: AGENTS.md:L276-L277
Useful? React with 👍 / 👎.
| `src/harness.rs` + `src/transcript/{mod,claude_code,redact}.rs`. Reads the session file a harness already writes for itself and stores each message as an ordinary memory, so `recall`/`search`/`context`/`consolidate` see the real conversation. | ||
|
|
||
| Two readers exist: `claude_code` and `codex`. Adding a third means adding a `ReaderKind` variant, which the two `match`es in `transcript/mod.rs` then force you to handle. | ||
| Six readers exist — `claude_code`, `codex`, `opencode`, `goose`, `copilot`, `qwen` — serving **nine** harnesses. Adding another means adding a `ReaderKind` variant, which the two `match`es in `transcript/mod.rs` then force you to handle. |
There was a problem hiding this comment.
This updated list says the new readers are supported, but the other authoritative documentation still says readers exist only for Claude Code and Codex (AGENTS.md:34, doc/engram.texi:797) and the manual's support table marks Opencode, Qwen, Goose, and Copilot CLI as unsupported (doc/engram.texi:1338-1342). Users following the command reference or manual will therefore be told not to use the functionality added by this commit.
Useful? React with 👍 / 👎.
| Some("toolRequest" | "toolResponse" | "toolConfirmationRequest") => { | ||
| stats.tool_use += 1; |
There was a problem hiding this comment.
For normal Goose toolResponse parts, this combined arm increments tool_use, so the serialized filter report always undercounts tool_result and inflates requests; sessions containing a request and response are reported as two uses and zero results. Split toolResponse into an arm that increments stats.tool_result so consumers can rely on the existing distinct counters.
Useful? React with 👍 / 👎.
1 participant
Opencode (and Z.ai's Z Code, whose CLI is an Opencode fork sharing its schema — the
claude_code/OpenClaude relationship again), Goose, Copilot CLI, and Qwen.engram ingestnow reads nine harnesses with no manual export step.Two of these were listed as
Unsupporteduntil #15 said otherwise. This is the follow-through.A store-backed reader has a different shape
Three things follow from "every session lives in one SQLite file":
readtakes the session id as well as the path. The dispatcher already holds theSessionRef, so it passes both.SQLITE_OPEN_READ_ONLY. It belongs to a program that may be running right now. A read-only handle makes "engram never writes to another tool's database" structural rather than a promise.--max-bytesmust measure the session, not the file.That last one bit immediately:
550 MB is the whole store — every session at once. For a file-per-session harness the file and the session are the same thing, which is why the check reads as a file-size check in
claude_codeandcodex; here it refused all 34 sessions of a project over a transcript nobody asked for. The guard exists to stop one runaway conversation exhausting memory, so the store readers accumulate text and check against that instead.SessionRef.bytesis now the session's own size too, summed from its rows, rather than 550 MB repeated 209 times.Per-reader, the things that would otherwise be rediscovered
Copilot CLI's row is two turns. The prompt and the reply share one row and one timestamp — the harness records no separate time for the reply. The halves need distinct
source_uuidsuffixes (0:user,0:assistant), or the second collides with the first throughturn_idand is silently dropped byINSERT OR IGNORE.Goose's
message_idis nullable, so identity falls back to the row's position — the same reasoningcodexalready uses for records carrying no id of their own.Qwen reuses the Claude Code reader. Its records carry Claude Code's envelope but put the body at
message.parts[], whose entries have notypefield at all.collect_textnow treats an untyped block carrying text as text — without it, an entire conversation counted asnon_messagewhile reporting a perfectly healthy-looking filter histogram. Only session discovery differs (projects/<mangled-cwd>/chats/), which is the only reasonqwen.rsexists.Opencode parts are mostly not conversation. A real store:
tool9854,step-start6439,step-finish6335,reasoning3945,text3322,patch461. Counting them rather than dropping them silently is the whole point of the filter histogram.Verified on real stores
One test note
The readerless-harness test now lists only harnesses that genuinely still have none (
antigravity,cursor,kimi,vscode). It must never be padded with one engram can read, or it would assert a refusal that should no longer happen.Gates
fmt·clippy -D warnings· 283 tests · REUSE 3.3 ·makeinfoclean🤖 Generated with Claude Code
https://claude.ai/code/session_016i16R4GhdSffsboRYq97Fs