Add engram import, for transcripts a harness exported - #14
Conversation
`ingest` reads the session file a harness writes for itself, which reaches a harness engram has a reader for and nothing else. That left two gaps. A harness may keep its history in a form engram cannot parse — protocol buffers, editor workspace state — while still offering its own export command. And an archive is not a live session: `save-chat` wrote `.texi` files engram had no way to read back, so engram's own output was the one format it could not ingest. `import` is deliberately not a `ReaderKind`. That enum is built around "one path per installed harness", discovered from a HarnessSpec and a working directory, and a file somebody hands you fits none of it. It feeds the shared pipeline directly instead: normalize_text, redact::scrub, ingest_turns. Identity is the message's content, not its file — a v5 uuid over scope, agent, role, timestamp and text. Every dedupe requirement then falls out of INSERT OR IGNORE with no separate pass: re-importing inserts 0, thirteen byte-identical copies of one archive import once, and the archive holding two concatenated copies of itself — residue of an earlier save-chat that appended rather than rewrote — collapses to one set of messages. Formats are sniffed by content, never by extension: a real chat/ directory held a standalone HTML palette editor next to genuine archives, and several source trees merely named `chat` are skipped the same way. Everything unrecognised is reported with its reason, which is the difference between "found nothing" and "could not read nine files". Three findings worth recording because each cost a debugging pass: A structural line is never message content — escaping doubles every literal `@`, so a single leading `@` is markup. Without that filter the trailing `@c Signed by:` and `@chapter Chat history…` of one document became the tail of the previous message's body, which is exactly why the double-appended archive did not deduplicate at first: the last message of each copy differed by the next copy's header. `--input-format`, not `--format`: the global output-format flag already owns that name, and a subcommand reusing it makes clap panic on the duplicate argument id rather than shadow it. The base directory is canonicalized before scope resolution. `import ./chat` resolves its parent to `.`, whose basename is empty, and the scope silently became `default` — one relative path away from filing a corpus under the wrong name. Missing timestamps are synthesised in order from a file-level anchor and flagged per file, never taken from the clock: stamping now() would collapse a conversation into one instant and destroy reading order, the same failure `transcript` refuses a wall-clock fallback to avoid. Verified against the real corpus: 100 files, 9,907 messages, 15 scopes. Gates: fmt, clippy -D warnings, 278 tests, REUSE 3.3, makeinfo clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016i16R4GhdSffsboRYq97Fs
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8ce264c362
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "engram-import:{scope}:{}:{}:{}:{}", | ||
| m.agent, m.role, m.created_at, m.text | ||
| ) |
There was a problem hiding this comment.
For Markdown exports without a Created header and for Claude scrollback, created_at comes from the file's mtime, and including it here makes the UUID change whenever an identical file is copied without preserving metadata or merely touched. Re-importing that byte-identical transcript then inserts every message again instead of deduplicating it, so approximate messages need identity based on stable export content and position rather than the mtime-derived timestamp.
AGENTS.md reference: AGENTS.md:L275-L275
Useful? React with 👍 / 👎.
| let base = path.parent().unwrap_or(&path); | ||
| let base = std::fs::canonicalize(base).unwrap_or_else(|_| base.to_path_buf()); | ||
| let resolved = rules::resolve_scope_in(scope.as_deref(), &base); |
There was a problem hiding this comment.
When the default save-chat output /work/foo/chat/archive.texi belongs to a non-Git project and --scope is omitted, resolving from the file's immediate parent makes the cwd fallback produce scope chat, not foo; a save/import round trip therefore silently files the history under the wrong scope. The base needs to reflect the containing project, including the parent of the managed chat/ directory when no Git root exists.
AGENTS.md reference: AGENTS.md:L85-L90
Useful? React with 👍 / 👎.
| if text.contains("**Session ID:**") | ||
| || text | ||
| .lines() | ||
| .any(|l| l.starts_with("## Assistant (") || l.trim_end() == "## Assistant") |
There was a problem hiding this comment.
With --recursive, any ordinary Markdown document containing a line exactly equal to ## Assistant or beginning ## Assistant ( is classified as an Opencode transcript even when it has no session header, and all following text is stored as an assistant memory. This can contaminate scopes when source trees contain prompt documentation, so automatic detection should require the documented session marker or another combination specific to an actual export.
AGENTS.md reference: AGENTS.md:L276-L276
Useful? React with 👍 / 👎.
| let Ok(entries) = std::fs::read_dir(&p) else { | ||
| continue; |
There was a problem hiding this comment.
If a requested path is misspelled, missing, a broken symlink, or an unreadable directory, is_file() is false and the read_dir error is silently discarded here, so engram import missing-path exits successfully with empty files and skipped arrays. This makes scripts treat a failed import as a successful no-op; preserve these traversal failures as skipped entries or return a structured error.
Useful? React with 👍 / 👎.
| path: String, | ||
| format: &'static str, | ||
| scope: String, | ||
| messages: usize, | ||
| inserted: usize, | ||
| skipped_existing: usize, |
There was a problem hiding this comment.
Each file resolves a ResolvedScope, but the response retains only its name, so callers cannot tell whether an explicit value, ENGRAM_SCOPE, a Git root, or the directory fallback selected the destination. This is particularly important for multi-project imports where an environment override can collapse all files into one scope, and it violates the response contract that scope resolution reports scope_origin; add the origin to each imported-file result.
AGENTS.md reference: AGENTS.md:L85-L87
Useful? React with 👍 / 👎.
| /// Import chat transcripts that were exported to files. | ||
| /// | ||
| /// Reads what a harness's own export command wrote, and what `save-chat` | ||
| /// archived, for the harnesses engram has no reader for. | ||
| Import { |
There was a problem hiding this comment.
Adding this CLI command without updating Command::Describe leaves import absent from both the manifest's commands list and its mcp.cli_only list. Clients using the JSON capability manifest for discovery will conclude that this command does not exist, so the new surface must be added to that manifest alongside the other CLI-only commands.
AGENTS.md reference: AGENTS.md:L38-L38
Useful? React with 👍 / 👎.
1 participant
ingestreads the session file a harness writes for itself — which reaches a harness engram has a reader for, and nothing else. Two gaps followed:save-chatwrote.texifiles engram had no way to read back — engram's own output was the one format it could not ingest.Not a
ReaderKind, on purposeThat enum is built around "one path per installed harness", discovered from a
HarnessSpecplus a working directory. A file somebody hands you fits none of it.importfeeds the shared pipeline directly —normalize_text→redact::scrub→ingest_turns— so an archive gets the same normalization and redaction a live transcript does. An archive is not more trustworthy for being old.Identity is the content, not the file
import_idis a v5 uuid over(scope, agent, role, created_at, text). Every dedupe requirement then falls out ofINSERT OR IGNORE, with no separate dedupe pass:That last one is real — residue of an earlier
save-chatthat appended rather than rewrote, still on disk.Three findings, each worth a debugging pass
A structural line is never message content. Escaping doubles every literal
@, so a single leading@is markup. Without that filter, the trailing@c Signed by:/@chapter Chat history…of one document became the tail of the previous message's body — which is exactly why the double-appended archive failed to deduplicate at first: the last message of each copy differed by the next copy's header. Caught because the numbers saidcollapsed=1where 2 was expected.--input-format, not--format. The global output-format flag already owns that name, and a subcommand reusing it makes clap panic rather than shadow it:Canonicalize before resolving scope.
import ./chatresolves its parent to., whose basename is empty — the scope silently becamedefault. One relative path away from filing an entire corpus under the wrong name.Formats — sniffed by content, never extension
@chaptercurrent,@sectionlegacy)^## (User|Assistant\b)--input-formatgatedA real
chat/directory held a standalone HTML palette editor next to genuine archives; extension-matching would have ingested it. Several vendored source trees merely namedchat(Kotlin, TypeScript, Rust) are skipped the same way. Everything unrecognised is reported with its reason — the difference between "found nothing" and "could not read nine files".Anchoring on the two literal speakers matters: the 294 KB Opencode export contains
## Root Cause Analysis,## Next Moveinside assistant replies. A naive^##split shatters one message into a dozen fake turns.Roles are not binary
The corpus holds
assistant6,395 ·user632 ·note136. The third is carried through, not coerced into one of the other two.Timestamps
Markdown and scrollback record none. Each message is offset one second from a file-level anchor — the export's own session time, else the file's mtime — and
approximate_timesis reported per file. Ordering is exact; absolute values are approximate. Stampingnow()would collapse a conversation into one instant and destroy reading order, the same failuretranscriptrefuses a wall-clock fallback to avoid.Verified against the real corpus
Two variants remain unhandled and are reported as skips, not silently dropped: older Claude Code scrollback with no
❯marker (forcing the format recovers its●assistant turns), and Goose terminal output.Gates
fmt·clippy -D warnings· 278 tests · REUSE 3.3 ·makeinfoclean🤖 Generated with Claude Code
https://claude.ai/code/session_016i16R4GhdSffsboRYq97Fs