From 75f770a11d709e711c46ca72ce7b6bbdcdf965c3 Mon Sep 17 00:00:00 2001 From: Carr1005 Date: Tue, 21 Jul 2026 11:44:06 +0800 Subject: [PATCH] agent-memory spec: remove tool-exposure & the three contradicted rows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-2 review: the tool-exposure row (old D11) presented semantic tool retrieval as an optional decision with 'pass all tools' as an equal option, when the course teaches it as best practice; and the three contradicted rows (distance strategy, index type, toolbox k) were low-stakes levers surfaced only because the course contradicted itself. Removes all four Ledger rows and renumbers to a contiguous D1-D15. No behavior lost, only relocated: - semantic retrieval stays baked into R4 (+AC4); toolbox store still D7 - distance strategy (cosine) & index (exact at fixture scale) -> §2, with the consistency invariant kept in R15/AC17 and exact-top-k in AC3 - toolbox k=5 -> R4/§4; single-config-point kept in R15/AC17 - the contradictions themselves stay as CTX-C4/C5/C6 provenance This spec now has zero contradicted rows. DEVIATION: this hand-edit runs ahead of the generation guide, which still promotes contradicted decisions to rows. Regenerating before the guide issue lands will re-introduce these rows -- do not regenerate until then. Tracked in a follow-up issue. Co-Authored-By: Claude Fable 5 --- .../spec.md | 92 +++++++++---------- 1 file changed, 45 insertions(+), 47 deletions(-) diff --git a/courses/agent-memory-building-memory-aware-agents/spec.md b/courses/agent-memory-building-memory-aware-agents/spec.md index 5294b9b..a8ba157 100644 --- a/courses/agent-memory-building-memory-aware-agents/spec.md +++ b/courses/agent-memory-building-memory-aware-agents/spec.md @@ -56,25 +56,21 @@ when you have reason to prefer another option. | # | Category | Decision | Invariant (must hold) | Default (course-derived) | Options | Trade-off | Owner | |---|----------|----------|-----------------------|--------------------------|---------|-----------|-------| -| D1 | learner | `[project]` — what the agent is for. Default is the **course's example realization** (its running demo scenario), expected to be swapped when your project differs — the invariants, not the example, are what must survive. | — | A **memory-aware agentic research assistant**: a CLI/REPL chat agent that finds, saves, and discusses research papers across sessions, remembering prior findings and preferences (§3 project precedence, branch 1: the course's example scenario shape re-expressed on the synthetic fixture corpus of §5). | Course example shape: research assistant (course default); any domain assistant needing persistence (support bot, coding copilot, ops runbook agent) | Swapping the project keeps every memory mechanism; only the toolset (D16) and seed data (D2) change with it. | learner | -| D2 | learner | `[data/inputs]` — what the agent operates on. Default is a fixture stand-in for the **course's example data**; swap when D1 changes. | — | The **synthetic fixture corpus of §5** (two authored paper-notes files + one authored 12-message seed conversation), ingested at setup. | Fixture corpus (default); the learner's own documents/threads; the course streamed 100 records of a public arXiv dataset and live arXiv PDFs (course default) | Real corpora need the same ingest path (embed + metadata); large corpora may need an ANN index (see D18). | learner | +| D1 | learner | `[project]` — what the agent is for. Default is the **course's example realization** (its running demo scenario), expected to be swapped when your project differs — the invariants, not the example, are what must survive. | — | A **memory-aware agentic research assistant**: a CLI/REPL chat agent that finds, saves, and discusses research papers across sessions, remembering prior findings and preferences (§3 project precedence, branch 1: the course's example scenario shape re-expressed on the synthetic fixture corpus of §5). | Course example shape: research assistant (course default); any domain assistant needing persistence (support bot, coding copilot, ops runbook agent) | Swapping the project keeps every memory mechanism; only the toolset (D15) and seed data (D2) change with it. | learner | +| D2 | learner | `[data/inputs]` — what the agent operates on. Default is a fixture stand-in for the **course's example data**; swap when D1 changes. | — | The **synthetic fixture corpus of §5** (two authored paper-notes files + one authored 12-message seed conversation), ingested at setup. | Fixture corpus (default); the learner's own documents/threads; the course streamed 100 records of a public arXiv dataset and live arXiv PDFs (course default) | Real corpora need the same ingest path (embed + metadata); large corpora may need an ANN index (see §2). | learner | | D3 | learner | `[goal]` — what "working" means. | — | **Cross-session continuity**: the agent answers follow-ups without re-doing discovery, keeps context under budget via recoverable summarization, and can still recover the thread's first question after summarizing (the §5 oracle). | Continuity guarantee (default); latency/cost targets; retrieval-precision targets | Stricter retrieval goals push toward reranking/hybrid search — out of course scope (§1 Not Included). | learner | | D4 | learner | `[model/provider]` — the LLM behind reasoning, summarization, augmentation, extraction. | Must support **native (OpenAI-style) tool calling** and a system role; its context-window size must be known to the budget monitor (R6). | **OpenAI**, per the materials: `gpt-5-mini` for the agent loop, `gpt-5` for docstring augmentation and entity extraction. Requires `OPENAI_API_KEY` (§2); LLM-dependent ACs are tagged `scripted` or `live` so the offline oracle still runs (§5). *Note (course-contradicted, carried here):* the L4 notebook's token-limit map lists only `gpt-5-mini: 256000` while `helper.py`'s lists only `gpt-5: 256000`, and helper function defaults say `gpt-5` where the L5 loop calls `gpt-5-mini` — the materials never reconcile the two; this spec pins the loop to `gpt-5-mini` with a 256,000-token budget (the L5 loop's actual call path). | OpenAI gpt-5-mini + gpt-5 (course default); any tool-calling model (Anthropic, local via an OpenAI-compatible server) | Changing provider means re-checking the tool-call message shapes and the token budget in R6; prompts (summarization, extraction) may need re-tuning. | learner | -| D5 | learner | `[environment]` — where it runs. | **All seven memory stores must survive process restarts** (memory is external to the model and persistent). | Local machine, Python 3.11+, single on-disk database file (realization: D15). First run needs network to download the embedding model; no Docker, no admin setup. | Local single file (default); the course ran a Dockerized Oracle instance (course default — see D15); any server environment | A shared/server environment adds connection management the course's local pattern doesn't cover. | learner | +| D5 | learner | `[environment]` — where it runs. | **All seven memory stores must survive process restarts** (memory is external to the model and persistent). | Local machine, Python 3.11+, single on-disk database file (realization: D14). First run needs network to download the embedding model; no Docker, no admin setup. | Local single file (default); the course ran a Dockerized Oracle instance (course default — see D14); any server environment | A shared/server environment adds connection management the course's local pattern doesn't cover. | learner | | D6 | learner | `[out-of-scope]` — anything explicitly unwanted. | — | Nothing beyond §1 "Not Included". | Add exclusions freely | Exclusions only shrink scope; they never relax the D5/D7 invariants. | learner | | D7 | design-argued | **Memory-core topology** — which memory types exist and their storage class. | Seven memory types, each with a dedicated store: **conversational + tool-log memory retrievable by exact key (thread) in chronological order; knowledge-base, workflow, toolbox, entity, and summary memory retrievable by semantic similarity.** | The course's seven stores: `CONVERSATIONAL_MEMORY` and `TOOL_LOG_MEMORY` as SQL-style tables; `SEMANTIC_MEMORY` (knowledge base), `WORKFLOW_MEMORY`, `TOOLBOX_MEMORY`, `ENTITY_MEMORY`, `SUMMARY_MEMORY` as vector stores — all fronted by one memory-manager abstraction (CTX-A, CTX-C1). | Seven-store split (course default); fewer stores (e.g. fold entity into KB); more (add semantic cache) | Lesson 3 argues the split: conversation needs *exact* retrieval by thread id, "not similarity search", while knowledge/workflow/toolbox/entity/summary need meaning-based lookup; collapsing stores loses the retrieval strategy matched to each type. | course | | D8 | design-structural | **Deterministic vs agent-triggered operation split** — which memory operations the harness runs every turn vs which the model may invoke. | Context-building reads (conversation, KB, workflow, entity, summary-index) and post-turn writes (conversation, workflow) run **deterministically every turn**; judgment operations (expand a summary, summarize-and-store, external search, entity writes) are **also exposed as model-invocable tools**. | The course's classification (CTX-C2): deterministic = the five preload reads + conversation/workflow writes + the >80% budget check; agent-triggered = `expand_summary`, `summarize_and_store`, external search, deep paper ingest; `read_toolbox` is both. *Note (course-contradicted, carried here):* the Lesson 3/L2-notebook classification table marks entity **writes** and `read_summary_context` agent-triggered, yet the Lesson 6 loop runs both deterministically (entity extraction after query and answer; summary-index read in every preload) — this spec follows the Lesson 6 working loop. | Course split (course default); fully agent-managed memory; fully hardcoded memory | Moving reads/writes to model discretion risks "forgot to save" gaps and the chicken-and-egg problem (CTX-B7); hardcoding judgment ops (e.g. always summarize) wastes tokens and clutters memory (CTX-C2). | course | | D9 | design-argued | **Partitioned context window + memory-aware system prompt** — how assembled memory is presented to the model. | The model input carries the question first, then **one labeled segment per memory type**, and the model instructions name each segment, its usage guidance, and a conflict-priority order (R16). | Markdown-heading partition in fixed order: `# Question`, then `## Conversation Memory`, `## Knowledge Base Memory`, `## Workflow Memory`, `## Entity Memory`, `## Summary Memory`, each segment self-describing ("what this memory is / how you should leverage it"). | Partitioned + self-describing (course default); one mixed context block; structured JSON context | Lesson 6 argues markdown headings let the model exploit its latent grasp of hierarchical structure, giving "structured, role-specific context instead of one mixed block"; a mixed block loses per-store semantics. | course | | D10 | design-argued | **Context-window reduction strategy** — what happens when the context outgrows its budget. | Reduction must be **recoverable**: whatever is compressed out of the live context stays retrievable in full from the store via an id-addressable link. | The course's combined pipeline: threshold-triggered (deterministic, >80% of budget) and tool-triggered (agent-invoked) **summarization with write-back links** — summarize unsummarized thread messages, store the summary, mark the exact source rows with the `summary_id`, keep only a summary reference in context, and expand on demand (JIT) via the expand tool (CTX-C3). | Recoverable summarize-and-link (course default); pure lossy summarization; pure compaction (offload raw content by id, no summary) | Lesson 5 argues the trade-off aloud: summarization "will always be a lossy technique", while compaction keeps everything but preloads nothing useful — the shipped design pairs a lossy summary for the live window with a lossless database path back to the originals. | course | -| D11 | design-argued | **Tool exposure: semantic tool retrieval (Toolbox pattern)** — how tools reach the model. | The model receives only a **query-relevant subset** of the registered tools each turn, retrieved by semantic similarity over tool descriptions; the registry itself may hold many more tools than are ever passed at once. | Register every tool (description + embedding) in the toolbox store; at each turn retrieve the top matches for the user query (count: pinned in D19) and pass only those schemas to the model. | Semantic retrieval (course default); pass all tools every turn; hand-picked static toolset per app | Lesson 4 argues passing all tools causes context bloat, tool-selection degradation, latency and cost growth (providers recommend ~10–20 tools max); retrieval scales to hundreds of tools but adds an embedding lookup per turn and can miss a relevant tool if descriptions are weak (mitigated by D12). | course | -| D12 | design-argued | **Tool-description augmentation** — what text gets embedded for each tool. | The text embedded for retrieval must be **rich enough to separate tools semantically** (retrieval is keyed on descriptions, not names). | Augmented registration as the course does for most tools: an LLM rewrites the docstring using the function's source, plus ~5 synthetic example queries; name + augmented description + signature + queries form the embedding text. Registration is idempotent per tool name (R5). | LLM-augmented descriptions (course default); raw docstrings only (the course registered its arXiv candidate-search tool unaugmented) | Lesson 4 argues augmentation buys higher separability and recall in the embedding space at the cost of one LLM call per registration; weak one-line docstrings retrieve poorly. | course+learner | -| D13 | design-argued | **Tool-result flow: full log + bounded excerpt** — what the model sees of a tool's output. | Every tool execution is **fully persisted** (args, complete result, status, errors) outside the context window; the model receives only a **bounded excerpt with an id pointer** back to the full record. | Persist every call to the tool-log store; pass at most 3,000 characters of the result to the model, appending a truncation notice naming the log id when cut (R10). | Log + bounded excerpt (course default); pass full outputs into context; discard raw outputs | Lesson 6 calls this context offloading — "move large payload handling out of the model context and into memory infrastructure"; skipping the log loses the audit trail and the JIT retrieval path, while full outputs blow up the window (CTX-B6). | course | -| D14 | design-argued | **Search-and-store acquisition** — what acquisition tools do with what they find. | Any tool that acquires external content **persists its results to knowledge-base memory with source metadata in the same call**, before/independent of what the model does with them. | Course pattern: search/fetch tools write each result (or chunk) into the knowledge base with source metadata (title, source id, timestamps, chunk indices), so information discovered once is retrievable in later turns without re-searching. | Search-and-store (course default); return-only tools (results live and die in one turn) | Lesson 4 argues this is how the agent "learns from its searches" — repeat questions stop costing API calls; the cost is knowledge-base growth and possible staleness of stored search results. | course | -| D15 | realization | **Persistent store (the Agent Memory Core)** — the database behind all seven stores. | **One persistent database is the memory core**: it serves both the exact-key/chronological stores and the semantic-similarity stores (topology: D7), and survives restarts (D5). | **SQLite** (single on-disk file, Python stdlib driver): SQL tables for conversational + tool-log memory; the five vector stores as tables holding text, JSON metadata, and the embedding, searched by exact similarity at fixture scale (index: D18; distance: D17). *§3 dependency precedence, branch 1* — the course's Oracle AI Database 26ai runs in Docker with admin/tablespace setup and a dedicated DB user, i.e. container-level setup (heavy); the taught subject is the memory pattern, which Oracle realizes but does not define, so the lightest self-contained realization is substituted. | Oracle AI Database 26ai via `langchain-oracledb`/OracleVS, Dockerized, with admin bootstrap and hybrid-search capability (course default); SQLite single-file (default); Postgres + pgvector; any DB offering both exact and vector retrieval | Switching stores means migrating schemas and re-embedding nothing (embeddings are store-agnostic) but re-implementing the store adapters; Oracle restores the course's exact stack incl. its vector indexes and hybrid search, at the cost of Docker + admin setup. | course+learner | -| D16 | realization | **External acquisition toolset** — which live tools the default agent registers. | At least one **agent-triggered external acquisition tool** exists and follows search-and-store (D14). | The course's **keyless arXiv tools only**: a candidate-search tool returning structured JSON (id, title, authors, published, abstract) and a deep-ingest tool (fetch PDF → text → chunk → store to KB), plus a local current-time utility and the summary tools (expand, summarize-and-store) and self-lookup (`read_toolbox`). *§3 dependency precedence, branch 1* — the course's Tavily web search needs an API key (heavy); arXiv access is keyless and setup-free, so it stays (its ACs are tagged `live`), while Tavily becomes an option. These are the **course's example tools** — swap them when `[project]` (D1) differs. | Course toolset incl. Tavily web search with `TAVILY_API_KEY` (course default); keyless arXiv-only toolset (default); the learner's own domain tools | Dropping Tavily loses general web search (the agent only acquires from arXiv); adding it back is one keyed client + one registered tool following D14. | course+learner | -| D17 | contradicted | **Vector distance strategy.** | One distance strategy is used **consistently across all five vector stores, for both write-time indexing and read-time search** (R15). | **Cosine** — the strategy the course used when the stores were introduced (L2 and L3 notebooks' store-manager instantiation) and the fallback its index helper assumes. *Contradiction (why this is a row):* the L4 and L5 notebooks instantiate the same stores with **Euclidean** distance, and the L2 notebook's own markdown names Euclidean while its code passes cosine; the course wipes all tables between lessons precisely to keep the strategy consistent (CTX-C5). | Cosine (default; one of two course-used values); Euclidean (course default in the final two lessons); dot product | With normalized sentence-transformer embeddings the rankings are near-equivalent; mixing strategies between write and read silently corrupts rankings — hence the wipe-and-rebuild warning the course ships. | course+learner | -| D18 | contradicted | **Vector index type.** | Similarity queries must return the true top-k at fixture scale — retrieval correctness must not depend on approximate-index tuning. | **Exact (brute-force) similarity search** — at the fixture corpus scale no ANN index is needed, and the SQLite default (D15) has none. *Contradiction (why this is a row):* the course's markdown and lesson objectives say it builds **HNSW** indexes, but its helper code deliberately creates **IVF** (`NEIGHBOR PARTITIONS`, target accuracy 95) instead, citing Oracle-version bugs with HNSW (CTX-C6). | Exact search (default); IVF neighbor-partitions index (course default — what the course's helper actually created); HNSW (named in course text, not what its code ran) | Exact search is O(n) per query — fine for fixtures, add an ANN index when corpora grow (re-check recall); ANN indexes trade exactness for speed and need distance-consistent creation (D17). | course+learner | -| D19 | contradicted | **Toolbox retrieval k** — how many tool schemas are passed per turn. | k is pinned in exactly one configuration point (R15) and small enough to keep the toolset focused (D11). | **k = 5** — what the course's Lesson 6 agent loop actually calls. *Contradiction (why this is a row):* the course's registered `read_toolbox` tool declares `k: int = 3` in its signature while its own docstring says "default: 5", and the loop calls `k=5`. | 5 (default; course's call-site value); 3 (course's signature default); any small integer | Larger k re-approaches the all-tools failure Lesson 4 warns about (selection degradation, bloat); smaller k risks missing the right tool for compound queries. | course+learner | +| D11 | design-argued | **Tool-description augmentation** — what text gets embedded for each tool. | The text embedded for retrieval must be **rich enough to separate tools semantically** (retrieval is keyed on descriptions, not names). | Augmented registration as the course does for most tools: an LLM rewrites the docstring using the function's source, plus ~5 synthetic example queries; name + augmented description + signature + queries form the embedding text. Registration is idempotent per tool name (R5). | LLM-augmented descriptions (course default); raw docstrings only (the course registered its arXiv candidate-search tool unaugmented) | Lesson 4 argues augmentation buys higher separability and recall in the embedding space at the cost of one LLM call per registration; weak one-line docstrings retrieve poorly. | course+learner | +| D12 | design-argued | **Tool-result flow: full log + bounded excerpt** — what the model sees of a tool's output. | Every tool execution is **fully persisted** (args, complete result, status, errors) outside the context window; the model receives only a **bounded excerpt with an id pointer** back to the full record. | Persist every call to the tool-log store; pass at most 3,000 characters of the result to the model, appending a truncation notice naming the log id when cut (R10). | Log + bounded excerpt (course default); pass full outputs into context; discard raw outputs | Lesson 6 calls this context offloading — "move large payload handling out of the model context and into memory infrastructure"; skipping the log loses the audit trail and the JIT retrieval path, while full outputs blow up the window (CTX-B6). | course | +| D13 | design-argued | **Search-and-store acquisition** — what acquisition tools do with what they find. | Any tool that acquires external content **persists its results to knowledge-base memory with source metadata in the same call**, before/independent of what the model does with them. | Course pattern: search/fetch tools write each result (or chunk) into the knowledge base with source metadata (title, source id, timestamps, chunk indices), so information discovered once is retrievable in later turns without re-searching. | Search-and-store (course default); return-only tools (results live and die in one turn) | Lesson 4 argues this is how the agent "learns from its searches" — repeat questions stop costing API calls; the cost is knowledge-base growth and possible staleness of stored search results. | course | +| D14 | realization | **Persistent store (the Agent Memory Core)** — the database behind all seven stores. | **One persistent database is the memory core**: it serves both the exact-key/chronological stores and the semantic-similarity stores (topology: D7), and survives restarts (D5). | **SQLite** (single on-disk file, Python stdlib driver): SQL tables for conversational + tool-log memory; the five vector stores as tables holding text, JSON metadata, and the embedding, searched by exact similarity at fixture scale (indexing & distance strategy: §2). *§3 dependency precedence, branch 1* — the course's Oracle AI Database 26ai runs in Docker with admin/tablespace setup and a dedicated DB user, i.e. container-level setup (heavy); the taught subject is the memory pattern, which Oracle realizes but does not define, so the lightest self-contained realization is substituted. | Oracle AI Database 26ai via `langchain-oracledb`/OracleVS, Dockerized, with admin bootstrap and hybrid-search capability (course default); SQLite single-file (default); Postgres + pgvector; any DB offering both exact and vector retrieval | Switching stores means migrating schemas and re-embedding nothing (embeddings are store-agnostic) but re-implementing the store adapters; Oracle restores the course's exact stack incl. its vector indexes and hybrid search, at the cost of Docker + admin setup. | course+learner | +| D15 | realization | **External acquisition toolset** — which live tools the default agent registers. | At least one **agent-triggered external acquisition tool** exists and follows search-and-store (D13). | The course's **keyless arXiv tools only**: a candidate-search tool returning structured JSON (id, title, authors, published, abstract) and a deep-ingest tool (fetch PDF → text → chunk → store to KB), plus a local current-time utility and the summary tools (expand, summarize-and-store) and self-lookup (`read_toolbox`). *§3 dependency precedence, branch 1* — the course's Tavily web search needs an API key (heavy); arXiv access is keyless and setup-free, so it stays (its ACs are tagged `live`), while Tavily becomes an option. These are the **course's example tools** — swap them when `[project]` (D1) differs. | Course toolset incl. Tavily web search with `TAVILY_API_KEY` (course default); keyless arXiv-only toolset (default); the learner's own domain tools | Dropping Tavily loses general web search (the agent only acquires from arXiv); adding it back is one keyed client + one registered tool following D13. | course+learner | ## 1. Objective @@ -94,7 +90,7 @@ resolve without repeating discovery and long threads never overflow the model's - **Reranking models, graph-traversal retrieval, memory decay/forgetting policies, and embedding-model fine-tuning** — mentioned as concepts in the lessons, never implemented. - **Oracle-specific operations** (tablespace admin, DB user provisioning, vector-memory pools) — - they belong to the course's store realization, not the pattern (D15). + they belong to the course's store realization, not the pattern (D14). - **Integration into an existing codebase** — this spec targets the standalone takeaway only; the Ledger's Invariant column is written to serve as the future integration contract. - `[out-of-scope]` — learner exclusions (D6; default: none). @@ -104,13 +100,13 @@ resolve without repeating discovery and long threads never overflow the model's | Component | Pinned choice | Note | |---|---|---| | Language | Python 3.11+ | Course-era Python version is not stated in the materials. | -| Persistent store | SQLite (stdlib `sqlite3`), one on-disk file | Decision Ledger **D15** — learners change it there, not here. | -| Vector search | Exact similarity over embeddings stored in the DB | Ledger **D18**; distance per **D17**. | +| Persistent store | SQLite (stdlib `sqlite3`), one on-disk file | Decision Ledger **D14** — learners change it there, not here. | +| Vector search | Exact (brute-force) similarity at fixture scale — add an ANN index only when corpora grow | One distance strategy — **cosine** — used consistently across all stores, write & read (R15). Both are tunable levers, not decisions: the course used both cosine and euclidean across lessons (tables wiped between them; near-equivalent for normalized embeddings) and built its ANN index (IVF/HNSW) only for the cloud store. | | Embeddings | `sentence-transformers/paraphrase-mpnet-base-v2` (768-dim, local CPU) | The course's single embedding model for all stores and both write/read paths (CTX-C5). First run downloads it (keyless network). | | LLM | OpenAI SDK (current stable) | Ledger **D4**. `OPENAI_API_KEY` required for scripted-interface parity and live ACs. | -| arXiv access | `arxiv` + `pymupdf` (current stable) | Ledger **D16**. The course reached arXiv through LangChain community wrappers — those names are perishable, see CTX-D. | +| arXiv access | `arxiv` + `pymupdf` (current stable) | Ledger **D15**. The course reached arXiv through LangChain community wrappers — those names are perishable, see CTX-D. | | Dependency pins | Pin exact versions **at build time** to current stable in a lockfile you generate | **Honest era note: the course's `requirements.txt` installs everything unpinned** (no versions at all, LangChain-family + `oracledb` + `openai` + `tavily-python` era); do not invent course pins. Era-specific import names → CTX-D. | -| Secrets | `.env` file loaded at startup; `OPENAI_API_KEY` (required), `TAVILY_API_KEY` (only if D16 is switched) | Never hardcoded and never committed. (The course notebooks hardcode local DB credentials — do not reproduce that.) | +| Secrets | `.env` file loaded at startup; `OPENAI_API_KEY` (required), `TAVILY_API_KEY` (only if D15 is switched) | Never hardcoded and never committed. (The course notebooks hardcode local DB credentials — do not reproduce that.) | Default store names (single-valued course constants; body defaults, not Ledger rows): `CONVERSATIONAL_MEMORY`, `SEMANTIC_MEMORY` (knowledge base), `WORKFLOW_MEMORY`, @@ -229,7 +225,7 @@ implementations may add fields but MUST NOT violate these. ``` Toolbox retrieval returns an array of `RetrievedToolSchema` with **unique** `function.name` -values (dedup rule R4). Retrieved tool count per turn: pinned in Ledger **D19**. +values (dedup rule R4). Retrieved tool count per turn: a single config value, **k=5** (R4). ## 4. Business Rules @@ -249,10 +245,12 @@ hardening (added by this spec; the course did not demonstrate it — said so exp `summary_id` is set; when nothing remains, the segment states that explicitly (prevents CTX-B5). → AC2, AC8 4. **R4 — Focused tool retrieval.** [C] Each turn passes the model only the toolbox's semantic - top matches for the query (count per D19), deduplicated by tool name, as OpenAI - function-format schemas (§3) (prevents CTX-B2). → AC4, AC17 + top matches for the query (top-k, **k=5** — a single config value), deduplicated by tool name, + as OpenAI function-format schemas (§3) (prevents CTX-B2). Semantic retrieval is the baked-in + approach the course teaches for scaling tool use; for a tiny static toolset it simply returns + all tools, so it is safe regardless of tool count. → AC4, AC17 5. **R5 — Idempotent, enriched tool registration.** [C] Registering a tool stores its - description + embedding in the toolbox store; with augmentation enabled (D12) the stored + description + embedding in the toolbox store; with augmentation enabled (D11) the stored description is LLM-enriched from docstring + source and ~5 synthetic queries join the embedding text; re-registering an existing tool name never writes a duplicate row. → AC5, AC18 6. **R6 — Context budget monitoring.** [C] Token usage is estimated as `len(chars) // 4` @@ -293,9 +291,9 @@ hardening (added by this spec; the course did not demonstrate it — said so exp knowledge base with source metadata (source, id, title, chunk index/count, timestamps) in the same call; deep ingestion chunks documents (default: recursive character splitting, chunk size 1,500, overlap 200 — the course's single ingest config) before storing. → AC16, AC19 -15. **R15 — Configuration coherence.** [C] One distance strategy (D17) governs every vector +15. **R15 — Configuration coherence.** [C] One distance strategy (cosine; §2) governs every vector store, write and read (the course wipes and rebuilds all tables between lessons specifically - to guarantee this); [H] distance strategy, toolbox k (D19), and token budget are each defined + to guarantee this); [H] the distance strategy, toolbox k, and token budget are each defined in exactly one configuration point. → AC17 16. **R16 — Memory-aware model instructions.** [C] The system prompt names each context segment and its purpose, instructs the model to consult memory before tools, sets the conflict @@ -324,25 +322,25 @@ Modes: **Offline** (no network after the one-time embedding-model download), **S | # | Mode | Given | When | Then | |---|---|---|---|---| -| AC1 | Offline | Fixtures ingested and seed thread written by process 1 | A **new OS process** opens the same DB file and reads seed-thread conversation and queries the KB for "streaming memory consolidation" | All 12 seed messages return chronologically, and a `kestrel-notes` chunk is the top KB hit — memory survived restart (D5/D15 invariants; R1) | +| AC1 | Offline | Fixtures ingested and seed thread written by process 1 | A **new OS process** opens the same DB file and reads seed-thread conversation and queries the KB for "streaming memory consolidation" | All 12 seed messages return chronologically, and a `kestrel-notes` chunk is the top KB hit — memory survived restart (D5/D14 invariants; R1) | | AC2 | Offline | Seed thread + 2 messages written to thread `other-01` | Reading conversational memory for `seed-01` with default limit | Only `seed-01` unconsolidated messages return, timestamp-ascending, each with role and timestamp; no `other-01` content (R3) | -| AC3 | Offline | Both fixture notes ingested | KB similarity query "entity graphs for routing tools", k=3 | A `heron-notes` chunk ranks first — the true nearest neighbor, exact at fixture scale — and carries its source-file metadata (R2; D7, D18 invariants) | -| AC4 | Offline | All 9 fixture tools registered | Toolbox query "find research papers on agent memory" with k per D19 | At most 5 schemas return, all valid `RetrievedToolSchema`, names unique, `paper_search` present (R4; D11/D19 invariants) | +| AC3 | Offline | Both fixture notes ingested | KB similarity query "entity graphs for routing tools", k=3 | A `heron-notes` chunk ranks first — the true nearest neighbor, exact at fixture scale — and carries its source-file metadata (R2; D7 invariant) | +| AC4 | Offline | All 9 fixture tools registered | Toolbox query "find research papers on agent memory" with k=5 | At most 5 schemas return, all valid `RetrievedToolSchema`, names unique, `paper_search` present (R4 invariant) | | AC5 | Offline | `paper_search` already registered | It is registered again under the same name | The toolbox store holds exactly one row for `paper_search` (R5) | | AC6 | Offline | A test budget of 1,000 tokens [H]; strings sized to 40% / 65% / 85% of it | Monitoring each | Statuses are `ok` / `warning` / `critical` respectively and each token estimate equals `len(chars)//4` (R6) | | AC7 | Scripted | Assembled context >80% of the test budget containing a `## Conversation Memory` section and a `# Question` line | The pre-model budget check runs | Returned context has the conversation section replaced by the offload stub, a `[Summary ID: …]` reference under `## Summary Memory`, and the `# Question` text byte-identical (R7; D10 invariant) | | AC8 | Scripted | Seed thread fully unconsolidated | Thread summarization runs, then immediately runs again | After run 1: every seed row carries the same new 8-char hex `summary_id`, unconsolidated count is 0, and the conversation read reports no unconsolidated messages; run 2 reports nothing to summarize (R3, R8, R9) | | AC9 | Scripted | AC8's summary id | The expand operation runs on it | Output contains the stored summary text AND all 12 originals with timestamps, chronological, including the first user message `"Find the Kestrel paper about streaming memory consolidation."` verbatim (R9; D10 invariant) | -| AC10 | Scripted | `fetch_notes` returns >3,000 chars inside an agent turn | The loop executes the tool call | A tool-log record holds the FULL result, args, `status="success"`, preview ≤2,000 UTF-8 bytes; the tool message given to the model is ≤3,000 chars + a truncation notice naming the log id (R10; D13 invariant) | +| AC10 | Scripted | `fetch_notes` returns >3,000 chars inside an agent turn | The loop executes the tool call | A tool-log record holds the FULL result, args, `status="success"`, preview ≤2,000 UTF-8 bytes; the tool message given to the model is ≤3,000 chars + a truncation notice naming the log id (R10; D12 invariant) | | AC11 | Scripted | A scripted run making exactly 2 tool calls then answering; plus one zero-step workflow row written directly as a control | The turn completes; then workflow memory is queried for a similar task | One workflow record exists with the query, 2 ordered steps with outcome markers, answer excerpt ≤200 chars, `num_steps=2`; the query returns it and never returns the zero-step control (R11; D8 invariant) | | AC12 | Scripted | A scripted model that always emits a tool call | `call_agent` runs a turn | The loop stops after exactly 10 iterations, `completed=false`, the final answer is the fixed inability message, and that answer is persisted to conversational memory (R13, R1) | | AC13 | Scripted | Any query on a fresh thread | The turn's exact model input is captured | The user content begins `# Question` and contains all five segment headings in order (Conversation, Knowledge Base, Workflow, Entity, Summary); the system message names all five segments and states the R16 conflict-priority order (R2, R16; D9 invariant) | | AC14 | Scripted | The seed transcript; scripted summary reply | Summarization runs | Output contains the four exact R8 headings; the id matches `^[0-9a-f]{8}$`; the label is 8–12 words and not in the generic-label reject set (R8) | | AC15 | Scripted | Scripted extractor returns 2 entities for the fixture text; a second scripted run throws | Entity extraction runs after a turn | Run 1 writes 2 `EntityRecord`s and the entity read returns both as formatted bullets; run 2's exception leaves the turn's final answer unchanged (R12) | -| AC16 | Offline | `fetch_notes` acquisition tool | It executes on `kestrel-notes` | KB gains chunk rows with `source`, `chunk_id`, `num_chunks` metadata **within the tool call itself**, and chunk sizes respect the 1,500/200 chunking default (R14; D14 invariant) | -| AC17 | Offline | The running configuration | All five vector stores and the toolbox retriever are inspected | Every store reports the identical distance strategy (D17) and the retrieval k resolves from the single D19 config point (R4, R15; D17/D19 invariants) | -| AC18 | Scripted | A tool registered with augmentation on; scripted LLM returns enriched text + 5 queries | Registration completes | The stored description equals the enriched text (not the raw docstring) and the embedded text contains the 5 synthetic queries (R5; D12 invariant) | -| AC19 | Live-keyless | Network available | The arXiv candidate-search tool runs with query "agent memory" | A JSON array returns; every element has `arxiv_id`, `entry_id`, `title`, `authors`, `published`, `abstract` (abstract ≤2,500 chars) (R14; D16 invariant) | +| AC16 | Offline | `fetch_notes` acquisition tool | It executes on `kestrel-notes` | KB gains chunk rows with `source`, `chunk_id`, `num_chunks` metadata **within the tool call itself**, and chunk sizes respect the 1,500/200 chunking default (R14; D13 invariant) | +| AC17 | Offline | The running configuration | All five vector stores and the toolbox retriever are inspected | Every store reports the identical distance strategy (§2/R15) and the retrieval k resolves from a single config point (R4, R15) | +| AC18 | Scripted | A tool registered with augmentation on; scripted LLM returns enriched text + 5 queries | Registration completes | The stored description equals the enriched text (not the raw docstring) and the embedded text contains the 5 synthetic queries (R5; D11 invariant) | +| AC19 | Live-keyless | Network available | The arXiv candidate-search tool runs with query "agent memory" | A JSON array returns; every element has `arxiv_id`, `entry_id`, `title`, `authors`, `published`, `abstract` (abstract ≤2,500 chars) (R14; D15 invariant) | | AC20 | Live-keyed | `OPENAI_API_KEY` set; fresh thread | Turns: "Find the MemGPT paper" → "Save the content of the paper" → "Summarize the conversation so far using your tool" → "What was my first question?" | Turn 2 resolves "the paper" from conversation memory without re-asking; turn 3 stores a summary and marks rows; turn 4's answer names the first question, having expanded the summary (R1, R2, R9, R16; D4 invariant — real native tool calling against the default provider) — the course's own end-to-end demonstration sequence (CTX-E, Lesson 6) | Every business rule reaches ≥1 AC and every AC traces to a rule (mapping inline above). Every @@ -358,10 +356,10 @@ demonstrated failure mode appears as rule + fixture + AC + CTX-B entry (§10 fou **Ask First** (course-spoken trade-offs — anything here forces rework or was shown lateral-or-worse) - **Changing the embedding model** — every vector store must be re-embedded; the course stresses the query-time model must match the ingest-time model (Lesson 2's RAG walkthrough). -- **Changing distance strategy or adding/altering a vector index after data exists** (D17/D18) — the course wipes and rebuilds all tables between lessons precisely to avoid inconsistent-strategy corruption. -- **Raising toolbox k or passing all tools to the model** (D11/D19) — Lesson 4 shows this degrades tool selection and bloats context; providers recommend ~10–20 tools max. +- **Changing distance strategy or adding/altering a vector index after data exists** (see §2) — the course wipes and rebuilds all tables between lessons precisely to avoid inconsistent-strategy corruption. +- **Raising the toolbox retrieval count or passing all tools to the model** (R4) — Lesson 4 shows this degrades tool selection and bloats context; providers recommend ~10–20 tools max. - **Rewriting the summarization prompt structure** (R8) — Lesson 5: summarization is lossy and "the prompting technique you use … will determine the quality of the output". -- **Switching the store realization** (D15) or **enabling Tavily web search** (D16) — migration/setup cost and a new secret respectively. +- **Switching the store realization** (D14) or **enabling Tavily web search** (D15) — migration/setup cost and a new secret respectively. **Never** - Invent citations, paper metadata, or provenance for content not present in memory. @@ -413,12 +411,12 @@ Stage by stage, each with the reason it exists: conversation into summaries that keep an id-link back to the originals, expandable just-in-time (D10). 5. **Semantic tool retrieval** — tools are procedural memory: register many, retrieve few per - query, so the system scales to hundreds of tools without degrading selection (D11, D12). + query, so the system scales to hundreds of tools without degrading selection (R4, D11). 6. **Logged execution with bounded excerpts** — full tool outputs live in the database; the - context window gets a pointer-bearing excerpt (D13). + context window gets a pointer-bearing excerpt (D12). 7. **Write-back learning** — conversations, workflows (steps + outcome), entities, and search results persist after each turn, so the agent improves across sessions instead of restarting - from scratch (D14). + from scratch (D13). ### CTX-B. Failure-mode catalog @@ -458,7 +456,7 @@ Stage by stage, each with the reason it exists: Reference only — decisions are made in the Decision Ledger, not here. -- **CTX-C1 — Why typed stores (→ Ledger D7, D15).** The course's taxonomy: short-term memory +- **CTX-C1 — Why typed stores (→ Ledger D7, D14).** The course's taxonomy: short-term memory (semantic cache, working memory/context window) vs long-term memory (procedural → workflow & toolbox; semantic → knowledge base, entity, summary; episodic → conversational, which is time-ordered and timestamp-addressable). Conversation and tool logs need exact, chronological, @@ -485,7 +483,7 @@ Reference only — decisions are made in the Decision Ledger, not here. chars; summary generation capped at 4,000 completion tokens; label prompt capped at 2,000; one retry then deterministic fallback; 8-char summary ids; budget estimate `chars//4`; thresholds 50/80; budget 256,000 tokens (see D4's note for the model-name discrepancy). -- **CTX-C4 — Toolbox mechanics (→ Ledger D11, D12, D19).** Tool retrieval is keyed on +- **CTX-C4 — Toolbox mechanics (→ Ledger D11; R4).** Tool retrieval is keyed on descriptions, not names: registration embeds name + description + signature (+ synthetic queries when augmented); a user query embeds to the same space; nearest tools win. Augmented registration has the LLM rewrite the docstring using the function source (summary, steps, @@ -494,20 +492,20 @@ Reference only — decisions are made in the Decision Ledger, not here. ~10–20 tools; the course passes the top 3–5. The self-lookup tool (`read_toolbox` as a registered tool) lets the agent discover capabilities mid-execution when the initial toolset proves insufficient. Registration deduplicates by tool name against the store. -- **CTX-C5 — Embedding & consistency (→ Ledger D17, §2).** One local sentence-transformers +- **CTX-C5 — Embedding & consistency (→ §2, R15).** One local sentence-transformers model (768-dim) serves every store; the query-time model must match the ingest-time model or similarity is meaningless. The course re-creates all tables at each lesson start explicitly "to guarantee a clean starting state with consistent distance strategy" — the strongest in-course signal that mixed strategies corrupt retrieval. Distance config homes: cosine in the L2/L3 notebooks' store setup; Euclidean in the L4/L5 notebooks' store setup; the L2 notebook's markdown names Euclidean while its code passes cosine. -- **CTX-C6 — Index background (→ Ledger D18).** The course text teaches HNSW (graph-based +- **CTX-C6 — Index background (→ §2).** The course text teaches HNSW (graph-based nearest-neighbor traversal) and its lesson objectives name it, but the helper that actually creates indexes builds IVF (neighbor-partition organization, target accuracy 95) to dodge store-version bugs — a reminder that index choice is a store-realization detail, while the invariant is retrieval correctness. Indexes exist to avoid full scans; at fixture scale exact search is correct and simpler. -- **CTX-C7 — Acquisition & ingestion (→ Ledger D14, D16; R14).** The course's acquisition tools: +- **CTX-C7 — Acquisition & ingestion (→ Ledger D13, D15; R14).** The course's acquisition tools: a web search (default 5 results per call, each written with title/url/score/query/timestamp metadata), a candidate search returning structured JSON (id, title, authors, published, abstract capped at 2,500 @@ -516,7 +514,7 @@ Reference only — decisions are made in the Decision Ledger, not here. timestamps). Chunking exists because embedding inputs are bounded — oversized inputs fail or truncate. Its bootstrap corpus streamed the first 100 records of a public arXiv dataset, concatenating title + subjects + abstract as the embedded text with the fields as metadata. -- **CTX-C8 — Loop & write-back parameters (→ Ledger D8, D13; R11–R13).** Max 10 iterations; +- **CTX-C8 — Loop & write-back parameters (→ Ledger D8, D12; R11–R13).** Max 10 iterations; tool results >3,000 chars truncated for the model with a log-id pointer; tool-log previews capped at 2,000 UTF-8 bytes; workflow records store the query, ordered outcome-marked steps, and an answer excerpt capped at 200 chars, and reads filter to `num_steps > 0`; entity @@ -557,14 +555,14 @@ where they disagree, the transcripts win. **Nothing in this spec requires platfo persistent infrastructure → CTX-A preamble. - **Lesson 2 — Why AI Agents Need Memory:** stateless-agent failure demo (CTX-B1); conversational memory and its limits; short/long-term memory taxonomy (CTX-C1); RAG-to-agent-memory bridge and - same-embedding-model requirement (CTX-C5); the Agent Memory Core (CTX-C1, D15). + same-embedding-model requirement (CTX-C5); the Agent Memory Core (CTX-C1, D14). - **Lesson 3 — Constructing The Memory Manager** *(notebook file L2)*: agent stack and memory layer; memory manager and memory units (CTX-C1); deterministic vs agent-triggered classification (CTX-C2, D8); context engineering and the memory lifecycle; store creation, SQL vs vector argument (D7), index creation (CTX-C6); dataset bootstrap ingest (CTX-C7). - **Lesson 4 — Scaling Agent Tool Use with Semantic Tool Memory** *(notebook file L3)*: the - toolbox pattern and its failure argument (CTX-B2, D11); memory-unit augmentation (CTX-C4, - D12); search-and-store with web search (D14, D16); arXiv candidate search and deep ingest + toolbox pattern and its failure argument (CTX-B2, R4); memory-unit augmentation (CTX-C4, + D11); search-and-store with web search (D13, D15); arXiv candidate search and deep ingest incl. chunking config (CTX-C7). - **Lesson 5 — Memory Operations: Extraction, Consolidation, and Self-Updating Memory** *(notebook file L4)*: summarization vs compaction trade-off (CTX-C3, D10); workflow memory @@ -572,7 +570,7 @@ where they disagree, the transcripts win. **Nothing in this spec requires platfo (CTX-B3/B4/B5, R6–R9). - **Lesson 6 — Memory Aware Agent** *(notebook file L5)*: the agent loop and harness; memory operations in/out of the loop (D8); partitioned context and memory-aware system prompt (D9, - R16); tool-log context offloading (D13, CTX-B6); iteration cap (CTX-B8); the end-to-end + R16); tool-log context offloading (D12, CTX-B6); iteration cap (CTX-B8); the end-to-end MemGPT-paper demonstration sequence reused as AC20. - **Lesson 7 — Conclusion:** recap of memory modeling, semantic retrieval, extraction, consolidation, write-back.