Feature: Auto extraction - #121
Merged
Merged
Conversation
…tions `@coasys/ad4m` and `@coasys/ad4m-connect` both move to `0.13.0-test-interpretation`, published today. It carries the whole surface this branch needs: `interpretationHint` on `ModelConfig` and `PropertyOptions`, `identity` on `PropertyOptions`, and the four `PerspectiveProxy` calls. So the stand-in declarations go — `ad4mDecoratorOptions.d.ts` and `ad4mInterpretationApi.d.ts` both delete. They existed only to let the repo build against a runtime that predated the stack, and a duplicate declaration that ever disagreed with the real one would fail the build rather than merge, which is why they were always going to be deleted rather than left as belt and braces. `runtimeSupportsInterpretation` and its guards **stay**. The web build reaches whatever executor the connect UI is pointed at, so one without the stack is now an ordinary thing to meet rather than the only thing available — the guard is the difference between a sentence naming what is missing and a `runInterpretation is not a function` stack trace. Verified on the published package with no worktree link: build, typecheck, lint, 28 schemas and the full suite green, and `@we/backend-ad4m` reports 162 passed with **0 skipped** — the two tests that skip themselves when a runtime drops `interpretationHint` now run, which is what says the hints reach the SHACL.
`watch`/`unwatch` were declared on the port and unimplemented. This fills them in against `addAutoProcessor`, which is the half of the interpretation engine that runs without anybody pressing anything. **The turns become a query rather than a list.** The one-shot path hands the engine turns the *host* read out of a collection; a watch cannot work that way, because the point is that it runs with nobody there to read anything. So the scope is SPARQL, and `transcriptScopeQuery` is where WE's layout gets written in it: children of the call, flagged as text blocks, with speaker and timestamp read off the **reifier of the body link** rather than from fields on the block. That is AD4M's own convention, and it is why a `TextBlock` needs no author property for this to work — every agent transcribes their own microphone, so the link's author is the speaker. Dialect in a string is what the port exists to avoid, and this is the adapter's own file, which is the one place it is allowed. The alternative is a query IR that can express reification, which is far more than one call site is worth. **`unwatch` deletes the config, because there is no other removal.** `addAutoProcessor` has no counterpart — not on `PerspectiveProxy`, not in the WS handler map, not in the engine. What it does is write an `AutoProcessorConfig` instance into the perspective's graph, which the watch loop reads back on every tick. So the registration is data and deleting the record is what stops it. That is also the right shape rather than a workaround: the config is shared, so removing it stops the watch for the neighbourhood rather than for whoever pressed stop. **Names in, URIs out.** `runInterpretation` matches shape *names*; `addAutoProcessor` takes target-class URIs. Two entry points to one engine that disagree about how a class is named — the same trap that cost a day on the one-shot path, where the wrong one is logged as "skipping class" and surfaces as "perspective has no subject classes to extract into". `targetClasses` converts explicitly and throws on an unknown name rather than dropping it, since dropping would silently narrow what a watch extracts. Feature-probed separately from `interpret`, because the port declares them separately and that is not a formality: one-shot is a function call, a watch is coordination. A backend can have the first without the second, and a caller assuming otherwise registers a watch that never fires and reports nothing wrong. The generated query is pinned by tests against the shape AD4M's own neighbourhood test uses — all three bindings present, the reified triple naming the same predicate as the body pattern, ordered by time. It is the one part that cannot be checked by reading: a query binding only speaker and text fails the gather, and a failed gather looks exactly like a quiet call.
Three layers of plumbing so a module can say "keep interpreting this collection" without holding a watch. **The module contract gains `watchCollection`/`unwatchCollection`,** which does not contradict its own note that `watch` is deliberately absent. A module names a collection and holds nothing: the watch id, the dataset, the containment predicate and the lifetime are the host's. That is what keeps a panel closing from leaving a registration behind — the failure the original note was written against. **The host derives the watch id from the collection** (`we-call:<id>`) rather than storing one. Any peer computing it must land on the same string, because the engine keys its processors inside the *shared* perspective: two members starting one call and disagreeing about the id would register two watches over one transcript and interpret every utterance twice. **The transcribe module pairs the watch with the collection assignment**, not with the record button and not with an effect over the signal. The collection appears late — whoever wins the creation election makes it and the rest adopt the announced id — so there is nothing to name at the press. And four separate paths assign it (adopt, create, resume, call-ends); an effect that missed one would leave a watch pointed at a call that is over, which fails silently and keeps spending an LLM call on a conversation nobody is having. `useCollection` makes the pairing structural instead of remembered. Best-effort throughout, logged at debug rather than warn: on any runtime without the auto-processor this throws on every call, and a warning there would teach people to ignore the console. The Extract button is the whole feature without it. Four tests pin the lifecycle — nothing before there is a collection, the right classes when there is one, removal when the call ends, and a backend that cannot hold a watch not taking the call down with it.
`addAutoProcessor` has no `parent` option. `basePrefix` decides the URI an instance is minted under — which confines a pass to a subtree and creates no edge — so nothing links a result to the call it came from. An unparented `TaskBlock` is a real, queryable record that no traversal-shaped route lists, so the call card and the graph would show a successful pass as nothing having happened. The board and the calendar find them regardless, reading `status` and `startDate` rather than containment. Two paths, because one is not enough. **The listener** parents on the `processed` event, guarded on `agentDid`. That guard is not defensive: the event stream carries *other peers'* passes — `agentDid` is documented as "which peer claimed/processed/backed off" — so without it every online member links the same records and the call collects one duplicate edge per participant. **The sweep** covers what the listener cannot see. It only fires on the client whose node ran the pass, and only while that client is open; on desktop the executor is a separate process, so it can win an election and finish a pass with the app closed. Those records would never get their place. `reconcile` runs when a call is adopted — the moment somebody is about to look — and links every instance under the call's `basePrefix` that the call does not already contain. Both halves are forward traversal, the call's children and instances by URI prefix, so no reverse scan is involved and it is cheap enough to run on open rather than on a schedule. The in-memory map of watch → parent is deliberately not persisted. A pass can complete on a peer's node while this client has never run, so *some* path has to repair after the fact regardless; once that exists the map is only an optimisation on the common case. Both delete the day `AddAutoProcessorConfig` grows a `parent` — with the edge written server-side there is no window in which a record exists without one. Also feature-tests the host wrapper **per method** rather than per object. The host always publishes a forwarding wrapper, so `interpretation?.` only answers "is there a wrapper"; an older host whose wrapper predates the watch sails past that and throws on the call. Caught by an existing test whose stub predates these methods — which is exactly the shape of host a released module will meet.
`SHACLShape.toJSON()` now emits `identity`, and `fromJSON()` reads it back. Verified in the published tarball rather than from the release note, since the previous release carried the `identity` decorator and the `toLinks` half while this exact serialiser still dropped it — and `ensureSubjectClasses` registers every class *through* `toJSON()`. So the dedup key reaches the perspective, the interpreter can see it when it assembles its "already exists" block, and a repeat pass stops re-minting everything it finds. That matters more for the standing watch than it did for the button: a watch runs repeatedly by design, so the same task was being re-minted on every pass rather than once per press.
A standing watch spends an LLM call on whichever member's node wins the election and writes what it finds into everyone's copy of the space. Left to each agent, one member could sign the rest up to both — so `Space.autoInterpret` is a property of the space, administer-only, and false unless somebody says otherwise. Joining a space should not be the same act as volunteering to run its extraction. The gate is *injected* rather than read: the setting lives on a `Space`, SpaceStore layers on DatasetStore, and the dependency only points one way. So DatasetStore takes `provideAutoInterpretGate` the way TemplateStore takes `provideSpaceLookup`, and an unset gate reads off — a decision nobody has made is not a decision to spend somebody's budget. Its own settings section rather than a row under Modules, because it is not a module: it is what one of them is allowed to do while nobody is watching. The copy says who pays, since that is the part easy to miss. Three generated artefacts move with it, and each was caught by its own check rather than by inspection: the store manifest the schema validator reads (the switch failed validation as an unknown member), the capability classification the surface test demands (an unclassified member is absent from a template's bag — the `$sources` failure again), and the core SHACL manifest the compiled-vs-written test compares.
A standing watch that produces nothing looks exactly like a call nobody said anything extractable in. Testing the first one against a real executor, the whole observable surface was an absence — no way to tell "never registered" from "registered and the gather returned nothing" from "another peer holds the claim". **The engine was already saying which.** `AutoProcessorEvent.step` names it exactly — `emptyTranscript`, `shapesMissing`, `backedOff`, `notCandidate`, `gatheringTranscript`, `processed` — and the listener filtered everything except `processed` because parenting was all it was written for. Throwing away the only evidence there is was the wrong trade for one `if`. Also logs the registration itself, from both sides. Module-side, because no log is equally consistent with "this never executed" and "it executed and threw nothing" — the ambiguity that made the first round of testing inconclusive. Adapter-side with the class URIs, and the scope query at debug, because a gather that binds nothing fails silently and the query is the one part of this that cannot be verified by reading — so it is emitted where it can be copied and run by hand.
Two bugs, found by running it. The second is the one that mattered. **`unwatch` queried through a shape that is not there.** The engine registers the config class as **`AutoProcessor`**; the ORM class is `AutoProcessorConfig`, and `findAll` resolves a shape *by name* — so the delete failed with "No SHACL shape stored for class 'AutoProcessorConfig'". Two names over one set of instances, which the model's own docs anticipate by telling callers to register it first. It now does. This is the same disagreement about how a class is named that `targetClasses` already exists for, one layer along; both are worth reading as one symptom. **And that failure took the next registration with it.** `unwatch` and `watch` sat in one `try`, so the throw skipped the call that starts watching the *new* collection — one failed teardown and nothing was watched again for the rest of the session. Which is exactly what happened: the first call registered fine, the error fired on ending it, and every call after that was silent for a reason that had nothing to do with them. They are now separate attempts with separate handling, because they are unrelated operations on different collections: stopping a watch on a call that ended and starting one on the call that just began are each worth doing when the other cannot be. A failed stop is a warning rather than debug — a watch left running keeps interpreting a call that is over, at an LLM call per pass. Pinned by a test that fails the teardown and asserts the next call still gets registered.
The id becomes part of a URI — the engine mints its processor node at `ad4m://autoprocessor/<id>`. A WE collection id is itself a `literal:string:…` URL, so passing it through raw produced `ad4m://autoprocessor/we-call:literal:string:xhzng…`: a URI with `literal:` inside it, on a layer that decides literal-from-URI by exactly that prefix. Not proven to be why no pass has ever run — the config is written, the RPC handler exists on the node being tested against, and the watch loop is wired into `start_background_tasks` at the version it is built from. But it is the one oddity visible from this side, and not worth leaving in the picture while diagnosing something else. Letters, digits and dashes, still derived rather than stored, so peers still agree on it.
`addAutoProcessor` resolving says the RPC was accepted. It does not say the config is in the perspective, and that is the claim that matters: the watch loop polls `load_processors`, which reads `AutoProcessorConfig` instances out of the graph. A write that did not land, or landed in a shape the loop cannot parse, leaves the loop with nothing to do and nothing to say about it — which from outside is indistinguishable from a loop that is not running at all. Those two want different people looking at them, so the client now says which. One query per call start, and it prints what the perspective actually holds.
…n is running Three things the first working run surfaced. **The task hint had no notion of scope.** A live call produced "Add more" and "Trigger the extraction" as tasks — from "maybe I need to add some more" and "I guess one more to trigger the extraction". Both are imperative and neither is work: they are about the act of testing. `EventBlock` already excludes "the conversation currently happening"; `TaskBlock` now requires work that outlives the conversation and names that failure mode directly, since the model plainly needs telling. **The diagnostics come down to debug.** The config read-back is gone — it cost a query per call start to answer a question that is now answered. The per-step logging stays, because it is what turned a day of silence into a five-minute diagnosis, and the scope query stays with it: a gather that binds nothing fails silently, so keeping the query one console-filter away is worth more than the noise it saves to delete it. **Auto-extraction says it is on.** It was otherwise invisible — a pass runs on somebody's node, writes records, announces nothing, so a call with nothing found yet looked identical to a space where the setting was never turned on. That ambiguity cost a day here and would cost a user the same question with no way to answer it. The card reads the space setting; surfacing the engine's own per-pass steps would be better but needs new port surface, so it stays a follow-up. That indicator also corrected a classification error: `autoInterpret` was `space-admin` state, copied from `enabledModules` beside it, which made it unreadable by the template that wanted to show it. It is `space-settings` now — the audience is every member, not only whoever may change it, and whether a space extracts is not an administrator's secret when the results land in everyone's copy. `tierFit` caught it, which is the check earning its place: an unreadable member failed a test rather than rendering nothing forever.
✅ Deploy Preview for coasys-we ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Auto-extraction — interpreting a call as it happens
Summary
Extraction has been one-shot: somebody presses Extract and a pass runs over the transcript so far.
This adds the standing form — a watch registered against a call's collection, which interprets new
turns as they arrive, on whichever member's node wins the election, with nobody pressing anything.
Four layers: the watch implemented over AD4M's auto-processor; registration driven from the call it
belongs to; two paths that attach what a pass produces; and a space setting that decides whether any
of it runs.
Pinned to
@coasys/ad4m@0.13.0-test-interpretation-1.Status: verified working end to end against a locally built executor, and inert against any
node that has not been rebuilt from the upstream gather fix this work uncovered
(
fix/interpretation-gather-expression-body, open onfeature/generic-extraction-ws-ts). The fix isRust-only, so no client release is involved — but until a node carries it, the watch registers, the
config lands, and no pass ever runs. One-shot extraction is unaffected and works today. See
Sequencing at the end.
Changes
The watch
interpretationAdapter.ts—watch/unwatch. Declared on the port since the interpretationbranch and unimplemented until now.
The turns become a query rather than a list. The one-shot path hands the engine turns the host
read out of a collection; a watch cannot work that way, because the point is that it runs with
nobody there to read anything. So the scope is SPARQL, and
transcriptScopeQueryis where WE'slayout gets written in it: children of the call, flagged as text blocks, with speaker and timestamp
read off the reifier of the body link rather than from fields on the block. That is AD4M's own
convention, and it is why a
TextBlockneeds no author property for this — every agent transcribestheir own microphone, so the link's author is the speaker.
unwatchdeletes the config, because there is no other removal.addAutoProcessorhas nocounterpart — not on
PerspectiveProxy, not in the WS handler map, not in the engine. What it doesis write an
AutoProcessorConfiginstance into the perspective's graph, which the watch loop readsback on every tick. So the registration is data, and deleting the record is what stops it. That is
also the right shape rather than a workaround: the config is shared, so removing it stops the watch
for the neighbourhood rather than for whoever pressed stop.
Names in, URIs out.
runInterpretationmatches shape names;addAutoProcessortakestarget-class URIs. Two entry points to one engine that disagree about how a class is named — the
same trap that cost a day on the one-shot path, where the wrong one is logged as "skipping class"
and surfaces as "perspective has no subject classes to extract into".
targetClassesconvertsexplicitly and throws on an unknown name rather than dropping it, since dropping would silently
narrow what a watch extracts. Raised with the runtime team as a consistency ask.
Registration
The module contract gains
watchCollection/unwatchCollection, which does not contradict itsown note that
watchis deliberately absent. A module names a collection and holds nothing: thewatch id, the dataset, the containment predicate and the lifetime are the host's. That is what keeps
a panel closing from leaving a registration behind — the failure that note was written against.
The watch id is derived, not stored (
we-call-<collectionId>, reduced to letters, digits anddashes). Any peer computing it must land on the same string, because the engine keys its processors
inside the shared perspective: two members starting one call and disagreeing would register two
watches over one transcript and interpret every utterance twice. The reduction matters because the
id becomes part of a URI — the engine mints its processor node at
ad4m://autoprocessor/<id>— and aWE collection id is itself a
literal:string:…URL, so passing it through raw produced a URI withliteral:inside it on a layer that decides literal-from-URI by exactly that prefix.The transcribe module pairs the watch with the collection assignment, not with the record button
and not with an effect over the signal. The collection appears late — whoever wins the creation
election makes it and the rest adopt the announced id — so there is nothing to name at the press.
Four separate paths assign it (adopt, create, resume, call-ends); an effect that missed one would
leave a watch pointed at a call that is over, which fails silently while still spending an LLM call
per pass.
useCollectionmakes the pairing structural rather than remembered.Attaching the results
AddAutoProcessorConfighas noparentoption.basePrefixdecides the URI an instance is mintedunder — which confines a pass to a subtree and creates no edge — so nothing links a result to the
call it came from. An unparented
TaskBlockis a real, queryable record that no traversal-shapedroute lists, so the call card and the graph would show a successful pass as nothing having happened.
The tasks board and the calendar find them regardless, reading
statusandstartDaterather thancontainment.
Two paths, because one is not enough:
processedevent, guarded onagentDid. Not defensive: the eventstream carries other peers' passes, so without it every online member links the same records and
the call collects one duplicate edge per participant.
pass, and only while that client is open; on desktop the executor is a separate process, so it can
win an election and finish a pass with the app closed.
reconcileruns when a call is adopted andlinks every instance under the call's
basePrefixthat the call does not already contain. Bothhalves are forward traversal, so no reverse scan is involved.
Both delete the day
AddAutoProcessorConfiggrows aparent. Raised with the runtime team; thereply confirmed
basePrefixis used for output, which it is — as a URI namespace. A namespace isnot an edge, and WE's routes reach content by traversal.
The gate
Space.autoInterpret, administer-only, false by default. A standing watch spends an LLM call onwhichever member's node wins the election and writes what it finds into everyone's copy — left to
each agent, one member could sign the rest up to both. Joining a space should not be the same act as
volunteering to run its extraction.
Injected rather than read: the setting lives on a
Space, SpaceStore layers on DatasetStore, and thedependency only points one way — so DatasetStore takes
provideAutoInterpretGatethe wayTemplateStore takes
provideSpaceLookup. An unset gate reads off.Its own settings section rather than a row under Modules, because it is not a module: it is what one
of them is allowed to do while nobody is watching.
What the first working run changed
Running it end to end surfaced three things worth fixing before anyone judges the feature.
The task hint had no notion of scope. A live call produced "Add more" and "Trigger the
extraction" as tasks — from "maybe I need to add some more" and "I guess one more to trigger the
extraction". Both imperative, neither work: they are about the act of testing.
EventBlock's hintalready excluded "the conversation currently happening";
TaskBlocknow requires work that outlivesthe conversation and names that failure mode directly, because the model plainly needs telling.
Auto-extraction says when it is on. It was otherwise invisible — a pass runs on somebody's node,
writes records and announces nothing, so a call with nothing found yet looked identical to a space
where the setting was never turned on. That ambiguity cost a day of debugging here and would cost a
user the same unanswerable question. The call card now reads the space setting and says so.
That indicator also corrected a classification error:
autoInterpretwasspace-adminstate, copiedfrom
enabledModulesbeside it, which made it unreadable by the template that wanted to show it. Itis
space-settingsnow — the audience is every member, not only whoever may change it, and whether aspace extracts is not an administrator's secret when the results land in everyone's copy. The write
stays admin-gated.
tierFitcaught it, which is that check earning its place: an unreadable memberfailed a test rather than rendering nothing forever.
The diagnostics came down to
debug. The config read-back is gone — a query per call start toanswer a question that is now answered. The per-step logging stays, because it is what turned a day
of silence into a five-minute diagnosis, and the generated scope query stays with it: a gather that
binds nothing fails silently, so keeping the query one console-filter away is worth more than the
noise saved by deleting it.
Known follow-ups
gather_transcript_sparqlskips every row whose?textis an expression-wrapped literal — which is what
@Propertyproduces by default, since it setsresolveLanguage: "literal"and the stored target becomes a signedliteral:json:{author,timestamp,data,proof}envelope.decode_literal_stringaccepts onlyLiteralValue::String, so the gather returns an empty transcript with no error and no event. Foundby instrumenting a locally built executor; patched, verified locally, and open upstream as
fix/interpretation-gather-expression-bodyonfeature/generic-extraction-ws-ts(
decode_transcript_body, applied to both gathers). The fix is Rust-only, so no client release isneeded and this branch's pin stays put — but the deployment will not extract automatically until
the node it connects to is rebuilt from that branch. Until then the watch registers correctly and
produces nothing.
parentonAddAutoProcessorConfig— removes the parenting listener and the reconcile sweepentirely.
runInterpretation(shape names) andaddAutoProcessor(target-classURIs). Converted in the adapter; the inconsistency is upstream.
batchReady,claimed,processed, …) and a "last pass 2 minutes ago" line would be far better, but those arrive on thebackend port and carrying them to a template needs new port surface.
unwatchfailed until the shape-registration fix, so configs fromearlier calls remain in those perspectives. They settle harmlessly — a finished call's turns get
marked processed and no batch forms — but they will each run once when the gather starts working.
Worth clearing a test space before judging output.
[timestamp]text hack. The one-shot WS turn is{speaker, text}, so the host folds timeinto the text and the class hints name that format. The auto path binds
?timestampproperly, so aclass's hints cannot describe both shapes accurately at once. Harmless; simplifies if the WS turn
gains the field.
worth exposing if it is ever pointed at something else.
Test plan
pnpm build(48 packages),pnpm typecheck,pnpm lint,pnpm test,pnpm --filter @we/schema-shared validate— all clean.exists, the right classes when it does, removal on call end, the repair sweep on adoption, a
backend that cannot hold a watch, and a failed teardown still registering the next call).
patch: utterances transcribed, a pass firing on the debounce, and typed instances landing —
"An upcoming task is testing the auto extraction" → a
TaskBlock, "an upcoming event isgoing to Bristol this weekend" → an
EventBlock.auto_processor_*e2e suite passes on the build this is pinned against (5/5), andthe full
perspectives::interpretationsuite is 87/90 — the three failures are model-qualitytests expecting
gemma3:12band fail identically without the patch.and config-write are confirmed working there; the pass is not.
agentDidguard on parenting, and peers converging onone shared config are reasoned from the engine's source, not observed.
Sequencing
The branch is complete and merge-ready as WE-side work. Auto-extraction will not produce anything
until: the gather patch merges upstream → a release carries it → Marvin is rebuilt on that release →
package.jsonmoves off0.13.0-test-interpretation-1.One-shot extraction is unaffected and works today, hosted or local.