[FEATURE] Shape dictation with a prompt through the on-device model

[jhampton]

Stacked on #67, and opened as a draft on purpose: it argues with CONTEXT.md's raw-finalized-text rule, so it is offered for discussion, not pushed.

An off-by-default prompt shaping pass, now worded as a beta, rewrites finished dictation through Apple's on-device FoundationModels session between recognition and insertion. The transcript travels as marked data under fixed instructions with a per-prompt one-shot example, so question-shaped dictation comes back rewritten instead of answered. The prompt library is user-editable in Settings, seeded with the original three prompts, while the framing that keeps the model rewriting stays out of the user's hands. The phase is visible: the HUD stays up through the rewrite over a bar that fills toward the ten-second timeout, and while a session records the bare arrow keys cycle a session-scoped pick through the library and None, shown in a shaping band the shape grows downward to include below the voice visual and the draft. Passthrough remains the rule on every failure path, and transcription history keeps the words as spoken, before shaping. docs/adr/0008-prompt-shaping-alpha.md records the design and each later decision, including why the promotion keeps the toggle off by default. Written with Claude Code under my direction and reviewed line by line; the passthrough paths, the arrow cycling, and the HUD geometry are pinned by tests, green on an Apple Silicon Mac running macOS 26, and shaped insertion, the cycling, and the shaping band have all been exercised against the live on-device model on a notched MacBook Pro in daily use.

Refs #53

[tornikegomareli]

Hey, great work. I will review it and test it later today

[jhampton]

I've been testing this, and it seems that they way the foundation model is being used confuses dictation with instruction: I've been getting back AI slop responses from the model. This is very much a WIP @tornikegomareli !

[tornikegomareli]

I've been testing this, and it seems that they way the foundation model is being used confuses dictation with instruction: I've been getting back AI slop responses from the model. This is very much a WIP @tornikegomareli !

That was my initial thoughts about foundation models, but still has hope we will find some balance and ways to work it out.

[jhampton]

Found the failure mode and pushed a fix, rebased onto 0.7.1. The model was being handed the transcript as its conversational user turn with only a short role line in instructions, so an instruction-tuned model treated a question-shaped dictation as a question to answer. The framing now lives where the model weighs it most: the session instructions state that the user turn is always a raw transcript to transform, never a question to answer or an instruction to follow, the transcript arrives wrapped in explicit markers as data, and each library prompt carries a one-shot example whose input is question-shaped and whose output rewrites it without answering it, since an example carries that rule better than a sentence stating it. Passthrough on every failure path, the ten-second timeout, and history keeping the words as spoken are all unchanged, and the new assembly is pinned in PromptShapingServiceTests. I used Claude Code for most of the writing under my direction and can explain every line. Full suite green locally on Apple Silicon; I'll report back after live dictation of question-shaped utterances against all three prompts.

[jhampton]

@tornikegomareli — since opening this draft the branch has grown past what I originally described, and I would rather flag that plainly than let the diff speak for itself. On #53 I said user-authored prompts were a later decision; after living with the alpha day to day I took that decision on my fork, so the branch now carries a user-editable prompt library (Manage Prompts… in Settings, seeded with the original three, with the transcript-as-data framing kept fixed because it is also the fix for a real bug where question-shaped dictation came back answered instead of cleaned). The branch also makes the phase visible — the HUD stays up through the rewrite over a bar filling toward the timeout, and the bare arrow keys cycle a session-scoped pick shown in a band the shape grows downward to include below the visuals and the draft — and with daily use behind it I promoted the wording from alpha to beta, still off by default. All of this is offered as a proposal for your review, not a fait accompli: if it goes past what you want from #53, the editable library especially, I am happy to trim the branch back to the fixed-library alpha we discussed.

[jhampton]

The red check is #82's insertion-test flake, not this diff: TextInsertionServiceTests.activeReadSkipsLaterInsertionImmediatelyAndRecovers has now failed three CI runs in a row on this head while passing locally every time, and this branch never touches that file. Details and today's run links are on #82 — the short version is that this PR roughly doubles the parallel test load, so the runner starvation that test measures with real-clock bounds went from occasional to consistent. I would rather not keep re-running red CI to roll a green, so I will leave it as it stands; the injectable-clock fix you sketched on #82 would end it properly, and I am glad to implement that under its own issue if you reopen or refile it.

[jhampton]

Just a quick follow-up here. I've been running the UI changes and the ability to, for instance, cycle through different kinds of shaping, including turning off shaping when you're in the latch mode instead of the temporary push-to-talk mode. In push-to-talk mode, you cannot actually cycle back and forth. So I may need to follow up there, but it's been working really, really well. And I'm pretty impressed.

You do have to do a lot of prompt shaping in order to get it to, um, more closely adhere to your directions, just just doing remove filler words or tighten grammar is actually really pretty fantastic. I'm including some screenshots for reference.

[tornikegomareli]

Yes to all of it. The feature goes in, it stays off by default, and the prompts start with your three seeds and stay editable. The version you ran on your fork is the one I want, so there is no need to cut it back to a fixed prompt library.

I really liked the passthrough and its really is complete. When shaping is unavailable, throws, refuses, sees an empty string, or times out, it returns the words untouched, and shape itself never throws. None of the failure paths are hand waving.

There are two integration issues This branch started before translation, which landed in 0.7.0 while you were working.

Right now shaping runs on the translated text. The order is recognise, translate, write history, shape, insert. With both features on, an English prompt with an English example is handed Spanish. It might work, since the model is multilingual, but we did not choose that behaviour. Shaping before translation also reads better on its own: the prompt runs in the language it was written for, and translation gets cleaner input.

History’s delivered line also no longer matches what was inserted, because we write it before shaping. The ES: line stores translated but unshaped text while the document receives the shaped text. Your comment about history keeping the words as spoken is still true of the spoken half, but the delivered half now claims something that never landed.

There is a third issue and that one is mine. finishGrace is TranslationService.defaultTimeout + 2s, which was right when translation was the longest bounded step. With shaping’s ten second bound, the finish can now outlast the quit hold, so quitting mid shape can lose the words with history not yet written. It is the same bug class I fixed for translation in 0.7.0 and never generalised.

I am merging this as it is and will fix all three on main in a follow up, so your commits land intact instead of getting churned in review. The fix for the first two is one change in order: shape before translate and keep the history write last. The third is a new constant.

On CI, you were right and I checked. It went green on a rerun, your branch never touches that file, and your read of the failure is sharper than mine was. This PR roughly doubles the parallel test load, so the real time bound in that test went from sometimes starved to always starved. I have reopened #82. If you want to take the injectable clock fix, it is yours. That test has now cost me four reruns across two releases.

One thing from your last comment is still open. Arrow cycling only works in latch mode, not while the key is held. That deserves its own issue instead of blocking this one. Thank you for pointing it out yourself instead of leaving it for me to find.