We are making a JSON map of the interface — composed from the GUI hierarchy (Accessibility / AX), the Chrome DOM, and CoreGraphics window geometry — across all available surfaces: every on-screen window of every application, the front app's menu bar, and the active web document, not just the focused window.
One interfacemap map call emits a single JSON document describing the whole
screen: every window with its bounds, layer, and z-order; each window's native
control tree (roles, labels, frames, states) as numbered refs; the Chrome
DOM outline when Chrome is present; and the cursor, displays, and frontmost
application. That map is the shared source of truth for everything else in the
repo.
On top of the map:
- Voice in — push-to-talk speech, transcribed on-device by Parakeet TDT v3 (FluidAudio / CoreML).
- One turn — a planner (OpenRouter, strict JSON schema) maps one spoken goal
plus the map's refs to a single
fill_formcommand covering every field. - Real actuation — each field is executed with a visible mouse glide, a real
CGEventclick, real Unicode typing, and read-back verification (AX read-back for desktop controls, ordinal-matched DOM re-reads for web controls). Any form, web or desktop. No AX writes, no DOM scripting, no shortcuts. - Voice out — results spoken locally by a Supertonic 3 server +
afplay.
Built fresh today for the Abridge hackathon. Companion demo fixture: the SILVERN clinical form app in abridge-hackaton-vised.
| command | does |
|---|---|
interfacemap map [--json] |
emit the full interface JSON map (all surfaces) |
interfacemap refs [--app NAME] |
print the numbered control refs |
interfacemap fill --command JSON|@file |
execute one fill_form with real clicks |
interfacemap goal "…" [--speak] |
plan one turn from a text goal, then fill |
interfacemap voice |
push-to-talk voice loop |
interfacemap say "…" |
TTS smoke test |
swift build -c release
.build/release/interfacemap map | python3 -m json.tool | lessSetup for permissions, speech models, and the planner key: see docs/SETUP.md.