feat: native voice (wake+STT+AEC) + Tauri app — Phases 1-5 - #8
Draft
nirhemo wants to merge 13 commits into
Draft
Conversation
…ke→STT→Router→Kokoro)
Phase 1 (voice-provider seam):
- HUD gets a voiceMode switch ("web" | "native"); browser Web Speech becomes the
fallback provider. Additive /voice/ws channel + NativeVoice client (handshake,
reconnect). Guards on all mic loops so mic ownership hands over cleanly.
Phase 2 (native audio pipeline, opt-in via voice.native_audio):
- atlas/interface/voice/audio_service.py: AudioService state machine
(idle→listening→transcribing→thinking→speaking) + NativeAudioRunner wiring the
real engines: sounddevice capture, openWakeWord (hey_jarvis) wake, energy VAD,
whisper.cpp (base.en) STT, shared Router.chat, Kokoro played NATIVELY. Streams
state/wake/final/reply/tts over /voice/ws via a thread-safe VoiceHub.
- Half-duplex for now (mic ignored while speaking); Phase 3 adds echo-cancelled
full-duplex barge-in. whisper non-speech markers ([BLANK_AUDIO]) filtered.
- tts.synth_pcm() for native playback; /api/voice/native status; settings
voice.native_audio + voice.aec_backend. HUD reflects native events + orb abort.
All deps optional (documented in requirements.txt) — without them the HUD keeps
Web Speech. 56 tests pass offline (state machine via injected fakes; WS native +
fallback paths; status endpoint). Real wake+STT engines verified loading/running
on-device; live mic capture needs interactive permission (owner's on-device test).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- atlas/interface/voice/aec.py: EchoCanceller (constrained overlap-save FDAF, pure numpy) removes the far-end (Kokoro) signal from the mic — measured ~9 dB echo reduction on synthetic room echo. make_aec() picks a backend: "os" (macOS VoiceProcessingIO — best; needs a native audio-unit bridge, falls back), "webrtc" (needs C++ build, falls back), "nlms" (the FDAF), "none" (passthrough). - audio_service.py: full-duplex mode. The LLM+TTS turn now runs OFF the mic thread (async_turn) so the mic loop stays live while ATLAS speaks; during "speaking", _maybe_barge() AEC-cleans each frame and runs wake on the residual → "Hey Jarvis" interrupts mid-sentence (abort + queue-to-listen) without self-triggering. Kokoro PCM is queued as the AEC reference; playback is interruptible. - Best-effort reference timing (needs on-device delay calibration; the OS backend avoids it entirely) — flagged in code for the owner's on-device tuning. 60 tests pass offline (FDAF echo reduction, barge-in decision, backend fallbacks; turn state machine made deterministic via async_turn=False). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wrap the existing HUD in a native desktop app (macOS now, Windows-ready): - Sidecar contract: `python -m atlas.server --port 0` picks a free port, binds 127.0.0.1, and prints "ATLAS_READY <port>" once serving (server/__main__.py). - deploy/atlas-core.spec + atlas_core_entry.py: PyInstaller freeze of the FastAPI core into a standalone binary (audio ML libs excluded — optional local install). VERIFIED: the frozen binary serves the API + HUD with no Python installed. - src-tauri/: Tauri 2 shell — spawns the sidecar, reads ATLAS_READY, loads the HUD webview; single-instance, tray show/hide + Quit, autostart, hide-on-close, clean sidecar kill on exit. Entitlements (mic + hardened runtime) for notarization. - deploy/build-app.sh: freeze → place sidecar with target-triple → `cargo tauri build`. - VERSION.json bumped to 0.6.0 (packaged-app fallback when there's no .git). The Rust compile + GUI launch + icon generation are the owner's on-device build step (`./deploy/build-app.sh`); the sidecar packaging is verified here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- .github/workflows/build-app.yml: macOS + Windows build matrix (tauri-action) on a v* tag — freezes the sidecar, builds the app; macOS signing + notarization activate when the Apple secrets are present (unsigned builds still succeed). - src-tauri/Info.plist: NSMicrophoneUsageDescription for the native voice mic prompt. - keychain_secret() is now cross-platform: macOS `security` CLI primary, OS keyring (Windows Credential Manager / Linux Secret Service) fallback. - deploy/NATIVE_APP.md: build steps, what's verified vs. needs-your-machine, and the Windows follow-ons (Azure signing, WASAPI-loopback AEC reference, CUDA whisper). Signing needs the owner's Apple Developer cert; the Windows build/test needs a Windows runner — both wired, both gated on their inputs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nirhemo
marked this pull request as draft
July 14, 2026 11:20
From live testing on the native app: - VAD no longer ends the utterance on the initial silence after the wake word — it waits for the owner to actually START speaking, then detects the end-pause. If nobody speaks within the onset window (5s), it drops back to idle instead of "thinking" at silence. This fixes "it goes straight to thinking without waiting." - Conversational: after ATLAS answers (or a barge-in), it keeps listening for a follow-up with NO wake word needed; an explicit "stop" goes quiet instead. 62 tests pass offline (+3: wait-for-speech/onset-timeout, follow-up-without-wake; updated full-turn/empty/abort for the new flow). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verified fixes from an end-to-end review of the photos/search feature: - parseArticleCommand() extracted — handleFrameVoice() called it but it never existed, so saying "open article N" with a frame open threw ReferenceError. - Native-voice reply now ALWAYS calls renderSearchMedia(media) (was guarded on !== undefined), so stale floating photos/article clear on a non-search reply — matching the text-chat path. - ask(speak) and speakReply() no longer trigger browser TTS in native mode (the native service owns the voice) — kills the dual-voice conflict from the TLDR button / text turns. - web_search.rich() only fetches images when the text search produced results, so a text-search failure can't leave orphaned photos with no article. Backend + rendering were already sound; these fix the native-mode paths. 62 tests pass; headless-verified (no crash, media clears, single voice). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…/stop)
Fixes "stop doesn't work while speaking" and "open article N does nothing":
- Full-duplex: while ATLAS speaks, the mic stays live (AEC-cleaned). Barge-in on
EITHER the wake word (echo-robust) OR sustained talk-over energy (~400ms), so
you can cut in mid-sentence — it stops and listens for your command.
- Native voice commands: "open article N" / "close" / "stop" are now recognized
and routed to the HUD (open the reader / close / go quiet) instead of being sent
to the model, which couldn't act on them. local_command() parses them; the
audio service publishes a {command} event the HUD handles.
- settings: aec_backend "os" (→ software AEC fallback) enables full-duplex.
64 tests pass offline (+2: open-routes-to-HUD, stop-goes-quiet).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- After an "open article N" / "close" voice command, the native pipeline now stays in listening (no wake word) so "close" / "next" work while you're reading the article — previously it dropped to idle and "close" alone wasn't heard. - Photos enlarged (~2× target) with fit-to-viewport scaling so the grid never overflows the window (390px on a 1280px window for a 6-photo grid; scales up on bigger windows / fewer results). Centred, staggered fly-in unchanged. 64 tests pass; headless-verified (bigger photos, 0 off-screen, no errors). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- local_command() now handles the many ways whisper phrases it ("open article
one", "open number 3", "open article number one", "show me article 2", "pull up
the third article"…) so voice "open article N" reliably drives the reader instead
of falling through to the model. Logs each voice turn's transcript + parsed
command (voice_turn event) for diagnosis.
- Tauri: closing the window now QUITS ATLAS (stops the backend + voice) instead of
silently hiding to the tray and continuing to speak. Quit also on the tray menu.
64 tests pass offline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The native pipeline loaded whisper + openWakeWord synchronously in the FastAPI
lifespan, blocking the server (and the app window) for ~15s. Now:
- NativeAudioRunner.start() returns immediately and builds the engines + opens the
mic on a background thread; the server serves in <1s.
- When the pipeline is live it publishes an updated {capabilities}/{ready} over the
hub; /voice/ws always subscribes, so a HUD that connected during load switches to
native the moment it's ready.
- capabilities.audio reflects runner.ready ("native audio loading…" until then).
Measured: ATLAS_READY in ~0.9s (was ~15s); native comes up in the background.
64 tests pass offline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Inter-sentence listen gaps let the mic (ducked to near-silence during playback on the DGM20/macOS) recover so 'Hey Atlas' or talk-over can interrupt; one persistent output stream replaces per-sentence sd.play churn; raw-mic wake barge with a separate reset-per-utterance model; clip in the mic callback to stop int16 overflow; extend the listen window while an article reader is open; report aec honestly. Adds a /api/voice/native mic-level readout + an echotest endpoint used to diagnose the output-ducks-the-mic behaviour. Barge path is unverified pending mic hardware. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the teal-delta app icons; drop the unused [lib] target and the externalBin sidecar (the shell now runs the repo venv python directly); mark run.sh executable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Research-backed design (Hermes Agent + OpenClaw) for giving ATLAS filesystem + code-execution tools: a confined ~/ATLAS-workspace, path jail + Apple Seatbelt OS sandbox, READ/WRITE/DESTRUCTIVE gating, git-backed undo, plan/apply folder organizer, and a SKILL.md learning loop, with a phased plan. Proposal for review — no implementation yet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns ATLAS into a native desktop app with a fully on-device voice pipeline. Everything is on the branch; the Python core, HUD, memory, and settings are reused verbatim. 60 tests pass offline.
Phases
web/nativeswitch; additive/voice/ws; browser Web Speech becomes the fallback. Mic-ownership guards.audio_service.py): mic (sounddevice) → openWakeWord (hey_jarvis) → VAD → whisper.cpp (base.en) STT → shared Router → Kokoro played natively, streaming state over/voice/ws. Opt-in viavoice.native_audio.aec.py): pure-numpy FDAF echo canceller (~9 dB reduction measured) lets ATLAS listen while speaking without hearing itself; "Hey Jarvis" interrupts mid-sentence. macOS VoiceProcessingIO backend scaffolded (native bridge = on-device follow-up).src-tauri/): the FastAPI core runs as a PyInstaller sidecar; the shell readsATLAS_READY <port>and loads the HUD. Tray, single-instance, autostart, clean shutdown. The frozen sidecar is verified serving the HUD with no Python installed.build-app.yml(macOS+Windows, tauri-action, signing gated on Apple secrets); mic entitlements; cross-platform secret store (Windows Credential Manager fallback); Windows follow-ons documented.Verified here
Needs the owner's machine (inherent limits)
./deploy/build-app.sh(Rust compile + GUI + icons); Apple cert for signing; a Windows box.Notes
🤖 Generated with Claude Code