AudioGraph

Live audio capture to speech recognition to temporal knowledge graph.

AudioGraph is a cross-platform desktop app (Tauri v2 + React) that taps system audio, runs it through a real-time pipeline of audio normalization, turn detection, speech recognition, speaker diarization, entity extraction, and chat, and streams the results into a live temporal knowledge graph. The provider registry includes local and cloud routes, but the current MVP intentionally enables new durable sessions only through Deepgram plus an enabled LLM route. Deferred saved routes remain inspectable and are re-enabled one at a time after capture, processing, and storage pass the MVP proof.

Product Modes

AudioGraph is being shaped around two related product modes:

Mode	What it does	Local Options	Cloud Options
Speech-to-notes / speech-to-temporal-graph	Captures desktop audio, transcribes it, extracts entities/relations, updates the temporal graph, and lets chat recall the session later.	rsac capture, local fixed-window turn fallback, Whisper, Sherpa-ONNX, local diarization, llama.cpp, mistral.rs	Groq/OpenAI-compatible ASR, AWS Transcribe/Bedrock, Deepgram, AssemblyAI, OpenAI-compatible LLMs, vLLM, planned OpenAI Realtime STT
Parallel speech-to-speech agent	Listens beside the graph pipeline, responds in realtime, and proposes graph/chat actions without blocking durable memory construction.	Rust fan-out, local LLM/vLLM reasoning, future local STT/TTS chain	Gemini Live today, planned OpenAI Realtime `gpt-realtime-2`, hybrid routes using cloud STT/TTS such as Deepgram, provider tool calls routed through backend approvals

Both modes share the same selected audio source, credential store, latency status bar, transcript stream, and temporal graph. The first mode optimizes for accurate memory and recall; the second optimizes for realtime collaboration.

Screenshots

TODO: screenshot. No captured screenshots or GIFs exist yet under docs/assets/. Contributions welcome — record a short GIF of a live capture session (knowledge graph + transcript + chat sidebar) and drop it into docs/assets/, then update this section.

Prerequisites

Requirement	Version	Notes
Rust	1.95+	Pinned in `rust-toolchain.toml`. Install via rustup.
Bun	latest	Used for frontend install + scripts. Prefer `bun` over `npm` in this repo. Install: `curl -fsSL https://bun.sh/install \| bash` (macOS/Linux) or `powershell -c "irm bun.sh/install.ps1 \| iex"` (Windows).
CMake	any recent	Required by default `local-ml` builds for `whisper-rs` and `llama-cpp-2`; not required for the cloud-only feature set.
C++ toolchain	C++17	Required by default `local-ml` builds. Use clang 10+ or gcc 9+ (Linux/macOS); MSVC via VS Build Tools 2022 "Desktop development with C++" workload (Windows).
clang / libclang	10+	Required by `bindgen` for FFI.

Platform-specific libraries

Linux (Debian/Ubuntu):

sudo apt install build-essential cmake clang libclang-dev \
    libpipewire-0.3-dev libspa-0.2-dev \
    libwebkit2gtk-4.1-dev libgtk-3-dev libayatana-appindicator3-dev librsvg2-dev

macOS: xcode-select --install then brew install cmake. Application-level capture requires macOS 14.4+ (Process Tap API).
Windows: Install VS Build Tools 2022 (Desktop C++ workload), CMake, and LLVM via winget (see Setup section for commands).

For build/capture issues, see the troubleshooting guide in the upstream rsac repository.

Build modes

The default Rust feature set includes local ML engines:

cd src-tauri
cargo +1.95.0 check --locked

Use the cloud-only feature set when you only need cloud providers such as Deepgram, OpenRouter, AWS, Gemini, or OpenAI-compatible endpoints:

cd src-tauri
cargo +1.95.0 check --locked --no-default-features --features cloud
cargo +1.95.0 test --locked --no-default-features --features cloud

Cloud-only builds omit whisper-rs, llama-cpp-2, and mistralrs. If a user selects Local Whisper, Local llama.cpp, or Mistral.rs in that build, commands return a structured provider_unavailable error naming the provider and the feature to enable (local-ml, asr-whisper, llm-llama, or llm-mistralrs).

Quick start

# 1. Clone AudioGraph. Cargo resolves the exact rsac v0.4.4 revision.
git clone https://github.com/Codeseys-Labs/audio-graph.git
cd audio-graph

# 2. Install frontend dependencies (use bun, not npm)
bun install

# 3. Download ML models (Whisper + optional extraction LLM)
./scripts/download-models.sh         # macOS / Linux
# .\scripts\download-models.ps1      # Windows PowerShell
# Or skip — models can be pulled in-app via the model manager.

# 4. Run in development mode (Tauri window + hot-reload)
bun run tauri dev -- --locked

The canonical dev command is bun run tauri dev -- --locked — this launches the Tauri shell with Vite hot-reload for the React frontend and cargo-rebuilds the Rust backend on change. bun run dev runs the Vite frontend only (no Tauri window) and is rarely what you want.

Current MVP workflow: configure Deepgram plus an enabled LLM, pick a supported source, click Start to begin capture, then click Transcribe. The transcript, notes, and knowledge graph update once the durable processing path is running. Native realtime providers remain visible for recovery/inspection but are not selectable for new MVP sessions yet.

Configuration

User settings

Non-secret app settings are stored in a user-level YAML file:

Linux/macOS : ~/.config/audio-graph/config.yaml
Windows     : %APPDATA%\audio-graph\config.yaml
              (e.g. C:\Users\<you>\AppData\Roaming\audio-graph\config.yaml)

Older installs that still have settings.json are imported on first launch when config.yaml is missing. Secrets are redacted out of settings during import and stay owned by the credential backend (OS keychain by default), never written into config.yaml.

Credentials (API keys)

Cloud provider API keys are saved locally and are not checked into the repo. Production desktop builds use the OS credential store by default: macOS Keychain, Windows Credential Manager, and Linux Secret Service. Use the in-app Settings page to add, replace, or clear saved keys; React receives saved/missing/source/readiness state, not stored plaintext values.

Older installs may still have a compatibility file at the OS config path:

Linux/macOS : ~/.config/audio-graph/credentials.yaml
Windows     : %APPDATA%\audio-graph\credentials.yaml
              (e.g. C:\Users\<you>\AppData\Roaming\audio-graph\credentials.yaml)

The first keychain migration wave keeps that file as a non-destructive import and recovery artifact. Headless/dev runs can opt into the file backend with AUDIO_GRAPH_CREDENTIAL_BACKEND=credentials_yaml; AUDIO_GRAPH_CREDENTIAL_BACKEND=keychain_with_file_fallback explicitly allows a keychain-to-file fallback for recovery testing.

Gemini Live reconnect / debugging

If Gemini Live drops its WebSocket, disconnects mid-session, or fails to reconnect, follow the Gemini reconnect runbook. It covers the gemini-status event, reconnect/backoff behavior, resumed-vs-fresh session status, and the manual recovery flow.

Pipeline config

Pipeline defaults (sample rate, turn-detection defaults, ASR model filename, graph parameters) are specified in src-tauri/config/default.toml. Current builds load this bundled TOML into typed defaults for the settings layer; runtime owners still opt in section by section as they are wired.

Model paths

Model	Purpose	Size	Location
`ggml-small.en.bin`	Whisper ASR	~500 MB	Tauri app data `models/` directory
`lfm2-350m-extract-q4_k_m.gguf`	Entity extraction + chat	~218 MB	Tauri app data `models/` directory

Local LLM with vLLM

AudioGraph talks to vLLM through the existing OpenAI-compatible API LLM provider. Run vLLM as a separate server, then set:

LLM provider: OpenAI-compatible API
Endpoint URL: http://localhost:8000/v1
Model: the vLLM model or --served-model-name
API key: blank for local unauthenticated servers, or the value passed to vllm serve --api-key

Example:

vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --host 127.0.0.1 \
  --port 8000 \
  --dtype auto \
  --served-model-name Qwen/Qwen2.5-1.5B-Instruct \
  --max-model-len 8192 \
  --generation-config vllm \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.85

For larger local reasoning, use the same command shape with an 8B/7B instruct checkpoint such as meta-llama/Llama-3.1-8B-Instruct or mistralai/Mistral-7B-Instruct-v0.3, sized to your GPU.

For CUDA graph performance, leave --enforce-eager off. That flag is a compatibility/debug fallback that disables CUDA graphs. See docs/ops/vllm-backend.md for warmup, remote GPU, API-key, and Windows/WSL2 notes.

Platform support matrix

Capture mode	Windows (WASAPI)	macOS (CoreAudio)	Linux (PipeWire)
System default	Yes	Yes	Yes
Specific device	Yes	Yes	Yes
Application (by PID)	Yes (process loopback)	Yes (Process Tap, macOS 14.4+)	Yes (pw-dump node)
Application (by name)	Yes (sysinfo → PID)	Yes (Process Tap, macOS 14.4+)	Yes (pw-dump → node serial)
Process tree	Yes	Yes (Process Tap, macOS 14.4+)	Yes

GPU acceleration	How to enable
macOS Metal	Automatic (default)
NVIDIA CUDA (Win/Linux)	`cargo +1.95.0 build --locked --features cuda`
Vulkan (Win/Linux, AMD/NVIDIA/Intel)	`cargo +1.95.0 build --locked --features vulkan`
CPU only	Default — no flags

Provider support at a glance

ASR: local Whisper, local Sherpa-ONNX (Zipformer, behind sherpa-streaming feature flag), Groq/OpenAI, AWS Transcribe, Deepgram, AssemblyAI.
LLM (extraction + chat): local llama.cpp, local Mistral.rs (Candle), OpenAI-compatible HTTP (OpenAI, OpenRouter, Ollama, LM Studio, vLLM, Together, Groq), AWS Bedrock.
Gemini Live: AI Studio API key or Vertex AI service account.
Planned realtime voice alternative: OpenAI Realtime (gpt-realtime-whisper for streaming STT and gpt-realtime-2 for speech-to-speech voice-agent mode).

See docs/ARCHITECTURE.md and docs/designs/provider-architecture.md for the full provider matrix and decision tree.

Setup (detailed, per-platform)

Windows — step by step

winget install Rustlang.Rustup
winget install Microsoft.VisualStudio.2022.BuildTools   # select "Desktop development with C++"
winget install Kitware.CMake
winget install LLVM.LLVM
powershell -c "irm bun.sh/install.ps1 | iex"

git clone https://github.com/Codeseys-Labs/audio-graph.git
cd audio-graph
bun install
.\scripts\download-models.ps1
bun run tauri dev -- --locked

For NVIDIA GPU acceleration: cd src-tauri && cargo +1.95.0 build --locked --features cuda.

macOS — step by step

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
xcode-select --install
brew install cmake
curl -fsSL https://bun.sh/install | bash

git clone https://github.com/Codeseys-Labs/audio-graph.git
cd audio-graph
bun install
./scripts/download-models.sh
bun run tauri dev -- --locked

Grant microphone permission when macOS prompts. Application-specific capture needs macOS 14.4+ (Sonoma).

Linux (Debian/Ubuntu) — step by step

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
sudo apt install build-essential cmake clang libclang-dev \
    libpipewire-0.3-dev libspa-0.2-dev \
    libwebkit2gtk-4.1-dev libgtk-3-dev libayatana-appindicator3-dev librsvg2-dev
curl -fsSL https://bun.sh/install | bash

git clone https://github.com/Codeseys-Labs/audio-graph.git
cd audio-graph
bun install
./scripts/download-models.sh
bun run tauri dev -- --locked

Development

bun run tauri dev -- --locked     # dev mode: Tauri window + hot-reload frontend + cargo rebuild
bun run tauri build -- --locked   # production bundle (installer / .app / .deb)
bun run dev               # frontend only (no Tauri window)
bun run typecheck         # tsc --noEmit
bun run test:local        # authoritative one-worker frontend tests

cd src-tauri && cargo +1.95.0 check --locked
cd src-tauri && cargo +1.95.0 test --locked -- --test-threads=1
cd src-tauri && cargo +1.95.0 clippy --locked --all-targets -- -D warnings

GPU-accelerated builds:

cd src-tauri && cargo +1.95.0 build --locked --features cuda       # NVIDIA CUDA 11.7+
cd src-tauri && cargo +1.95.0 build --locked --features vulkan     # Vulkan SDK
# macOS Metal: automatic, no flag needed

Documentation

The docs/ directory is organized by purpose:

docs/README.md — documentation index (start here).
docs/ARCHITECTURE.md — full architecture overview (fan-out realtime pipeline, event model, provider abstraction).
docs/designs/ — design proposals (provider architecture, provider refactor, session management).
docs/ops/ — operational runbooks (Gemini reconnect runbook).
docs/reviews/ — loop-by-loop code review notes, gap analyses, and UX first-run review.
docs/RELEASE.md — release process.
docs/MODEL_MANAGEMENT_DESIGN.md — model download + management.
docs/SETTINGS_DESIGN.md — settings page architecture.
docs/GEMINI_LANGUAGES.md — Gemini Live language support.
docs/SYSTEM_TRAY_WIDGET_PROPOSAL.md — tray widget proposal.

Releasing

AudioGraph consumes rsac v0.4.4 from the official Git repository at the full revision pinned in src-tauri/Cargo.toml. The manifest and committed src-tauri/Cargo.lock are the build/release source of truth; a sibling checkout is not required and must never be selected implicitly.

Run repository and release Cargo commands with Rust 1.95 and --locked. Developers changing both repositories can opt into a sibling checkout without editing the tracked manifest. Create the gitignored .cargo/rsac-local.toml:

[patch."https://github.com/Codeseys-Labs/rust-crossplat-audio-capture.git"]
rsac = { path = "../rsac" }

Pass the override explicitly from the repository root, omitting --locked because the local patch intentionally changes the committed resolution:

cargo +1.95.0 --config .cargo/rsac-local.toml \
  check --manifest-path src-tauri/Cargo.toml

Do not commit the override or a lockfile generated with it. Normal verification uses cargo +1.95.0 metadata --manifest-path src-tauri/Cargo.toml --format-version 1 --locked followed by locked checks and tests. Linux, macOS, and Windows release evidence must all report the same Cargo-resolved revision.

Contributing

See docs/CONTRIBUTING.md for branch workflow, commit conventions, code review expectations, and the pre-submit checklist.

License

Part of the rsac (Rust Cross-Platform Audio Capture) project. See the root LICENSE for details.

Name		Name	Last commit message	Last commit date
.github		.github
.seeds		.seeds
ci/storage-probe		ci/storage-probe
docs		docs
e2e		e2e
scripts		scripts
src-tauri		src-tauri
src		src
.gitattributes		.gitattributes
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
CONTEXT.md		CONTEXT.md
README.md		README.md
biome.json		biome.json
bun.lock		bun.lock
index.html		index.html
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json
tsconfig.node.json		tsconfig.node.json
vite.config.ts		vite.config.ts
vitest.config.ts		vitest.config.ts

Navigation Menu

Uh oh!

Repository files navigation

AudioGraph

Product Modes

Screenshots

Prerequisites

Platform-specific libraries

Build modes

Quick start

Configuration

User settings

Credentials (API keys)

Gemini Live reconnect / debugging

Pipeline config

Model paths

Local LLM with vLLM

Platform support matrix

Provider support at a glance

Setup (detailed, per-platform)

Development

Documentation

Releasing

Contributing

License

About

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

AudioGraph

Product Modes

Screenshots

Prerequisites

Platform-specific libraries

Build modes

Quick start

Configuration

User settings

Credentials (API keys)

Gemini Live reconnect / debugging

Pipeline config

Model paths

Local LLM with vLLM

Platform support matrix

Provider support at a glance

Setup (detailed, per-platform)

Development

Documentation

Releasing

Contributing

License

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages