Add VoxCPM2 as a TTS engine, in pure Rust - #75
Draft
JRufer wants to merge 2 commits into
Draft
Conversation
VoxCPM2 (OpenBMB, Apache-2.0) is a 2B-parameter multilingual speech model offering the same two ways to pick a voice that Breeze-TTS-2 does — natural language voice design and cloning from a reference clip — plus style control layered on a cloned voice. The settings UI mirrors the Breeze-TTS-2 layout. The upstream reference implementation is Python and PyTorch, which has no place in this app. Synthesis instead runs in process through the pure-Rust `voxcpm-rs` crate on Burn, the same shape of dependency as the `pocket-tts` crate already behind Pocket-TTS and Breeze-TTS-2. No Python, no subprocess, no ONNX Runtime. Latency, targeting first audio under a second: - The loaded model is cached for the TTS worker thread's lifetime, so the 20-25 s checkpoint load is paid at most once. Prewarm defaults to on and moves it to startup, and warms with a real short generation so the first spoken utterance does not pay for GPU shader compilation either. - Generation is streamed into the rodio sink chunk by chunk, so the wait is the first chunk rather than the whole utterance. - chunk_patches defaults to 2 rather than the crate's 5, halving the patches generated before playback starts (~80 ms of audio each). - inference_timesteps defaults to 6 rather than upstream's 10, the floor of the range that keeps quality intact. - Reference clips are decoded and resampled once, then cached as raw PCM. - Stopping hands the generator a cancel token polled between diffusion steps, so the stop key abandons the current step instead of finishing the chunk. The GPU (wgpu) backend is compiled by default; voxcpm2-cpu selects the portable ndarray backend. A build with neither still downloads the model and saves its settings, and Settings says so rather than failing on the first utterance. Covered by 15 Rust tests over readiness, prompt composition and voice-mode selection, and 12 Svelte tests over the settings section. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QactcDkKLabdkXqys9uyMm
Playback appended one rodio SamplesBuffer per generated chunk and started on the first one. Two things follow from that, and both were audible. The sink's queue runs dry whenever the next chunk is not finished when the previous one stops playing, and rodio does not wait for it: SourcesQueueOutput splices in a filler block of silence and carries on. Playback consumes audio in real time while generation does not produce it in real time, so on any machine where the model is not comfortably faster than playback, speech stalled and resumed over and over. A bigger chunk_patches only bought a longer run between gaps, which is exactly the symptom reported. Every chunk was also its own entry in the sink's queue, making each chunk boundary a seam in the audio pipeline rather than a continuous stream. Both are fixed: - Audio now reaches the sink as a single continuous StreamingPcm source per utterance, fed over a channel. There is no per-chunk queue boundary left, and a starved sink degrades to brief silence inside one stream instead of a queue transition. It never blocks the audio thread, and dropping the sender is what ends the source, so sleep_until_end still returns. - Playback waits for a lead buffer instead of starting on the first chunk. The engine measures the real-time factor of generation from the second chunk onward, excluding the one-off prefill so the figure is the steady-state rate. Under RTF 0.8 the buffer grows by itself once playback starts, so the configured 400 ms lead is already safe. At or above it, the lead is extended to cover (RTF - 1) * remaining with a jitter margin. If generation finishes before the lead is reached, the utterance is played complete in one buffer. The lead is exposed as a Lead Buffer slider, documented as the control to raise when speech breaks up. chunk_patches is no longer a latency control at all — it governs throughput, since AudioVAE decode work scales as O(N^2/chunk_patches) — so its default rises from 2 to 4, which makes generation faster and the lead easier to hold. Adds 7 tests over lead sizing: faster-than-realtime keeps the minimum, slower demands proportional cover, the lead is monotonic in both slowness and remaining audio, never drops below the configured floor, and survives a degenerate rate measurement rather than never playing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QactcDkKLabdkXqys9uyMm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds VoxCPM2 as a sixth TTS engine, with voice design and voice cloning behind a settings section that mirrors the Breeze-TTS-2 one.
No Python
The upstream VoxCPM2 reference is Python + PyTorch. This does not use it. Synthesis runs in process, in pure Rust through the
voxcpm-rscrate on Burn — the same shape of dependency as thepocket-ttscrate already behind Pocket-TTS and Breeze-TTS-2. No Python interpreter, no subprocess, no ONNX Runtime anywhere in this path.VoxCPM2 is Apache-2.0 and the weights are ungated, so unlike Breeze-TTS-2 and Pocket-TTS there is no access token and no non-commercial restriction.
Voice selection
(description)prefix on the text, which the engine composes..wavfrom the voice folder already shared with Pocket-TTS and Breeze-TTS-2, so a clip dropped in once works with every engine. An optional style instruction shapes delivery without changing the identity. A sibling<name>.txttranscript upgrades cloning from imitation to continuation, which tracks the reference speaker more closely.Latency
The target was first audio under a second. In descending order of impact:
chunk_patchesdefaults to 2, not the crate's 5 — patches generated before playback starts, each roughly 80 ms of audio and one autoregressive step. This is the direct time-to-first-sound knob.inference_timestepsdefaults to 6, not upstream's 10 — the floor of the range that keeps quality intact, and linear in generation cost.Stopping is prompt: the generator gets a cancel token that the model polls between diffusion steps, so the stop key abandons the current step rather than finishing the chunk. First-chunk latency is logged per utterance and surfaces in Settings as Run speed.
Build flags
The GPU backend (
wgpu: Vulkan / Metal / DX12, no vendor SDK) ships in the default build, because a 2B model on CPU is far slower than realtime and cannot meet the target.voxcpm2-cpuselects the portable ndarray backend. A build with neither still downloads the model and saves its settings; only synthesis is unavailable, and Settings says so up front rather than failing on the first utterance.Two defects found and fixed during self-review
max_lencap was never passed to the generator, so a pathological input could generate unbounded speech.Testing
cargo test -p voxctrl-configcargo test -p voxctrl-tts --features voxcpm2-cpucargo test -p voxctrl-app --features voxcpm2npx vitest runnpx svelte-checkcargo clippyon the new codeNew coverage: 15 Rust tests over checkpoint readiness (including that an in-flight
.partfile never counts as ready), prompt composition and voice-mode selection; 12 Svelte tests over the settings section, covering both voice modes, missing-file reporting, and every reason Test TTS is blocked.The full-workspace build could not be verified here:
ort-sysdownloads ONNX Runtime at build time and that host is blocked by this environment's egress proxy, a pre-existing constraint the repo documents. The Tauri app was verified to compile with--no-default-features --features custom-protocol,voxcpm2.Docs
docs/tts.md(engine notes and the latency tuning guide),docs/configuration.md(thevoxcpm2sub-object),docs/api.md(voxcpm2_status,download_voxcpm2),docs/architecture.md, and the README feature list plus a new build-flags section.🤖 Generated with Claude Code
https://claude.ai/code/session_01QactcDkKLabdkXqys9uyMm
Generated by Claude Code