Real-time captions for hybrid rooms and events. The desktop app captures a presenter’s microphone, Windows system audio (whatever is playing — Zoom, Teams, a browser tab, a media player), or both and renders captions in a transparent, always-on-top overlay. Built for the Digital Humanities and Artificial Intelligence in African Studies / Humanités numériques et intelligence artificielle en études africaines workshop (STIAS, Stellenbosch, 21–24 September 2026).
It has two deliberately separate modes:
- Live translation — speech is auto-detected and translated into English or French by
Google Gemini (
gemini-3.5-live-translate-preview) or OpenAI (gpt-realtime-translate). Their generated audio is discarded; only transcript text is displayed. Gemini also captions speech that is already in the selected target language, so mixed-language meetings do not go blank during same-language passages. - Subtitles — a built-in English/French product demonstration requires no setup, while
real-time same-language speech recognition uses Mistral Voxtral Mini Transcribe Realtime
(
voxtral-mini-transcribe-realtime-2602) or Google Gemini (gemini-3.5-transcribe-live). Both detect the spoken language themselves; Gemini covers over 70 languages and cleans fillers and false starts out of the subtitle, Voxtral is flat-rate and has no session length limit. The transcript can be saved as plain.txtor Markdown; SRT/VTT export and the native Windows Save As dialog are available in version 1.2.3.
The app opens on a deterministic Built-in demo: no publisher key, account, microphone,
language pack, network, or per-minute charge. It drives the real caption UI, overlay, elapsed
timer, level meter, transcript and export path using clearly labelled bundled scripted
content; it does not recognize live speech. Live microphone and system subtitles use Mistral
or Gemini, and live translation uses Gemini or OpenAI with your own provider key.
docs/microsoft-store.md explains why that split is what made
Microsoft Store distribution possible.
How the app handles audio, provider keys and transcripts is set out in
docs/privacy.md, published at
https://fmadore.github.io/Live-translation/privacy.
Provider details and verified wire formats are documented in
docs/gemini-live-api.md,
docs/openai-realtime-api.md, and
docs/mistral-realtime-api.md.
Version 1.2.3 adds native Windows Save As, SRT/VTT transcript export, and capture of one selected application and its child processes. The user confirmed Save As and application selection in the ARM64 test package. The Store update is submitted separately through Partner Center; release preparation records remaining testing and screenshot work.
Get it from the Microsoft Store — native x64 and ARM64, signed by Microsoft, and it updates itself. This is the recommended route.
The releases page also carries an unsigned x64 NSIS installer and MSI. Being unsigned, they meet a SmartScreen "Windows protected your PC" warning on first launch — choose More info → Run anyway.
Either way you need Windows 11 and the Microsoft Edge WebView2 Runtime, which current Windows 11 installs already have.
While the app is running it keeps an icon in the notification area, so a live session stays reachable — open the window, show or hide the overlay, stop the session, or quit — with the operator window out of the way. Closing the window quits the app unless you turn on Keep running in the tray when I close this window.
Tauri app (Rust core + SvelteKit front-end)
├── Audio capture
│ ├── Microphone — cpal
│ └── System audio — WASAPI loopback on Windows (any app's output)
│ └── mono PCM16: 16 kHz Gemini/Mistral or 24 kHz OpenAI
├── Bounded realtime pipeline — one capture + WebSocket session per source
│ ├── Gemini/OpenAI → translated transcript captions
│ └── Mistral → same-language subtitle captions
├── Built-in deterministic demo → caption/level/status events without capture or network
└── Windows
├── Operator — mode/source/provider controls, meters, monitor, export
└── Overlay — transparent, always-on-top, click-through captions
The shared runner provides connection timeouts, bounded queues, stale-audio discard,
exponential reconnect backoff, provider-error classification, turn isolation, and graceful
provider flushes. Keys remain in Windows Credential Manager and are used only by Rust. See
docs/architecture.md for the complete flow.
Windows only. System-audio capture is WASAPI loopback and the app is not built or
released for any other platform. The Linux lane in CI is a compile check for the
non-cfg(windows) code, not a supported target.
- Windows 11
- Node.js 24 LTS and npm (Node.js 22.12+ remains CI-tested)
- Stable Rust
- Tauri prerequisites for Windows
- No key is needed for the built-in demonstration. Live modes need the corresponding provider key:
- Google AI Studio for Gemini translation
- OpenAI for OpenAI translation
- Mistral Studio for Mistral subtitles
- Gemini subtitles reuse the same AI Studio key as Gemini translation — save it once
Every provider bills per minute of streamed audio, so the meter runs for as long as a session is open. Rates verified 27 August 2026 against Gemini, OpenAI and Mistral pricing.
| Mode | Model | Rate | One hour |
|---|---|---|---|
| Translation | gemini-3.5-live-translate-preview |
$0.0053/min audio in + $0.0315/min audio out | $1.25–2.21 |
| Translation | gpt-realtime-translate |
$0.034/min | $2.04 |
| ↳ source monitor | gpt-realtime-whisper |
$0.017/min | $1.02 |
| Translation | OpenAI total | $3.06 | |
| Subtitles | voxtral-mini-transcribe-realtime-2602 |
$0.006/min | $0.36 |
| Subtitles | gemini-3.5-transcribe-live |
$0.005/min audio in + $0.004/min text out | $0.30–0.54 |
| Built-in caption demonstration | bundled scripted content | free | $0.00 |
Both Gemini rows work the same way: the input leg is billed on the full wall clock because
silence stays in the stream, while the output leg accrues only while the model is producing
something — pauses, slide changes and Q&A gaps lower the bill, hence the ranges. For
translation that output is audio, charged even though the app discards it; for subtitles it is
the transcript text, which is far cheaper. OpenAI is duration-billed and therefore flat:
silence costs the same as speech, and the gpt-realtime-whisper source transcription that
feeds the operator monitor is a separate charge on top. Voxtral is likewise flat.
The two subtitle engines cost about the same, so choose on behaviour rather than price.
Gemini covers over 70 languages, detects the spoken one per utterance, and applies Google's
smart transcription — fillers, false starts and spoken self-corrections are cleaned out, and
punctuation and casing are applied. Against that, its live sessions cap at ten minutes, so a
long room session reconnects several times an hour with a one- to two-second caption gap each
time; and it decides its own segment boundaries, which for a speaker who rarely pauses can
mean a caption that grows to a paragraph before it settles. Voxtral has no session cap and
finalizes on short pauses, so it produces shorter, more even lines. Both behaviours are
measured rather than assumed — see docs/gemini-live-api.md.
Selecting Both as the source doubles every figure — the pipeline opens one capture and one WebSocket session per origin.
Scaled to the workshop, the programme's captionable sessions (panels, keynotes, discussions and plenaries — the Day 2 and Day 3 afternoons are excursions) total 18 hours, or about 21 hours of wall clock if the session is left running through coffee breaks. Translation over those hours costs roughly $25–45 on Gemini or $55–65 on OpenAI for a single source, doubling to about $50–90 and $110–130 if the room microphone and the Teams feed are both captioned throughout.
Both translation models are preview-tier; re-check the rates alongside the wire formats before the event.
npm install
npm test
npm run check
npm run build
# Frontend preview only (no native capture)
npm run dev
# Full desktop app
npm run tauri devThe operator stores each provider key separately in Windows Credential Manager. For
development, copy .env.example to an uncommitted .env
and set GEMINI_API_KEY, OPENAI_API_KEY, or MISTRAL_API_KEY.
npm test runs two vitest projects. Pure logic runs in node — faster, and a test cannot
start depending on the DOM by accident. Anything that renders a component opts in to
jsdom by being named *.svelte.test.ts; those get Testing Library, jest-dom matchers and
automatic cleanup via vitest-setup-client.ts.
User-facing text belongs in the message catalogs, not in a component:
docs/localization.md has the contract — the interface language is
independent of the caption language, the Rust core names failures and the catalog words them,
and npm run check plus npm test fail on anything missing.
UI work has one more standard to meet: docs/accessibility.md has
the contrast, focus, heading, announcement and contrast-theme rules, which of them
npm test enforces on its own, and the manual Narrator / contrast-theme / text-scaling walk
that belongs to a release.
Before opening a PR that touches the UI, run the browser-preview smoke test: npm run dev,
load http://localhost:5173, and check the console is clean. The operator window degrades
deliberately without a Tauri runtime, and this catches the class of bug where a component
invokes a Tauri command before the app knows it is running in a browser. Note that this
preview cannot exercise audio capture, provider sessions or the overlay — anything touching
those needs npm run tauri dev on Windows.
Build Windows installers with npm run tauri build. The Store packages are built separately —
npm run bundle:msix:x64 and npm run bundle:msix:arm64 write an unsigned .msix each, which
CI then combines into the multi-architecture bundle described in
docs/packaging-msix.md.
- Choose Live translation or Subtitles, then select the audio source. For translation, select the target language and Gemini/OpenAI provider — F2 swaps the direction, but only before you start, because the provider is given the target once at session start. (Subtitles opens on the built-in demonstration; switch the provider to Mistral for live speech.)
- Start the session and confirm the source meter and live monitor move.
- Use Move overlay to position/resize captions on the projector, then lock it back into click-through mode. In move mode the overlay itself has the keyboard: Enter locks it in place, Esc cancels and restores where it was, arrow keys nudge by a pixel (Shift for 10), and +/− resize the text. Hide overlay blanks the captions mid-session (a video clip, a coffee break) without stopping anything. For system audio or Both, System capture also offers One application. Select an open application window, then run the audio test. This captures its process tree, not an individual browser tab. Application capture requires Windows build 20348 or later; unsupported systems retain the existing output capture option. See application capture for scope and testing limits.
- Choose Markdown, plain text, WebVTT, or SubRip in the transcript's format selector, then Save as…. The native Windows dialog lets you choose the folder and filename, confirms overwrites, and remembers the last successful destination folder. Timed exports require caption timing; older untimed recovery files can still be saved as text or Markdown. See transcript export.
- Rehearse the real Zoom + room-microphone + projector chain before the event. The realtime provider surfaces should be re-verified shortly beforehand.
src/ SvelteKit operator and overlay windows
lib/ApiKeyPanel.svelte provider key management
lib/TranscriptMonitor.svelte monitor and text/Markdown/SRT/VTT export
lib/transcript.ts pure export formatting (unit tested)
src-tauri/src/audio/ capture, metering, resampling
src-tauri/src/realtime.rs shared WebSocket lifecycle
src-tauri/src/{gemini,openai,mistral}/ provider protocols
src-tauri/src/ondevice/ deterministic built-in caption demonstration
.github/workflows/ CI, tag-driven releases, Store submission
Pull requests and main pushes run frontend tests/type-check/build/audit, Rust format,
Clippy and tests on Linux and Windows, RustSec, and actionlint. Dependabot checks npm, Cargo,
and GitHub Actions weekly.
Pushing a v* tag builds the installers, both architectures' MSIX, and the multi-architecture
.msixbundle the Store submission uses. That bundle is then uploaded to Partner Center by
hand: the submission API is a Company-account feature, and this is an Individual account —
see docs/store-updates.md.
MIT — see LICENSE.