Sync bridge with current Voice Agent API, validated end-to-end via Bluejay - #2
Conversation
Verified against the live AssemblyAI docs and assemblyai-api-spec:
- WebSocket URL: /v1/realtime → /v1/ws (renamed April 2026)
- Voice catalog: replace retired voices (ivy, james, ...) with the
current documented set (alba, eve, ... anna, charles) plus a
LANGUAGE_SPECIFIC_VOICES dict (giovanni, lola, juergen, rafael,
estelle); GET /v1/voices is the authoritative live list
- Send session.end on hangup instead of just closing the socket —
closing alone keeps the session billable for a 30s resume window
- Buffer input.audio until session.ready per the documented event flow
- Pass is_error on tool.result so the agent knows a tool call failed
- Expose the newer session.input/output tuning surface in
agent_config.py: transcription_mode, transcription_prompt,
language_codes, voice_focus(+threshold), turn_detection
(vad_threshold, min/max_silence, interrupt_response,
interruption_delay), and output volume
- Support stored agents via AAI_AGENT_ID env var (binds the session
with {"agent_id": ...} instead of inline config)
- Handle input.speech.stopped and session.ended (logs billed durations)
- README: document all of the above, incl. EU endpoint host
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found during a live end-to-end smoke test against the Voice Agent API: - mcp SDK 2.0 renamed streamablehttp_client to streamable_http_client and now yields a (read, write) 2-tuple; requirements.txt had mcp unpinned so fresh installs were already broken. Pin mcp==2.0.0 and use the new API. - The docs MCP server at mcp.assemblyai.com/docs is deprecated (it now prepends a migration warning to every result and was slated for shutdown July 16, 2026). Point at https://www.assemblyai.com/docs/mcp and map the agent-facing tools (search_docs, read_docs_page) onto its tools (search_assembly_ai, query_docs_filesystem_assembly_ai). Cap tool results at 8k chars to keep voice reply latency low. - Procfile: python -u so deploy logs stream instead of block-buffering. - Handle null audio_duration_seconds in the session.ended log line. Smoke-tested live: greeting, STT transcription, tool calls, barge-in (interrupted transcript), and clean session.end teardown all verified against wss://agents.assemblyai.com/v1/ws. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- aiohttp 3.9.4 -> 3.14.3 (several CVEs fixed since 3.9.4; re-validated with a live end-to-end session on 3.14.3) - main.py docstring still claimed scipy resampling; it's audioop.ratecv - audioop was removed in Python 3.13, not 3.14 (PEP 594) - agent_config.py docstring referenced an internal repo and a re-sync workflow that no longer applies - README: generic example credentials Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![aikido-pr-checks[bot]](https://avatars.githubusercontent.com/in/898896?s=60&v=4){kind=small}
| "type": "input.audio", | ||
| "audio": base64.b64encode(pcm24k).decode(), | ||
| }) | ||
| if not session_ready.is_set(): |
There was a problem hiding this comment.
pre_ready_buffer is only flushed when a later audio frame arrives; if session.ready comes after speech ends, buffered audio is never sent and user speech is dropped.
Details
✨ AI Reasoning
The new buffering logic is trying to preserve input.audio frames that arrive before the upstream session is ready. However, those buffered frames are only flushed when another binary frame arrives after readiness. If session.ready is received after the user has already finished speaking, no later frame triggers that flush path, so previously buffered speech is silently dropped. This contradicts the stated behavior of buffering until ready and can produce missing user audio in normal timing scenarios.
🔧 How do I fix it?
Trace execution paths carefully. Ensure precondition checks happen before using values, validate ranges before checking impossible conditions, and don't check for states that the code has already ruled out.
Reply @AikidoSec feedback: [FEEDBACK] to get better review comments in the future.
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info
2 participants
What
Brings the bridge up to date with the current AssemblyAI Voice Agent API so customers can run their own Bluejay simulations against it, verified against the live docs (
assemblyai.com/docs) and theassemblyai-api-specrepo.API sync (
ceecf2e)/v1/realtime→/v1/ws(renamed April 2026)alba…vera, plus opt-in language-specific voices). Verified live: the oldivy-era names still work as undocumented legacy values, but they no longer appear in the docs or voice list, so new deployments shouldn't rely on themsession.endon hangup — just closing the socket leaves a 30-second billable resume window, which adds up fast across simulation runsinput.audiountilsession.readyper the documented event flowis_errorontool.resultagent_config.py:turn_detection(vad threshold, silence windows, barge-in, interruption delay),transcription_mode,transcription_prompt,language_codes,voice_focus, outputvolumeAAI_AGENT_IDenv varFixes found during live testing (
3c19a48)mcpwas unpinned and 2.0 renamedstreamablehttp_client→ fresh installs were broken; pinnedmcp==2.0.0and updated to the new APImcp.assemblyai.com/docs, past its announced shutdown date) tohttps://www.assemblyai.com/docs/mcp, mapping agent-facing tools onto its new tool names; tool results capped at 8k chars for voice latencyProcfileusespython -uso deploy logs streamPublish polish (
9ac4f73)aiohttp3.9.4 → 3.14.3 (multiple CVEs fixed since 3.9.4; re-validated live)Validation
(interrupted)transcript, cleansession.endteardown — all against the live APIOpen decisions (not in this PR)
🤖 Generated with Claude Code