AI Role Player — Realtime Voice Demo

English | 简体中文

Important

Live demo: https://aia.jiangwenjie.cn

A single-repository React + Node/TypeScript application for configurable realtime voice sales role-play. Learners choose an SQLite-backed sales scenario, compatible customer persona, and difficulty; the browser then connects its microphone to Qwen qwen-audio-3.0-realtime-plus through a server-side WebSocket gateway, streams transcripts into a chat timeline, and plays the selected persona's voice. Finalized conversations, heard audio, launch snapshots, pause state, and active practice time are stored in SQLite and listed in responsive history navigation. Active sessions can be paused, continued through a fresh Qwen connection, or restarted with the same configuration; ended sessions become immutable and receive asynchronous Qwen text-model coaching, optional weighted scoring, highlighted moments, and a reviewable transcript. Conversations can be downloaded as a transcript, one alternating-speaker MP3, or a ZIP containing both. A responsive admin console provides persona/scenario CRUD and an inspectable model-Instructions preview.

The UI uses one responsive React component tree for mobile and desktop. Ant Design supplies the standard controls and theme algorithms; project CSS handles the chat layout, message bubbles, recording overlay, and audio-reactive visuals. In addition to push-to-talk, the composer supports click-to-start long recording and a hands-free free-conversation mode that automatically submits a turn after sustained silence.

Repository shape

This is one root package, not a monorepo. Client, server, tests, and shared protocol definitions use the same package.json, lockfile, ESLint configuration, TypeScript setup, and .gitignore.

.
├── public/                         # Browser AudioWorklet modules
├── scripts/
│   ├── initialize-catalog.ts       # Explicit idempotent catalog defaults
│   ├── split-database.ts           # One-time legacy database splitter
│   └── smoke-realtime.ts           # Live realtime smoke harness
├── src/
│   ├── client/
│   │   ├── app/                    # Root application controller, route renderer, theme/errors/previews
│   │   ├── admin/                  # Persona/scenario management console
│   │   ├── audio/                  # Microphone capture and streamed playback
│   │   ├── catalog/                # Catalog API and selection state
│   │   ├── components/             # Shared header, chat messages, waveform, and voice orb
│   │   ├── conversations/          # History, feedback, download APIs, rail/mobile Drawer
│   │   ├── i18n/                   # Locale state, persistence, Ant Design locale
│   │   ├── learner/                # Scenario/persona/difficulty launcher
│   │   ├── realtime/               # WebSocket client, runtime projection/recovery, settlement
│   │   ├── routing/                # SPA URL parsing and History API coordination
│   │   ├── session/                # Active-session views, controls, and lifecycle operations
│   │   └── voice/                  # Hold, long-recording, and hands-free controllers
│   ├── server/
│   │   ├── catalog/                # Catalog repository, CRUD/generation routes, initializer
│   │   ├── conversations/          # History, export, feedback, and goal-detection services
│   │   ├── database/               # SQLite lifecycle and migrations
│   │   └── realtime/               # Qwen gateway and context repair
│   └── shared/                     # Protocol, catalog/history/feedback schemas, prompt compiler
├── test/                           # Unit and adapter tests
├── docs/                           # Architecture and engineering contracts
├── index.html
├── eslint.config.js
├── vite.config.ts
└── package.json

pnpm-workspace.yaml contains only pnpm's dependency-build allowlist for esbuild. It does not define workspace packages.

Prerequisites

• Node.js 22.13.0 or newer (node:sqlite is used directly)
• pnpm 11 or newer
• An Alibaba Cloud Model Studio API key in the China (Beijing) region
• A Model Studio Workspace ID with access to qwen-audio-3.0-realtime-plus

Official setup references:

• Get an API key
• Get a Workspace ID
• Qwen Audio Realtime guide
• Qwen OpenAI-compatible chat completions
• Qwen structured JSON output

Setup

1. Install dependencies:

   pnpm install

2. Create a local environment file:

   cp .env.example .env

3. Configure .env. Add the Beijing-region credentials before starting a voice session:

   DASHSCOPE_API_KEY=sk-ws-...
   DASHSCOPE_WORKSPACE_ID=ws_...
   DASHSCOPE_FEEDBACK_MODEL=qwen3.6-flash

   DASHSCOPE_FEEDBACK_MODEL is shared by feedback generation, catalog draft generation, and conservative success detection. All three use non-thinking JSON mode and validate the returned structure on Node.

   SQLite uses two files by default: data/catalog.sqlite for personas, scenarios, presets, and compatibility; and data/conversations.sqlite for conversation snapshots and finalized messages. Override them with CATALOG_DATABASE_PATH and CONVERSATION_DATABASE_PATH. Relative paths resolve from the process working directory and parent directories are created automatically.

   If upgrading an existing checkout that still has data/role-player.sqlite, stop the server, wait for its -wal/-shm files to disappear, then run pnpm database:split once. The command preserves the legacy source and refuses to overwrite either destination.

4. Initialize the database-backed bilingual persona/scenario choices and starter catalog:

   pnpm catalog:init

   This command opens CATALOG_DATABASE_PATH, applies pending schema migrations, and transactionally inserts missing bilingual catalog defaults from JSON. SQLite generates numeric IDs; stable JSON seed keys and conflict-tolerant writes make repeated runs safe without duplicating data or overwriting administrator edits. It does not require Qwen credentials. A run that reports only skipped rows succeeded and simply had nothing new to insert.

5. Start the React and Node development servers:

   pnpm dev

6. Open http://localhost:5173, choose a training scenario, compatible persona, and difficulty, then select Start voice role-play and allow microphone access. Use the left history rail on wide screens—or its header Drawer button on smaller screens—to reopen an active session or review an ended one. The admin console has its own route at http://localhost:5173/admin. Active conversations use /chat/:conversationId; ended-session feedback uses /feedback/:conversationId. Refreshing either address reloads its durable data. The interface starts in English; use the upper-right language control to switch to Chinese.

7. Hold Hold to talk while speaking. Release to send, or slide upward at least 72 px before releasing to cancel. While the selected persona is speaking, the control changes to Hold to interrupt and talk; holding it stops the current playback, begins context reconciliation, and records the next turn. The Chinese interface uses the equivalent 按住说话 and 按住打断并说话 labels.

Do not paste a real API key into source code, commit it, or expose it through a VITE_* variable.

UI-only development previews

When reviewing layout without granting microphone permission or opening a Qwen session, use the development-only fixtures below while Vite is running:

• http://localhost:5173/?preview=session — populated conversation while Alex is speaking
• http://localhost:5173/?preview=paused — persisted paused state with the continue action
• http://localhost:5173/?preview=recording — active recording waveform and composer spacing
• http://localhost:5173/?preview=long — click-to-start long recording with end/cancel actions
• http://localhost:5173/?preview=free — transcript-free voice-orb conversation surface

These URLs reuse the production React components but inject static in-memory state. Voice controls are intentionally not functional there, and preview is ignored in a production build.

Commands

Command	Purpose
pnpm dev	Run Vite and the Node server together
pnpm dev:client	Run only the React development server
pnpm dev:server	Run only the Node TypeScript server
pnpm catalog:init	Apply migrations and idempotently add missing presets/starter personas using TypeScript sources
pnpm catalog:init:prod	Run the built initializer against the deployment database before starting the built server
pnpm database:init:prod	Apply both built migration chains and idempotently initialize catalog data for a deployment
pnpm database:split	One-time copy from the legacy combined database into fresh catalog and conversation files
pnpm database:split:prod	Run the built one-time database splitter
pnpm lint	Run the shared ESLint configuration
pnpm typecheck	Type-check client, server, and shared code
pnpm test	Run all tests once
pnpm smoke:realtime <pcm-file> [interrupt flag]	Exercise a normal or interrupted live Qwen turn through the local Node gateway
pnpm build	Build the Node server/initializer to dist/server and React to dist/client
pnpm check	Run lint, type-check, tests, and both builds

Docker deployment

Build the single production image from the repository root. When building on Apple Silicon for a typical x86_64 Linux server, select the target platform:

docker buildx build \
  --platform linux/amd64 \
  -t ai-role-player:latest \
  --load .

On an x86_64 server itself, the shorter equivalent is:

docker build -t ai-role-player:latest .

The image build applies every catalog and conversation migration and inserts the bilingual catalog defaults into /app/data. A newly created Docker volume is populated from those image files on first use. The runtime entrypoint repeats the same deployment initializer so an existing volume or bind mount also receives later migrations and any missing seed rows; this is idempotent and does not overwrite administrator edits.

Direct HTTPS with mounted certificates

compose.yaml terminates TLS directly in Fastify. Prepare a PEM certificate chain and its matching PEM private key on the deployment host:

certs/fullchain.pem
certs/privkey.pem

fullchain.pem must contain the server certificate plus any required intermediate certificates; a raw public-key file is not sufficient. The files are ignored by Git and Docker builds and are mounted read-only at runtime. The container runs as UID/GID 1000, so that user must be able to read both files.

Add these deployment values to .env alongside the DashScope credentials:

PUBLIC_ORIGIN=https://role-player.example.com
TLS_CERT_HOST_PATH=./certs

PUBLIC_ORIGIN must be the exact browser origin and must match the names in the certificate. Then start the already-built image:

docker compose up -d

Inspect startup and TLS-file errors with:

docker compose logs -f role-player

Open the configured PUBLIC_ORIGIN. Compose publishes host port 443, mounts both SQLite databases in the named role-player-data volume, and mounts the certificate directory at /app/certs:ro. Fastify serves HTTPS and WSS on the same port. The entrypoint applies both database migration chains and runs the idempotent bilingual catalog initializer before every server start; later starts preserve administrator edits. The image health check detects TLS automatically and calls /api/health internally without weakening browser-side certificate validation.

After replacing either PEM file, restart the process to load it:

docker compose restart role-player

Keep DASHSCOPE_API_KEY and DASHSCOPE_WORKSPACE_ID in runtime environment variables or a secret manager, never in the image. Do not omit the /app/data volume: replacing a container without it loses catalog changes, conversations, audio, and feedback. A self-signed certificate is suitable only when its root CA is explicitly trusted on every learner device; otherwise browsers reject HTTPS/WSS and microphone access.

Optional live smoke test

With pnpm dev:server running, send any headerless PCM16, 16 kHz, mono recording through the same gateway used by the SPA:

pnpm smoke:realtime /absolute/path/to/input.pcm

The command creates a normal durable conversation through the local REST API, then succeeds only when it receives a persisted user transcript, an assistant transcript, streamed assistant audio, and the response-specific response.persisted acknowledgement after simulated playback completion. Its finalized text therefore appears in the history list. It never reads the Qwen credentials; those remain inside the Node server process.

Add --interrupt to wait until generation finishes, simulate stopping queued playback partway through, and verify Qwen acknowledges the assistant-item delete/recreate repair transaction:

pnpm smoke:realtime /absolute/path/to/input.pcm --interrupt

Use --interrupt-during-generation to exercise the cancellation path. With no trusted speech-rate history yet, this case must delete the partial assistant item and conservatively retain no estimated text.

Current behavior

• One responsive Ant Design SPA for learner launch, admin catalog, and voice chat on mobile and desktop; no separate mobile application or duplicated component tree
• Browser routes for the learner launcher (/), admin console (/admin), refreshable active conversations (/chat/:conversationId), and ended-session reviews (/feedback/:conversationId)
• Fixed global utility bar with product identity on the left, language/theme controls on every route, and a textual admin entry everywhere except the admin console itself
• English and Chinese UI with English as the first-run default, Ant Design locale synchronization, and the saved role-player:locale preference in localStorage
• Light and dark themes, initialized from the saved choice or OS preference and switchable without resetting the current surface
• Learner launcher with searchable scenario/persona selectors, compatibility filtering, Ant Design easy/medium/hard Radio buttons, preparation-focused scenario/persona cards (goals, skills, success criteria, background, personality, communication style, behavior notes, motivations, and concerns), and the exact compiled Instructions preview with an enforced actual/12000 budget
• Responsive admin console with independent persona/scenario editors, separate compatibility management, optional derived scoring weights, standalone Instructions previews, and bilingual editable random drafts generated by the configured Qwen text model
• Database-backed bilingual persona presets plus scenario presets for training goals, skill focus, success criteria, and tone style; no persona/scenario business options are authored in the client
• Independent Chinese/English fields for every localized persona/scenario value; the current language is displayed first with fallback to the other, while admin saves update only the language being edited and never persist fallback text as a translation
• Fully bilingual JSON-defined starter personas/scenarios loaded into SQLite; user-authored content is never machine-translated
• Free-form persona name, age, background, and behavior notes, with existing non-preset values preserved when editing older/custom personas
• Persona owns reusable character attributes and the Qwen voice. Scenario owns situation plus optional goals, skills, success criteria, derived scoring weights, and tone/pace/interjection behavior
• Deterministic bilingual compileRolePlayInstructions templates selected by the current UI/session locale; no extra LLM is called to turn structured catalog fields into the Qwen system prompt
• Shared 12,000-character Instructions budget, checked across every compatible persona and all three difficulty levels before an association can be saved
• Session-start snapshot sends the selected persona's voice and the compiled persona/scenario/difficulty Instructions to Qwen, so later catalog edits affect only future sessions
• Durable SQLite conversation history with immutable launch snapshots, finalized user/assistant text and PCM audio, activity ordering, and full transcript reload
• Active-conversation download as a UTF-8 transcript, one chronological mono MP3 with short gaps between speakers, or a ZIP containing both; request-time speech-aware loudness normalization balances microphone/model turns, and interrupted assistant exports exclude the conservative unheard suffix
• Responsive history navigation: persistent 288 px left rail from 1200 px, shared Ant Design Drawer below that breakpoint, current-item state, and new-practice action
• Text-context continuation through a fresh Qwen WebSocket: Node restores stored Instructions/voice and waits for recent conversation.item.create acknowledgements before declaring the session ready
• Explicit pause/continue controls with a durable paused state: pausing safely settles the current turn, removes the microphone composer, and stops active-time accounting; continuing opens a fresh Qwen connection from finalized SQLite history
• In-place restart with confirmation: the same conversation ID, persona/scenario snapshots, difficulty, Instructions, and voice are retained while owned transcript/audio and elapsed practice time are cleared before reconnecting
• Conversation switching/new-practice/pause/restart/end actions are serialized and wait for response-specific user/assistant persistence acknowledgements before disconnecting; failed settlement is reported instead of silently dropping the last turn
• Conservative in-session goal detection after each complete AI response: a separate asynchronous Qwen text assessment suggests ending only when every scenario success criterion has explicit evidence and at least 0.9 confidence; it never forces the conversation to end
• Durable end-of-session lifecycle: ending locks further messages/realtime restoration, starts an asynchronous Qwen text-model review, survives process restarts, and exposes retryable pending/processing/completed/failed states
• Responsive coaching page focused exclusively on the real learner salesperson, with one server-calculated score set and faithfully translated English/Simplified Chinese assessment, breakdown, strengths, improvements, tips, and moments saved from a single generation; language switching reads the stored report without regeneration. A conversation with at least three learner turns must retain at least three distinct, transcript-grounded highlighted moments after server validation; shorter conversations keep the useful review and show a bilingual insufficient-evidence warning instead of inventing cards. The page also includes session metadata, a copyable transcript, text/audio export, permanent record deletion, and Try again creation from the same persona/scenario/difficulty; stale prompt versions regenerate lazily when opened, and failures identify the data/model/validation/storage stage
• Bottom-anchored conversation history with live user and assistant drafts, timestamps, and interrupted-turn labels
• Press-and-hold recording for mouse, touch, pen, Space, and Enter; release sends and upward slide cancels
• Click-to-start long recording with explicit End speaking and Cancel recording actions; cancellation waits for the upstream audio-buffer clear acknowledgement and never submits the discarded turn
• Audio-reactive microphone waveform, recording duration, and release instruction while a gesture is active
• Browser microphone capture with requested echo cancellation/noise suppression, AGC disabled, a short initialization settling window, an 80 Hz high-pass filter, and privacy-safe effective-settings diagnostics
• Streaming downsampling from the browser device rate to PCM16 16 kHz mono
• Tail-buffer acknowledgement before audio commit, avoiding clipped final syllables
• Node WebSocket proxy with server-only Qwen authentication
• Streamed PCM16 24 kHz Qwen playback with volume, mute, stop-response, and end-session controls
• Response-aware playback receipts and best-effort interrupted-response reconciliation
• Separate SQLite catalog/conversation files, Fastify lifecycle ownership, rollback-journal transactions without persistent WAL/SHM sidecars, foreign keys, busy timeout, append-only migrations, durable catalog CRUD, and an explicit transactional/idempotent catalog initializer
• Phase-aware error handling: a first-time startup failure returns to the launcher; once ready, the chat stays visible, errors use a five-second Ant Design message at the top, fatal failures rebuild safely from finalized SQLite text, and a failed rebuild can be retried from the composer
• Bounded realtime response recovery: 30-second start/progress watchdogs, validation of consumed Qwen events, one context-cleaned retry for failed/empty/malformed-audio AI turns, and recovery-time retry of an unanswered finalized learner turn without recommitting audio

Persistence status

Fresh catalog and conversation files have independent migration histories and contain only their own domain tables. Every preset domain has its own physical table, and catalog records reference preset IDs instead of copying localized labels. The historical combined file retains migrations 1–20 so pnpm database:split can upgrade and copy old data safely. Schema migrations own structure only; current business defaults are installed explicitly. The catalog REST API is:

Method	Path	Purpose
GET	/api/catalog	Read qwenVoices, personaPresets, scenarioPresets, bilingual personas/scenarios, and compatibility IDs
POST	/api/catalog/generate/persona, /api/catalog/generate/scenario	Generate a validated bilingual editable draft while excluding persisted content and the submitted current drawer draft; no catalog row is saved
POST, PUT, DELETE	/api/personas, /api/personas/:id	Create, replace, or delete a persona
POST, PUT, DELETE	/api/scenarios, /api/scenarios/:id	Create, replace, or delete a scenario

Every successful admin mutation first updates local catalog state, then reloads the authoritative catalog. Learner selections therefore reflect saved changes immediately without a rebuild or restart, and remain accurate if the follow-up read temporarily fails. Persona deletion is rejected while a scenario references it; remove the compatibility link first. Scenario deletion cascades only its compatibility rows.

The conversation REST API is:

Method	Path	Purpose
POST	/api/conversations	Resolve authoritative persona/scenario IDs, store a bilingual snapshot, compile Instructions, and create a durable conversation
GET	/api/conversations	List all conversations by latest persisted activity
GET	/api/conversations/:id	Read one immutable launch snapshot and its ordered finalized messages
POST	/api/conversations/:id/pause	Persist a paused state and stop active-time accounting
POST	/api/conversations/:id/resume	Start a new active-time segment before reconnecting realtime voice
POST	/api/conversations/:id/restart	Clear transcript/audio and reset active time while retaining the same launch snapshot and ID
POST	/api/conversations/:id/end	Lock a settled conversation and enqueue coaching feedback
GET	/api/conversations/:id/feedback	Read feedback state/results and the review transcript
POST	/api/conversations/:id/feedback/retry	Retry a failed feedback job
GET	/api/conversations/:id/download?format=audio|text|both	Download one MP3, one UTF-8 transcript, or a ZIP containing both
DELETE	/api/conversations/:id	Permanently delete one ended conversation and all owned snapshots, messages/audio, and feedback

Business defaults are defined only in src/server/catalog/initial-data/*.json and installed with pnpm catalog:init (source) or pnpm catalog:init:prod (built). The initializer inserts bilingual Qwen voice names, bilingual presets, three starter personas, three starter scenarios, and compatibility links. Stable seed keys and transactional conflict-tolerant writes make repeated runs safe without duplicate data or overwritten existing rows.

Conversation snapshots, selected difficulty, compiled Instructions, voice, finalized transcript text, matching finalized-message PCM, pause state, and accumulated active practice time are persisted in the conversation database. Paused time is excluded from feedback duration. Restart keeps the same conversation and immutable launch snapshot but transactionally removes its messages/audio and resets its practice clock. Cancelled input, streaming drafts, and generated-but-unheard assistant suffixes are not stored. Pre-feature text-only conversations remain available for transcript download but cannot be reconstructed as audio. An ended record can be permanently deleted from its feedback page; the server cancels any in-process feedback job before deleting the session so owned snapshots, messages/audio, and feedback cascade together. Try again creates a distinct conversation with the source catalog persona/scenario IDs and the previous difficulty, resolved against the current catalog and locale—it never reopens or mutates the ended session. The current private single-user deployment exposes one global history and has no automatic retention job. See Catalog and prompt compilation and Database for the complete contracts.

The default data/ directory is ignored by Git. A future single-container deployment must mount that directory as persistent storage; embedding the database file in an ephemeral image layer would lose catalog edits when the container is replaced.

Current limitations

Interrupted-response truncation is an estimate because Qwen does not provide word-level audio timestamps and browsers cannot prove what reached the user's physical output device. The application prefers deleting the entire interrupted assistant turn when evidence is weak.

Scenario voiceBehavior.interruptFrequency changes prompt-level conversational patience/interjection/challenge behavior only. Push-to-talk, long recording, and free conversation all preserve manual Qwen turn detection (turn_detection: null); free conversation adds browser-side automatic speech/silence segmentation. The learner can barge in while the persona speaks, but Qwen cannot autonomously begin speaking over an uncommitted learner utterance.

History continuation is text-level context reconstruction, not revival of the old Qwen session or replay of original audio. It restores semantic transcript context but not acoustic details such as the learner's tone or emotion. The model receives the most recent 50 user turns—the maximum supported by qwen-audio-3.0-realtime-plus—while SQLite and the UI keep the complete transcript. Older turns are not summarized during reconnect because a second model call would add latency, cost, failure modes, and possible semantic drift.

The demo does not yet include authentication/admin authorization, per-user history ownership, automatic retention controls, rubric-version administration, automatic multi-attempt feedback backoff, or production rate limiting.

The build already separates artifacts as follows:

dist/client/   # Vite SPA output
dist/server/   # Node server, catalog initializer, and database splitter output

The production image packages both directories and exposes only the Node service. Fastify serves dist/client with an extensionless SPA fallback that excludes /api, /ws, and missing asset paths. The container entrypoint runs the built idempotent catalog initializer against the mounted database directory before starting Fastify; initialization does not depend on Qwen credentials.

Troubleshooting

The persona editor says required presets are missing

Stop the development server if necessary, confirm CATALOG_DATABASE_PATH, run pnpm catalog:init, then reload the SPA. In a built deployment, run pnpm catalog:init:prod against the same persistent volume before starting the service. These commands do not require Qwen credentials.

Catalog initialization rejects an oversized scenario link

The named starter persona/default-scenario pair exceeds the 12,000-character Instructions limit in the reported difficulty, usually after administrator edits. Shorten that persona or scenario configuration, then rerun initialization; the failed run committed none of its initializer data writes.

The start button says credentials are not configured

Create .env, add both required values, and restart pnpm dev. The server reads secrets only at process startup.

Qwen returns HTTP 401 or 403

Confirm that:

• the API key belongs to the China (Beijing) region;
• the Workspace ID is from the same region;
• the workspace has access to qwen-audio-3.0-realtime-plus;
• neither value contains quotes or trailing spaces.

Microphone access fails

Microphone capture requires localhost or HTTPS. Check the browser's site-level microphone permission, confirm that an input device exists, then reload the page.

The transcript works but no audio is heard

Check the page volume and mute controls, system output device, and browser tab audio permission. The app uses Web Audio because Qwen returns raw PCM rather than MP3 or WAV.

The server cannot open the SQLite database

Confirm that the parent directories of CATALOG_DATABASE_PATH and CONVERSATION_DATABASE_PATH are writable by the Node process. Relative paths are resolved from the directory where the process starts. Do not put production databases in a read-only or ephemeral container path.

Further documentation

• Architecture
• UI interactions
• Realtime protocol
• Database
• Catalog and prompt compilation
• AI contributor instructions