Omni-LiveKit

LiveKit voice gateway for WebRTC-based voice AI applications.

Overview

LiveKit provides pure WebRTC voice without phone network involvement. While PSTN gateways (Twilio, Telnyx, Vonage, Plivo) also offer WebRTC SDKs, they primarily bridge browser/app connections to the phone network. LiveKit is designed for direct browser-to-browser and app-to-app communication.

┌───────────────┐        ┌─────────────────┐        ┌───────────────────┐
│  Browser/App  │◄──────►│  LiveKit Cloud  │◄──────►│   OmniVoice       │
│   (WebRTC)    │ WebRTC │    or Server    │ WebRTC │   Voice Gateway   │
└───────────────┘        └─────────────────┘        └───────────────────┘

Use Cases

Use Case	PSTN Gateways	LiveKit
Phone calls (PSTN)	Yes	No
Browser WebRTC	Yes (bridges to PSTN)	Yes (native)
Mobile apps	WebRTC SDK available	WebRTC SDK available
Browser-to-browser	Via PSTN bridge	Direct WebRTC
Latency	300-500ms (PSTN hop)	<200ms
Cost model	Per-minute telephony	Infrastructure only

Installation

go get github.com/plexusone/omni-livekit

Native Dependencies

This package requires native audio libraries for Opus encoding/decoding and resampling:

macOS:

brew install opus opusfile libsoxr

Ubuntu/Debian:

apt-get install libopus-dev libopusfile-dev libsoxr-dev

Fedora:

dnf install opus-devel opusfile-devel soxr-devel

Build Tags

Voice agents must be built with -tags opus to enable proper Opus codec support:

# Required for voice agents
go run -tags opus ./cmd/voice-agent
go build -tags opus ./cmd/voice-agent

# Without -tags opus, audio won't be decoded properly and STT will fail

Quick Start

Go Backend (AI Agent)

package main

import (
    "context"
    "log"
    "os"

    "github.com/plexusone/omni-livekit/omnivoice/gateway"
    coregateway "github.com/plexusone/omnivoice-core/gateway"
)

func main() {
    // Create gateway implementing coregateway.WebRTCGateway interface
    gw, err := gateway.New(gateway.Config{
        LiveKitURL:    os.Getenv("LIVEKIT_URL"),
        LiveKitAPIKey: os.Getenv("LIVEKIT_API_KEY"),
        LiveKitSecret: os.Getenv("LIVEKIT_API_SECRET"),
        RoomName:      "voice-agent",
        AgentIdentity: "ai-agent",
        SampleRate:    24000,
    })
    if err != nil {
        log.Fatal(err)
    }

    // Handle participant joins (WebRTC-specific handler)
    gw.OnParticipantJoined(func(p *coregateway.ParticipantInfo) error {
        log.Printf("Participant joined: %s (%s)", p.DisplayName, p.Identity)
        return nil // Accept participant
    })

    ctx := context.Background()
    if err := gw.Start(ctx); err != nil {
        log.Fatal(err)
    }
}

Generate Client Token

import "github.com/plexusone/omni-livekit/room"

client, _ := room.NewClient(room.Config{
    APIKey:    os.Getenv("LIVEKIT_API_KEY"),
    APISecret: os.Getenv("LIVEKIT_API_SECRET"),
    URL:       os.Getenv("LIVEKIT_URL"),
})

// Generate token for web/mobile client
token, _ := client.GenerateClientToken("voice-agent", "user-123", "John")

React Frontend

import { LiveKitRoom, useVoiceAssistant } from '@livekit/components-react';

function VoiceAgent() {
    const token = await fetchTokenFromBackend();

    return (
        <LiveKitRoom
            serverUrl={process.env.LIVEKIT_URL}
            token={token}
            connect={true}
            audio={true}
        >
            <VoiceUI />
        </LiveKitRoom>
    );
}

Interface

omni-livekit implements the coregateway.WebRTCGateway interface from omnivoice-core:

type WebRTCGateway interface {
    Name() ProviderName
    Start(ctx context.Context) error
    Stop() error
    OnParticipantJoined(handler ParticipantHandler)
    JoinRoom(ctx context.Context, roomName string) error
    LeaveRoom() error
    CurrentRoom() string
    GetSession(participantID string) (WebRTCSession, bool)
    ListSessions() []WebRTCSession
    GenerateClientToken(roomName, identity, displayName string) (string, error)
}

This is different from the Gateway interface used by PSTN providers (Twilio, Telnyx, etc.) which uses phone numbers and MakeCall().

Architecture

Audio Flow

Client (Browser/Mobile)
    │
    ▼ WebRTC Audio Track (Opus)
    │
┌───────────────────────────────────────┐
│           LiveKit Server              │
└───────────────────────────────────────┘
    │
    ▼ WebRTC Audio Track (Opus)
    │
┌───────────────────────────────────────┐
│       Go Backend (omni-livekit)       │
│                                       │
│  ┌─────────────────────────────────┐  │
│  │  PCMRemoteTrack                 │  │
│  │  (Opus → PCM16 decode)          │  │
│  └─────────────┬───────────────────┘  │
│                ▼                      │
│  ┌─────────────────────────────────┐  │
│  │  Voice Pipeline                 │  │
│  │  STT → LLM → TTS                │  │
│  │  (or Voice-to-Voice)            │  │
│  └─────────────┬───────────────────┘  │
│                ▼                      │
│  ┌─────────────────────────────────┐  │
│  │  PCMLocalTrack                  │  │
│  │  (PCM16 → Opus encode)          │  │
│  └─────────────────────────────────┘  │
└───────────────────────────────────────┘

Key Differences from PSTN Gateways

Aspect	PSTN Gateways	LiveKit
Primary use	Phone network access	Browser/app-to-app
Connection model	Phone number dialing	Room-based participation
Signaling	HTTP webhooks + WebRTC	WebRTC/WebSocket
Audio format	mulaw 8kHz (PSTN), Opus (WebRTC)	Opus (configurable rate)
Identity	Phone number or SIP URI	Participant ID
Session model	Call legs (inbound/outbound)	Room participants

Human Participation

When an AI agent joins a LiveKit room, humans can join via several methods:

Option 1: LiveKit Meet (No Code Required)

LiveKit provides a free hosted web UI. Generate a join URL with token:

// Generate token and URL
token, _ := client.GenerateClientToken("my-room", "user-123", "Alice")
joinURL := fmt.Sprintf("https://meet.livekit.io/custom?liveKitUrl=%s&token=%s",
    url.QueryEscape(serverURL), token)
// Share joinURL with the human participant

The human opens the link in their browser, grants mic/camera permissions, and joins.

Option 2: LiveKit Cloud Dashboard

If using LiveKit Cloud, create rooms and generate join links from the dashboard UI.

Option 3: Custom Frontend

Build your own UI with LiveKit's client SDKs:

Platform	SDK	Install
Web (JS/TS)	`livekit-client`	`npm install livekit-client`
React	`@livekit/components-react`	`npm install @livekit/components-react`
iOS	LiveKit Swift SDK	Swift Package Manager
Android	LiveKit Android SDK	Maven
Flutter	`livekit_client`	pub.dev

React Example:

import { LiveKitRoom, VideoConference } from '@livekit/components-react';
import '@livekit/components-styles';

function MeetingRoom({ token, serverUrl }) {
  return (
    <LiveKitRoom token={token} serverUrl={serverUrl} connect={true}>
      <VideoConference />
    </LiveKitRoom>
  );
}

Minimal HTML/JS:

<script src="https://unpkg.com/livekit-client/dist/livekit-client.umd.js"></script>
<script>
  const room = new LivekitClient.Room();
  await room.connect('wss://your-server.livekit.cloud', token);

  // Enable microphone
  await room.localParticipant.setMicrophoneEnabled(true);
</script>

Recommendation

Use Case	Recommended Approach
Testing/Demos	LiveKit Meet
Quick prototype	LiveKit React components
Production app	Custom frontend with full UX control

Environment Variables

# LiveKit Cloud or self-hosted server
export LIVEKIT_URL="wss://your-app.livekit.cloud"
export LIVEKIT_API_KEY="your-api-key"
export LIVEKIT_API_SECRET="your-api-secret"

Audio Configuration

The gateway supports configurable sample rates for optimal voice AI performance:

gateway.Config{
    SampleRate: 24000,  // 24kHz for high quality (OpenAI Realtime API)
    // SampleRate: 16000, // 16kHz for most STT/TTS
    Channels: 1,        // Mono audio
}

Session Events

session, ok := gw.GetSession(participantID)
if !ok {
    return
}

// WebRTCSession provides participant info
fmt.Printf("Room: %s\n", session.RoomName())
fmt.Printf("Participant: %s\n", session.Participant().DisplayName)

// Handle events
for event := range session.Events() {
    switch event.Type {
    case coregateway.EventUserTranscript:
        fmt.Printf("User said: %s\n", event.Data)
    case coregateway.EventAgentTranscript:
        fmt.Printf("Agent said: %s\n", event.Data)
    case coregateway.EventInterruption:
        fmt.Println("User interrupted agent")
    }
}

// Send audio to participant
samples := make([]int16, 480) // 20ms at 24kHz
session.SendAudio(samples)

Integration with OmniVoice

import (
    "github.com/plexusone/omni-livekit/omnivoice/gateway"
    "github.com/plexusone/omnivoice"
    _ "github.com/plexusone/omnivoice/providers/all"
)

// Use OmniVoice for STT/TTS
stt, _ := omnivoice.GetSTTProvider("deepgram", omnivoice.WithAPIKey(apiKey))
tts, _ := omnivoice.GetTTSProvider("elevenlabs", omnivoice.WithAPIKey(apiKey))

// LiveKit gateway receives audio → STT → LLM → TTS → sends audio

Voice Pipeline Modes

Standard Pipeline (STT → LLM → TTS):

Audio In → STT → Text → LLM → Text → TTS → Audio Out

This mode uses separate providers for speech-to-text, language model, and text-to-speech.

Voice-to-Voice (Lower Latency):

Audio In → Voice Model → Audio Out

OmniVoice supports direct voice-to-voice with:

Deepgram - Nova-3 Voice Agent
Google - Gemini Live
OpenAI - Realtime API

Voice-to-voice eliminates the text intermediate step, reducing latency for real-time conversations.

Avatars

The avatar package provides a unified interface for visual representations of voice agents.

Supported Providers

Provider	Type	Status	Best For
`none`	Audio-only	✅ Built-in	Voice-only agents
`static`	Pre-encoded H.264	✅ Built-in	Simple visual presence
Tavus	Realistic humans	✅ Implemented	Video-trained digital twins
HeyGen	Realistic humans	🔜 Planned	Highest quality realism
D-ID	Photo-to-avatar	🔜 Planned	Quick setup from photo
bitHuman	Animated + realistic	🔜 Planned	Animated characters, edge/CPU
Simli	Realistic + animated	🔜 Planned	Budget-friendly

See Avatar Provider Comparison for detailed evaluation.

Quick Start with Factory

import (
    "github.com/plexusone/omni-livekit/avatar"
    _ "github.com/plexusone/omni-livekit/avatar/tavus" // Register Tavus provider
)

// Configure avatar
result, err := avatar.Setup(avatar.SetupConfig{
    Provider: "tavus",  // or "static", "" (none)
    Tavus: avatar.TavusConfig{
        APIKey: os.Getenv("TAVUS_API_KEY"),
        PalID:  os.Getenv("TAVUS_PAL_ID"),  // Optional
    },
    LiveKitURL:       os.Getenv("LIVEKIT_URL"),
    LiveKitAPIKey:    os.Getenv("LIVEKIT_API_KEY"),
    LiveKitAPISecret: os.Getenv("LIVEKIT_API_SECRET"),
})

switch result.Mode {
case avatar.SetupModeStatic:
    // Apply static image to agent options
    agentOpts.MediaMode = agent.AudioWithImage
    agentOpts.Image.H264Path = result.StaticImage.H264Path

case avatar.SetupModeLive:
    // Start live avatar session after agent joins room
    err := result.Session.Start(ctx, avatar.StartOptions{
        Room:          agent.Room(),
        AgentIdentity: "my-agent",
        // ... LiveKit credentials
    })
    defer result.Session.Close(ctx)

    // Wait for avatar to join
    result.Session.WaitForJoin(ctx, 30*time.Second)

    // Stream TTS audio to avatar for lip-sync
    audioOut := result.Session.AudioOutput()
    audioOut.CaptureFrame(ctx, ttsAudio)
    audioOut.Flush(ctx)
}

Static Image Avatar

For a simple visual presence without lip-sync:

result, _ := avatar.Setup(avatar.SetupConfig{
    Provider: avatar.ProviderStatic,
    StaticImage: avatar.StaticImageConfig{
        H264Path: "/path/to/avatar.h264",  // Pre-encoded H.264
        // Or: UseDefault: true            // Use embedded default
    },
})

Live Lip-Sync Avatar (Tavus)

For realistic talking head video with Tavus:

result, _ := avatar.Setup(avatar.SetupConfig{
    Provider: avatar.ProviderTavus,
    Tavus: avatar.TavusConfig{
        APIKey: os.Getenv("TAVUS_API_KEY"),
        PalID:  "your-pal-id",  // Or use default
        FaceID: "optional-face-override",
    },
    // LiveKit credentials for avatar token generation
    LiveKitURL:       os.Getenv("LIVEKIT_URL"),
    LiveKitAPIKey:    os.Getenv("LIVEKIT_API_KEY"),
    LiveKitAPISecret: os.Getenv("LIVEKIT_API_SECRET"),
})

The avatar joins the LiveKit room as a participant with synchronized lip movements.

For animated characters or other avatar styles, see the Avatar Provider Comparison.

Recording

Record rooms to a local file or S3 via LiveKit Egress:

import "github.com/plexusone/omni-livekit/room"

recorder := room.NewRecordingClient(
    os.Getenv("LIVEKIT_URL"),
    os.Getenv("LIVEKIT_API_KEY"),
    os.Getenv("LIVEKIT_API_SECRET"),
)

info, err := recorder.StartRecording(ctx, room.RecordingConfig{
    RoomName: "my-meeting",
    Layout:   room.LayoutGrid, // or LayoutSpeaker, LayoutSingleSpeaker
    Format:   room.FormatMP4,  // or FormatOGG
    FilePath: "/recordings/my-meeting.mp4", // or use S3: &room.S3Config{...}
})
if err != nil {
    log.Fatal(err)
}

// ... later
recorder.StopRecording(ctx, info.EgressID)

See the Room Client API for the full RecordingConfig/S3Config reference.

OmniMeet Integration

This package also provides the LiveKit provider for OmniMeet, the PlexusOne meeting abstraction.

import "github.com/plexusone/omni-livekit/omnimeet"

provider, _ := omnimeet.NewProvider(omnimeet.Config{
    APIKey:    os.Getenv("LIVEKIT_API_KEY"),
    APISecret: os.Getenv("LIVEKIT_API_SECRET"),
    ServerURL: os.Getenv("LIVEKIT_URL"),
})

// Create meeting
m, _ := provider.CreateMeeting(ctx, meeting.CreateRequest{
    Name: "Team Standup",
})

// Join as agent
factory := provider.(provider.AgentParticipantFactory)
agent, _ := factory.CreateAgentParticipant(provider.AgentParticipantOptions{
    AutoSubscribe: true,
})

See the OmniMeet LiveKit Provider documentation for details.

Related Packages

omnimeet-core - Meeting abstraction core
omnivoice-core - Voice interfaces
omnivoice - STT/TTS providers
livekit/server-sdk-go - LiveKit Go SDK

License

MIT License

Name		Name	Last commit message	Last commit date
Latest commit History 121 Commits
.github		.github
agent		agent
avatar		avatar
cmd		cmd
docs		docs
omnimeet		omnimeet
omnivoice/gateway		omnivoice/gateway
room		room
.gitignore		.gitignore
.golangci.yaml		.golangci.yaml
CHANGELOG.json		CHANGELOG.json
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
go.mod		go.mod
go.sum		go.sum
mkdocs.yml		mkdocs.yml

Navigation Menu

Uh oh!

Repository files navigation

Omni-LiveKit

Overview

Use Cases

Installation

Native Dependencies

Build Tags

Quick Start

Go Backend (AI Agent)

Generate Client Token

React Frontend

Interface

Architecture

Audio Flow

Key Differences from PSTN Gateways

Human Participation

Option 1: LiveKit Meet (No Code Required)

Option 2: LiveKit Cloud Dashboard

Option 3: Custom Frontend

Recommendation

Environment Variables

Audio Configuration

Session Events

Integration with OmniVoice

Voice Pipeline Modes

Avatars

Supported Providers

Quick Start with Factory

Static Image Avatar

Live Lip-Sync Avatar (Tavus)

Recording

OmniMeet Integration

Related Packages

License

About

Releases

Contributors

Languages

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Omni-LiveKit

Overview

Use Cases

Installation

Native Dependencies

Build Tags

Quick Start

Go Backend (AI Agent)

Generate Client Token

React Frontend

Interface

Architecture

Audio Flow

Key Differences from PSTN Gateways

Human Participation

Option 1: LiveKit Meet (No Code Required)

Option 2: LiveKit Cloud Dashboard

Option 3: Custom Frontend

Recommendation

Environment Variables

Audio Configuration

Session Events

Integration with OmniVoice

Voice Pipeline Modes

Avatars

Supported Providers

Quick Start with Factory

Static Image Avatar

Live Lip-Sync Avatar (Tavus)

Recording

OmniMeet Integration

Related Packages

License

About

Resources

Stars

Watchers

Forks

Releases

Contributors

Languages