LiveKit

LiveKit is a realtime communication platform built around WebRTC, providing rooms, participants, and tracks on top of a horizontally scalable SFU written in Go. The LiveKit Agents framework lets AI agents join rooms as first-class participants, streaming audio and data through STT–LLM–TTS pipelines while LiveKit handles signaling, media routing, and telephony ingress.

Pipecat, in contrast, is an application-layer orchestration framework that models voice agents as pipelines of frame processors connected by transports (WebRTC, WebSockets, telephony, etc.), focusing on low-latency AI orchestration rather than media infrastructure. These two systems are complementary: LiveKit provides the “phone network” (WebRTC SFU and rooms), while Pipecat provides the “brain” and control flow for voice agents.

LiveKit high-level architecture

Core building blocks

LiveKit’s core abstractions are:

• Rooms – virtual spaces where realtime communication happens; each room has a unique name and configuration such as maximum participants and empty timeout.
• Participants – users, AI agents, SIP callers, or services that join rooms; each participant has an identity and may publish or subscribe to tracks.
• Tracks – media streams (audio, video, data) that participants publish and subscribe to within a room; each track can be independently controlled and subscribed to.

Together, these form the foundation of LiveKit’s realtime model and are the same primitives used for both human participants and AI agents.

SFU-based media layer

LiveKit uses a Selective Forwarding Unit (SFU) rather than a peer-to-peer or MCU architecture. In an SFU setup, each publisher sends a single encoded audio/video stream to the SFU, which then forwards copies of that stream (tracks) to interested subscribers without decoding or re-encoding the media.

This design trades some downstream bandwidth efficiency for flexibility: subscribers receive individual tracks and can independently control which tracks to subscribe to, quality levels, and layout. LiveKit’s SFU is implemented in Go on top of Pion WebRTC and supports horizontal scaling: multiple SFU nodes share routing information over Redis so all participants in a given room end up on the same node.

Cloud vs self-hosted deployment

LiveKit can run self-hosted as a “single-home SFU” (one server per room) or on LiveKit Cloud, which uses a global mesh of SFU nodes with edge routing and region-aware deployments. In both cases, a given room is anchored to a specific node, but Cloud adds global infrastructure, observability, built-in inference, and managed agent hosting.

Self-hosted deployments include an embedded STUN/TURN server for NAT traversal and media relay, and Redis for node coordination in multi-node setups. Load balancers in front of the cluster route clients to the appropriate SFU node based on room and region.

High-level architecture diagram

flowchart LR
  subgraph Clients[Clients]
    WebC[Web / Mobile SDKs]
    SIP[PSTN / SIP callers]
  end

  subgraph LiveKit[LiveKit Cluster]
    LB[Load balancer]

    subgraph Node1[SFU Node]
      Sig[Signaling & room logic]
      SFU[WebRTC SFU]
      TURN[STUN/TURN]
    end

    Redis[(Redis routing)]
    AgentSrv[Agent servers]
  end

  subgraph AI[AI Providers]
    STT[STT]
    LLM[LLM]
    TTS[TTS]
  end

  WebC --> LB
  SIP -->|SIP / Telephony| LB
  LB --> Sig
  Sig --> SFU
  Sig <--> Redis

  AgentSrv --> Sig
  SFU <--> AgentSrv

  AgentSrv --> STT
  STT --> LLM
  LLM --> TTS
  TTS --> AgentSrv

This diagram shows LiveKit as the media and signaling fabric; agents connect as participants and use external AI providers for STT/LLM/TTS over HTTP or WebSockets.

WebRTC usage inside LiveKit

WebRTC roles and responsibilities

LiveKit terminates WebRTC connections from clients at the SFU nodes using Pion’s Go-based implementation. Clients establish a WebRTC peer connection through the LiveKit SDKs over a WebSocket-based signaling channel, which handles SDP offer/answer exchange and ICE candidate negotiation.

Once the peer connection is established, media flows as RTP packets between the client and the SFU; the SFU routes tracks between publishers and subscribers, optionally adjusting bitrate and resolution based on downstream bandwidth measurements. An embedded STUN/TURN server helps traverse NAT and firewalls, relaying media when direct UDP paths are not available.

WebRTC vs other transports

LiveKit uses WebRTC for all user-facing realtime audio/video and for browser/mobile clients because of its low latency, congestion control, device access, and built-in echo cancellation. For backend communication from agents to AI providers (STT, LLM, TTS), LiveKit Agents uses HTTP and WebSockets instead of WebRTC, since those links run in data centers with stable networks and only need data transport.

Telephony (PSTN/SIP) users connect via SIP integrations that bridge phone calls into LiveKit rooms, effectively converting PSTN audio into a WebRTC track inside the SFU.

How clients join a LiveKit room

Authentication with access tokens

Every participant connects to LiveKit using a JWT access token generated on a trusted backend using LiveKit API keys and secrets. The token encodes:

• Participant identity and display name
• Target room name
• Grants such as roomJoin, and capabilities for publishing audio, video, or data
• Optional SIP-specific grants for telephony scenarios

Tokens are signed with the API secret to prevent forgery and may be refreshed by the server to support reconnection and long-running sessions.

Client connection flow

The generic client connection flow is:

1. Backend creates or reuses a room using the RoomService API or allows LiveKit to auto-create the room on first join.
2. Backend generates an access token for the user with the appropriate grants and room name.
3. Frontend obtains the token and WebSocket URL (e.g., wss://<project>.livekit.cloud).
4. Client SDK connects using Room.connect(wsUrl, token) in JS, Swift, Flutter, etc.
5. Signaling handshake: the client opens a WebSocket, sends a JoinRequest with the token, and receives a JoinResponse describing the room and existing participants.
6. WebRTC negotiation: the client and SFU exchange SDP offer/answer and ICE candidates over the signaling channel.
7. Media established: once ICE connectivity checks succeed, audio/video/data tracks start flowing over the WebRTC connection.

After connection, the SDK exposes a Room object with localParticipant and a map of remoteParticipants, plus events for participant joins, track publications, and data messages.

Join flow sequence diagram

sequenceDiagram
  participant FE as Your backend
  participant U as User client (SDK)
  participant LB as LiveKit entry
  participant LK as LiveKit SFU node

  FE->>FE: Create/reuse Room via RoomService
  FE->>FE: Generate JWT access token (room, grants)
  FE-->>U: Return wsUrl + token

  U->>LB: WebSocket connect(wsUrl)
  LB->>LK: Route to SFU node for room

  U->>LK: JoinRequest(token)
  LK-->>U: JoinResponse(room, participants)

  U->>LK: WebRTC SDP Offer
  LK-->>U: WebRTC SDP Answer
 

  Note over U,LK: WebRTC connection established
 

LiveKit Agents architecture for voice

Agent server, jobs, and sessions

The LiveKit Agents framework lets Python or Node.js programs participate in rooms as fully-fledged realtime participants. The main runtime pieces are:

• Agent server – a long-lived process that registers with a LiveKit server or Cloud project and waits for dispatch requests.
• Jobs – short-lived subprocesses spawned by the agent server when a dispatch occurs (e.g., when a room is created or a user joins); each job joins a room as a participant.
• AgentSession and Agent – high-level abstractions that manage the STT–LLM–TTS pipeline, turn detection, tools, and multi-agent handoffs during a conversation.

This separation allows LiveKit Cloud to orchestrate agent scaling, load balancing, and lifecycle management across regions, while job processes focus on a single active conversation.

Voice agent media and AI flow

Once both the user and agent job are in the same room, media and data flow as follows:

1. User connects via WebRTC or telephony and publishes an audio track into the room.
2. Agent job joins the room as a participant (using the Agents SDK) and subscribes to the user’s microphone track via RoomInputOptions.
3. Audio is streamed into the agent’s STT component, which performs streaming transcription with turn detection.
4. Transcripts and conversation state are passed to the LLM, often via provider plugins or LiveKit Inference.
5. LLM responses are streamed to TTS, which synthesizes audio in chunks.
6. Agent publishes its synthesized audio as an output track (speaker track) back into the same room using RoomOutputOptions.
7. User client subscribes to the agent’s audio track and plays it to the user via WebRTC.

LiveKit provides abstractions for multi-agent handoff, so different Agents can take turns handling the same user session within a shared AgentSession and shared user data.

Voice agent sequence diagram

sequenceDiagram
  participant User
  participant Client as Web/Mobile/SIP Client
  participant SFU as LiveKit SFU
  participant AgentJob as LiveKit Agent Job
  participant STT
  participant LLM
  participant TTS

  User->>Client: Speak into mic / phone
  Client->>SFU: WebRTC audio track
  SFU->>AgentJob: Forward subscribed audio track

  AgentJob->>STT: Stream audio
  STT-->>AgentJob: Partial transcripts
  AgentJob->>LLM: Send transcript + context
  LLM-->>AgentJob: Streaming text reply
  AgentJob->>TTS: Stream text
  TTS-->>AgentJob: Synthesized audio chunks

  AgentJob->>SFU: Publish agent audio track
  SFU->>Client: Forward agent audio track
  Client->>User: Play audio

This pattern generalizes to multi-agent setups where the AgentSession hands off control from one agent to another (for example, an intro agent followed by a story agent), all within a single room.

---

Key LiveKit architectural concepts for voice agents

For building voice agents, the most relevant LiveKit concepts are:

• Rooms / participants / tracks – shared spaces and media streams used by both humans and agents.
• Access tokens & grants – JWTs carrying room, identity, and permissions, generated on the backend.
• SFU nodes and routing – Pion-based SFU nodes with Redis-backed routing; one node per room in self-hosted, mesh SFU in Cloud.
• STUN/TURN – built-in TURN server and STUN integration for NAT traversal and relay.
• Agents framework – agent servers, jobs, AgentSession, multi-agent handoff, model plugins, and LiveKit Inference.
• Frontends and telephony – Web/mobile SDKs, telephony integration via SIP, and data channels for control messages or tool calls.

These pieces together let LiveKit act as the unified realtime substrate for complex AI-native applications.

Pipecat architecture overview

Core abstractions: frames, processors, pipelines

Pipecat approaches voice AI from the application and orchestration layer rather than the media infrastructure layer. Its core concepts are:

• Frames – typed data packets such as AudioFrame, TextFrame, and control frames like UserStartedSpeakingFrame flowing through the system.
• Frame processors – specialized workers that consume specific frame types and emit others, e.g., an STT processor converts audio frames to text frames, an LLM processor converts text to response text, and a TTS processor converts text to audio frames.
• Pipelines – ordered collections of frame processors that define end-to-end flows; frames move through the pipeline as an asynchronous stream, enabling parallel processing and streaming behavior.

A basic voice AI pipeline might be:

pipeline = Pipeline([
    transport.input(),      # Receives user audio
    stt,                    # AudioFrame -> TextFrame
    context_aggregator.user(),
    llm,                    # TextFrame -> TextFrame (reply)
    tts,                    # TextFrame -> AudioFrame
    transport.output(),     # Sends audio back to user
    context_aggregator.assistant(),
])

Pipecat supports parallel branches and complex patterns using ParallelPipeline and a PipelineRunner that manages lifecycle, metrics, and error handling.

Transports and connection layer

Rather than implementing its own SFU, Pipecat relies on transports to connect users to the pipeline. Transports encapsulate how audio/video/data reach the bot and include:

• DailyTransport – WebRTC transport using Daily’s global WebRTC infrastructure.
• LiveKitTransport – WebRTC transport on top of LiveKit’s SFU and rooms.
• FastAPIWebsocketTransport – WebSocket-based transport for telephony and custom server-to-server links.
• SmallWebRTCTransport – direct peer-to-peer WebRTC without cloud infrastructure.

Transports provide transport.input() and transport.output() endpoints for pipelines, and can be switched without changing the bot logic.

Pipecat emphasizes ultra-low latency, streaming processing: while the LLM is still generating later tokens, earlier tokens are already converted to speech and played to the user, and interruption is handled by special control frames that can cancel downstream tasks when the user starts speaking.

Session initialization

Session initialization is transport-specific but follows common patterns.

• For room-based WebRTC (e.g., Daily), the runner creates a room and tokens via the provider API, then starts the bot with room URL and token.
• For telephony (Twilio SIP), a webhook server responds to incoming calls by creating a room (e.g., Daily room), starting a bot process, and bridging the PSTN call into the WebRTC room via SIP.
• For LiveKitTransport, Pipecat connects its transport to an existing LiveKit room using the LiveKit URL and token, then runs the pipeline in the same way as with other transports.

LiveKit vs Pipecat architectures

Conceptual roles

A useful mental model is:

• LiveKit – infrastructure and transport layer: WebRTC SFU, rooms, participants, tracks, telephony ingress, and agent hosting.
• Pipecat – application and orchestration layer: frame-based pipelines, STT/LLM/TTS wiring, context management, and interruption handling.

LiveKit handles the hard parts of NAT traversal, media routing, and scaling SFU nodes; Pipecat handles the hard parts of composing multiple AI services into a coherent, low-latency conversational agent.

Architectural differences table

Dimension	LiveKit	Pipecat
Primary role	WebRTC SFU and realtime communication platform	Voice/multimodal AI orchestration framework
Core primitives	Rooms, participants, tracks, access tokens, SFU nodes	Frames, frame processors, pipelines, transports
Media handling	Terminates WebRTC, routes RTP via SFU; built-in STUN/TURN	Delegates media transport to external providers via transports (LiveKit, Daily, P2P, WebSockets)
Agent model	Agents join rooms as participants; Agents framework manages agent servers, jobs, sessions	Bots are pipelines of processors; Pipecat Cloud or runners manage sessions and pipelines
AI integration	Plugins and LiveKit Inference for STT/LLM/TTS; integrated into AgentSession	STT/LLM/TTS are processors in the pipeline; provider-agnostic and modular
Turn-taking & interruption	Built-in turn detection and multi-agent handoff inside Agents framework	Frame-based VAD and UserStartedSpeakingFrame cancel downstream work, fine-grained interruption control
Deployment focus	LiveKit Cloud (global mesh SFU) or self-hosted SFU with Redis; Kubernetes-friendly	Runs as Python application or via Pipecat Cloud; relies on external media infra (Daily, LiveKit, Twilio, etc.)
Telephony	Native SIP integration into rooms	Telephony via WebSocket transports and SIP integrations (e.g., Twilio + Daily)

Sources: LiveKit docs and SFU internals, Agents framework docs, and self-hosting guides; Pipecat docs on frames, pipelines, and transports plus independent analyses.

How the voice-agent flow differs

In a typical LiveKit-only voice agent (using LiveKit Agents):

• The agent is conceptually a participant in the room, with audio in/out represented as tracks; the Agents framework hides the internal AI pipeline behind AgentSession abstractions.
• WebRTC is always terminated by LiveKit; all user media flows through the SFU before reaching the agent job.
• Infrastructure and orchestration are tightly integrated: LiveKit Cloud manages SFU nodes, agent servers, jobs, and observability in one system.

In a Pipecat-based voice agent using LiveKitTransport:

• LiveKit still operates as the SFU and room manager, but Pipecat owns the pipeline, treating LiveKitTransport as just another transport source/sink.
• The bot’s behavior is expressed explicitly as a pipeline of processors, making it easy to branch, log, and extend frames (e.g., add guardrails, billing, observability) without changing media infrastructure.
• The same pipeline can run over different transports (Daily, P2P, WebSocket) by swapping the transport component, whereas LiveKit Agents is more tightly coupled to LiveKit’s own transport stack.

Combined architecture diagram

A combined architecture using LiveKit as transport and Pipecat as pipeline looks like this:

flowchart LR
  subgraph Users
    UC[User client<br/>WebRTC via LiveKit SDK]
  end

  subgraph LiveKit[LiveKit SFU]
    LKRoom[Room & SFU node]
  end

  subgraph Pipecat[Pipecat Bot]
    TIn[LiveKitTransportInput]
    STT[STT processor]
    Ctx[Context aggregator]
    LLM[LLM processor]
    TTS[TTS processor]
    TOut[LiveKitTransportoutput]
  end

  UC <--> LKRoom
  LKRoom <--> TIn
  TIn --> STT --> Ctx --> LLM --> TTS --> TOut
  TOut <--> LKRoom

Here, LiveKit provides the room, SFU, and WebRTC connectivity, while Pipecat handles the AI pipeline, making the two layers cleanly separable.