'
AUDIO_URL='https://example.com/media/speech.wav'
BASE_URL='https://console.spatius.ai/v1/open'
REQUEST_ID=$(uuidgen | tr '[:upper:]' '[:lower:]')
BODY=$(jq -n --arg avatar "$SPATIUS_AVATAR_ID" --arg audio "$AUDIO_URL" \
--arg request "$REQUEST_ID" \
'{avatarId:$avatar,audioUrl:$audio,requestId:$request}')
# Keep REQUEST_ID and BODY unchanged when retrying this creation.
CREATED=$(curl --fail-with-body --silent --show-error --retry 3 \
"$BASE_URL/videos" \
-H "X-App-ID: $SPATIUS_APP_ID" -H "X-API-Key: $SPATIUS_API_KEY" \
-H 'Content-Type: application/json' --data "$BODY")
JOB_ID=$(jq -er '.jobId' <<< "$CREATED")
printf 'Job ID: %s\n' "$JOB_ID"
while true; do
DETAIL=$(curl --fail-with-body --silent --show-error --retry 3 \
"$BASE_URL/video-jobs/$JOB_ID" \
-H "X-App-ID: $SPATIUS_APP_ID" -H "X-API-Key: $SPATIUS_API_KEY")
STATUS=$(jq -er '.job.status' <<< "$DETAIL")
jq '{status:.job.status,progress:.job.progress,error:.job.error}' <<< "$DETAIL"
case "$STATUS" in
succeeded)
DOWNLOAD_URL=$(jq -er '.videoUrl' <<< "$DETAIL")
curl --fail --location --show-error "$DOWNLOAD_URL" --output video.mp4
break ;;
failed|expired) exit 1 ;;
*) sleep 15 ;;
esac
done
```
Only `avatarId` and `audioUrl` are required. Add `backgroundUrl`, `name`, or `video` settings when needed. The [Create a Video reference](/api-reference/videos/create-a-video) lists every field and response.
## Access and safe retries
Use a public Avatar, an Avatar actively assigned to your account, or one explicitly allowed for your account. Spatius checks Avatar access when accepting the request and again before rendering.
Apps owned by the same account share video jobs, retry keys, and limits. A job owned by another account returns `404`. Video access can allow reads while creation is paused; removing access also revokes reads.
| Creation request | Result |
| ------------------------------------- | --------------------------------------------------------------------------- |
| New `requestId`, or no `requestId` | Create a new job. |
| Same `requestId` and normalized input | Return the existing job's current status without counting another creation. |
| Same `requestId`, different input | Return `409` with `conflict`. |
An omitted setting and its explicit default are equivalent for retry comparison. Keep the UUID and request body when retrying a timeout or uncertain response. To render again after a terminal failure, use a new UUID. The body field `requestId` is your retry key; `error.requestId` and the response header `X-Request-ID` identify an individual HTTP request for support.
## Media requirements
| Input | Supported Content-Type | Maximum size |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| Audio | `audio/mpeg`, `audio/mp3`, `audio/wav`, `audio/x-wav`, `audio/wave`, `audio/vnd.wave`, `audio/mp4`, `audio/x-m4a`, `audio/aac`, `audio/ogg` | 500 MiB |
| Background image | `image/png`, `image/jpeg`, `image/webp` | 50 MiB |
Use public HTTP(S) URLs, including signed URLs. Embedded usernames/passwords and private-network destinations are rejected. Authentication headers are not forwarded to source hosts. Each connection and redirect is checked.
Keep source URLs accessible for the preparation window of up to **30 minutes**; each transfer has a **five-minute** deadline. Empty files, raw PCM, and `application/octet-stream` are not supported. Wrap raw PCM in a WAV container using its actual sample rate and channel count before hosting it. Audio decoding and duration checks happen asynchronously, so acceptance does not guarantee a successful render.
## Presentation settings
| Field in `video` | Default | Allowed values |
| --------------------------------- | --------- | ---------------------------------------------------------------------- |
| `width`, `height` | 1024 each | Even integers from 64 to 1920; combined area at most 2,073,600 pixels. |
| `fit` | `crop` | `crop`, `contain` |
| `backgroundColor` | `#000000` | Six-digit RGB hex color. |
| `backgroundFit` | `cover` | `cover`, `contain`, `stretch` |
| `leadInSeconds`, `leadOutSeconds` | 0 | Finite numbers from 0 to 60; additional idle time. |
Opening and closing transitions are included even when additional idle time is zero. Output duration can therefore exceed audio duration, with timing rounded to video frames. Spatius manages encoding settings.
## Progress and downloads
| Status | Meaning |
| ------------ | --------------------------------------------------------------------------- |
| `queued` | Accepted and waiting to start. |
| `processing` | Preparing media or rendering. Inspect `job.progress.stage`. |
| `succeeded` | MP4 ready; the detail response includes `videoUrl` and `videoUrlExpiresAt`. |
| `failed` | Terminal failure; inspect `job.error`. No new render starts automatically. |
| `expired` | Output has expired, or a submission remained unresolved for seven days. |
Poll every **15 seconds**. Progress stages describe work, not percentages or completion estimates. [List Video Jobs](/api-reference/video-jobs/list-video-jobs) accepts repeated filters such as `statuses=queued&statuses=processing` and uses the [shared pagination rules](/api-reference/errors#pagination). List responses contain summaries without download links.
Output is currently retained for **seven days after render submission**. Use `job.expiresAt` as the retention deadline. Detail reads return fresh links, currently valid for **15 minutes**, capped by that deadline. Download without App authentication headers and save the MP4 in your own storage if you need it longer.
If a link refresh returns `503`, retry the detail request; the saved job remains successful. After output expiry, the job returns `expired` without a link, while history remains available. Refreshing a link does not extend retention.
## Job errors
Job failures appear under `job.error`, separately from [HTTP request errors](/api-reference/errors). Use `code` to decide what to fix. `retryable` means a new job may succeed after addressing the cause; it does not restart the failed job.
| Code | Action |
| ------------------------------------------------------------- | ------------------------------------------------------------------ |
| `source_unavailable` | Check that the media URL remains publicly accessible. |
| `unsupported_media_type`, `empty_media`, `media_too_large` | Fix the file or Content-Type. |
| `preparation_timeout` | Use a source that can finish within the preparation deadline. |
| `app_access_revoked`, `avatar_forbidden` | Restore App or Avatar access. |
| `session_token_expired`, `checksum_mismatch`, `render_failed` | Check `retryable`; submit a new job when appropriate. |
| `renderer_rejected`, `invalid_manifest` | Check the request; contact support with the job ID if it persists. |
## Limits
Video access requires explicit account limits. Creation and read counters are independent of the Avatar API, and all Apps owned by one account share them. Accepted jobs count even if they later fail; repeated requests that return an existing job do not count again.
| Video limit | Hard ceiling; your configured limit may be lower |
| ---------------------------------------------- | ------------------------------------------------ |
| Creations per minute / hour / rolling 24 hours | 20 / 200 / 2,000 |
| Reads per minute | 1,200 |
| Reads per second / burst | 20 / 40 |
The Console API checks video access and request rates. It does not impose an active-video-job quota; the video service manages execution concurrency. Back off after `429` responses. The Video API does not reserve Avatar Creations; existing usage accounting still applies.
This version supports URL inputs and polling. Public uploads, developer callbacks, cancellation, text-to-speech, and permanent output storage are not available.
## Next steps
# Get a Video Job
Source: https://docs.spatius.ai/api-reference/video-jobs/get-a-video-job
/openapi/video-open-api.json get /video-jobs/{jobId}
Returns a job owned by the authenticated account. Other Apps owned by that account can read it; cross-account reads return 404.
Successful, unexpired jobs include a fresh download URL. If refreshing the link fails, the request returns 503 while the saved job remains succeeded. Output expiry returns expired without a link. Download the MP4 before job.expiresAt; refreshing a link does not extend retention.
# List Video Jobs
Source: https://docs.spatius.ai/api-reference/video-jobs/list-video-jobs
/openapi/video-open-api.json get /video-jobs
Lists the authenticated account's jobs, newest first. Results contain summaries; use Get a Video Job for download links. Video reads remain available when creation is disabled, as long as video access remains configured.
# Create a Video
Source: https://docs.spatius.ai/api-reference/videos/create-a-video
/openapi/video-open-api.json post /videos
Accepts an Avatar ID and public audio URL, saves a job, and immediately returns its ID. Media processing happens asynchronously; poll the job every 15 seconds until it succeeds, fails, or expires.
Video access must be enabled separately from Avatar creation. Apps owned by one account share video jobs, retry keys, and independent video rate limits. Accepted jobs count toward creation limits even if processing later fails. The Console API does not impose an active-video-job quota; the video service manages execution concurrency.
Use requestId to retry uncertain submissions without creating another render. A retry returns the existing job's current state. Use a new requestId to render again after a terminal failure.
# Backend Mode Client
Source: https://docs.spatius.ai/backend-mode/client-sdk
Feed audio and motion data delivered by your backend into AvatarKit.
The client receives each response turn from your backend and renders it locally with AvatarKit. It never connects to Motion Server or holds a Spatius API Key.
```mermaid actions={false} theme={null}
---
config:
"look": "handDrawn"
"theme": "base"
"themeVariables":
"background": "#ffffff"
"textColor": "#111827"
"lineColor": "#64748b"
"primaryColor": "#e8f4fd"
"primaryTextColor": "#111827"
"primaryBorderColor": "#2196F3"
---
flowchart LR
A["Your backend"] -->|audio + turn ID| B["AvatarKit Client"]
A -->|motion data + turn ID| B
B -->|audio and motion feed APIs| C["Local rendering"]
```
## Implement the client
Use the same App ID, Avatar ID, region, and audio format as the backend session. Backend Mode does not use a Spatius Session Token on the client.
Use your application's WebSocket or streaming transport. For every response, deliver the audio before its first motion data payload, and include the same turn ID on both message types.
Pass each audio payload to the platform's audio-yield method. Store the conversation ID it returns against your backend turn ID, then use that conversation ID when you pass the matching motion data to AvatarKit.
```typescript theme={null}
import {
AvatarManager,
AvatarSDK,
AvatarView,
DrivingServiceMode,
} from '@spatius/avatarkit'
await AvatarSDK.initialize('your-app-id', {
drivingServiceMode: DrivingServiceMode.backend,
audioFormat: { channelCount: 1, sampleRate: 16000 },
})
const avatar = await AvatarManager.shared.load('your-avatar-id')
const avatarView = new AvatarView(
avatar,
document.getElementById('avatar-container')!,
)
const controller = avatarView.controller
const conversationIds = new Map()
const decodeBase64 = (value: string) =>
Uint8Array.from(atob(value), (character) => character.charCodeAt(0))
document.getElementById('connect-button')!.addEventListener('click', async () => {
// Browsers require audio initialization inside a user gesture.
await controller.initializeAudioContext()
const socket = new WebSocket('wss://your-backend.example/avatar')
socket.addEventListener('message', (event) => {
const message = JSON.parse(event.data)
if (message.type === 'audio') {
const id = controller.yieldAudioData(
decodeBase64(message.payload),
message.end,
)
if (id && !conversationIds.has(message.turnId)) {
conversationIds.set(message.turnId, id)
}
}
if (message.type === 'motion') {
const id = conversationIds.get(message.turnId)
if (id) {
controller.yieldFramesData(
message.payloads.map(decodeBase64),
id,
)
}
if (message.end) conversationIds.delete(message.turnId)
}
if (message.type === 'interrupt') {
controller.interrupt()
conversationIds.clear()
}
})
})
```
Interrupt discarded turns, close your application transport, and dispose of the Avatar view when the experience ends. Do not call the Direct Mode `start()` or `send()` methods on this path.
Exact Backend Mode method names are in the [Web](/sdk-reference/web-sdk/reference), [iOS](/sdk-reference/ios-sdk/api-reference), [Android](/sdk-reference/android-sdk/api-reference), and [Flutter](/sdk-reference/flutter-sdk/api-reference) references.
## Next steps
# Backend Mode Setup
Source: https://docs.spatius.ai/backend-mode/server-sdk
Connect your backend audio pipeline to Motion Server with a Spatius Server SDK.
In Backend Mode Integration, your backend sends avatar speech audio to Motion Server, then delivers the response audio and returned motion data to each client.
```mermaid actions={false} theme={null}
---
config:
"look": "handDrawn"
"theme": "base"
"themeVariables":
"background": "#ffffff"
"textColor": "#111827"
"lineColor": "#64748b"
"primaryColor": "#e8f4fd"
"primaryTextColor": "#111827"
"primaryBorderColor": "#2196F3"
---
flowchart LR
A["Your backend
(ASR / LLM / TTS)"] -->|avatar speech audio| B["Spatius Server SDK"]
B -->|audio| C["Motion Server"]
C -->|motion data| B
A -->|response audio| D["Client"]
B -->|motion data| D
```
## Before you start
Get the App ID and API Key from [Apps](https://app.spatius.ai/apps) (**Developer → API Key**) and the Avatar ID from the [Avatar Library](https://app.spatius.ai/avatars/library).
Keep the API Key on your backend. The App ID and Avatar ID must match the values used by the client.
## Run the Python reference implementation
The Backend Mode demo contains a complete Python backend and Web, iOS, Android, and Flutter clients:
```bash theme={null}
git clone https://github.com/spatius-ai/spatius-integration-demo.git
cd spatius-integration-demo/backend-mode
cp servers/python/.env.example servers/python/.env
# Fill the required values in servers/python/.env
./start.sh
```
Open the URL printed by the script, initialize the Avatar, then send text or microphone input.
## Build your backend
```bash theme={null}
pip install spatius
```
```bash theme={null}
go get github.com/spatius-ai/spatius-sdk-go
```
Configure the session with the API Key, App ID, Avatar ID, region, audio format, and a motion data callback. Then initialize the session and open its Motion Server connection.
```python theme={null}
from datetime import datetime, timedelta, timezone
from spatius import AudioFormat, new_avatar_session
session = new_avatar_session(
api_key="your-api-key",
app_id="your-app-id",
avatar_id="your-avatar-id",
region="us-west",
expire_at=datetime.now(timezone.utc) + timedelta(minutes=5),
sample_rate=16000,
audio_format=AudioFormat.PCM_S16LE,
transport_frames=lambda payload, last: print(
f"Motion payload: {len(payload)} bytes, last={last}"
),
)
await session.init()
connection_id = await session.start()
print(f"Connected: {connection_id}")
```
```go theme={null}
package avatar
import (
"context"
"log"
"time"
spatius "github.com/spatius-ai/spatius-sdk-go"
)
func StartSession(ctx context.Context) (*spatius.AvatarSession, error) {
session := spatius.NewAvatarSession(
spatius.WithAPIKey("your-api-key"),
spatius.WithAppID("your-app-id"),
spatius.WithAvatarID("your-avatar-id"),
spatius.WithRegion("us-west"),
spatius.WithExpireAt(time.Now().Add(5*time.Minute).UTC()),
spatius.WithSampleRate(16000),
spatius.WithAudioFormat(spatius.AudioFormatPCMS16LE),
spatius.WithTransportFrames(func(payload []byte, last bool) {
log.Printf("Motion payload: %d bytes, last=%t", len(payload), last)
}),
)
if err := session.Init(ctx); err != nil {
return nil, err
}
connectionID, err := session.Start(ctx)
if err != nil {
_ = session.Close()
return nil, err
}
log.Printf("Connected: %s", connectionID)
return session, nil
}
```
Assign the response a turn ID. Send its audio to the client first, then send the same audio to Spatius. Forward every motion data payload from the Server SDK callback to the client with that turn ID. Keep response turns serial unless your delivery layer can route overlapping callbacks correctly.
Your application transport can be WebSocket or HTTP streaming. It must preserve message order and the turn ID. See the [complete Python implementation](https://github.com/spatius-ai/spatius-integration-demo/blob/main/backend-mode/servers/python/app/session.py) or the [Go SDK lifecycle](/sdk-reference/go-sdk/go-sdk#session-lifecycle).
Stop forwarding a discarded turn when the conversation is cancelled, and close the Avatar session when the client leaves. Exact lifecycle APIs are listed in the Server SDK Reference.
## Next steps
# Audio
Source: https://docs.spatius.ai/concepts/audio
How avatar speech audio is formatted, sent, finalized, and interrupted for synchronized Avatar playback.
**Avatar speech audio** is the audio the Avatar should speak, usually TTS output from a voice-agent pipeline. It is not the user's microphone audio.
Motion Server uses this audio to generate synchronized motion data. AvatarKit plays the speech audio locally while rendering the resulting motion.
## Send timing
Send each new audio chunk when it is generated. Do not delay chunks to match wall-clock playback time.
Motion Server needs enough audio to generate the next motion window before AvatarKit consumes the current one. TTS output usually arrives faster than playback, allowing both buffers to stay ahead.
Do not feed audio that is already arriving at 1x playback speed back into Spatius chunk by chunk. The current segment can finish before the next synchronized segment is ready, causing playback to stall.
### If your source is paced
If 1x playback-speed audio is your only source, pre-buffer it before forwarding it. Start each speech turn with enough buffered audio for Motion Server to remain ahead, then continue filling that buffer while sending. This adds startup latency but avoids repeated playback gaps.
Reset the buffer when a turn is interrupted or a new turn begins.
## Input format
The canonical input is **mono 16-bit PCM (`s16le`)** at the sample rate configured for the session. Convert source audio first when its channels, encoding, or sample rate do not match.
Some SDK entry points also accept Opus. Format support and configuration differ by platform and integration, so use the exact SDK reference rather than mixing PCM and Opus within one session:
[Web](/sdk-reference/web-sdk/reference#configuration) · [iOS](/sdk-reference/ios-sdk/api-reference#audioformat) · [Android](/sdk-reference/android-sdk/api-reference#audioformat) · [Flutter](/sdk-reference/flutter-sdk/api-reference#initialize)
## End input or interrupt
These actions have different meanings:
* **End input** marks the final chunk of the current speech turn. Motion Server can finalize that turn, and AvatarKit continues playing the buffered result.
* **Interrupt** cancels the active turn and clears queued audio and motion data. Use it for barge-in or when the response should stop immediately.
Do not use interruption as the normal end marker for every response.
# Avatar
Source: https://docs.spatius.ai/concepts/avatar
Avatar IDs, avatar assets, and how AvatarKit loads them.
An **Avatar** is the 3D character rendered by AvatarKit on your user's device.
## Sources
### Public Avatars
Official characters or Avatars made public by other people. You can browse them on [app.spatius.ai](https://app.spatius.ai/avatars/library), copy the `avatar-id`, and use it directly.
![Public Avatars]()
### Personal Avatars
Avatars you create yourself on [Spatius Studio](https://app.spatius.ai).
## Related Concepts
### `avatar-id`
The unique ID of an Avatar. The client passes it to AvatarKit when loading the corresponding avatar assets.
Reference: [Web](/sdk-reference/web-sdk/reference#avatarmanager) | [iOS](/sdk-reference/ios-sdk/api-reference#avatarmanager) | [Android](/sdk-reference/android-sdk/api-reference#avatarmanager) | [Flutter](/sdk-reference/flutter-sdk/api-reference#load-an-avatar)
### `avatar assets`
The Avatar asset package downloaded by the client SDK.
* **Format**: 3DGS (3D Gaussian Splatting) avatar assets.
* **Size**: About 5-10 MB.
* **Limit**: The Avatar can only be rendered after the download completes.
# Avatar Background
Source: https://docs.spatius.ai/concepts/avatar-background
Download the optional 16:9 Avatar background from Spatius Studio and render it behind AvatarKit in your app.
An **Avatar Background** is an optional image generated with a Personal Avatar. It is a separate client-side asset, not part of the avatar assets or Motion Server flow.
## Mental model
Treat the background and Avatar as one fixed stage:
* The background is the **16:9 master stage**.
* The Avatar keeps the same position and scale inside that stage.
* Other aspect ratios crop a window from the same stage.
![Avatar rendered on a 16:9 background stage]()
The downloaded asset is that stage without the Avatar:
![Extracted 16:9 Avatar background asset]()
## Get the background asset
Go to [Spatius Studio](https://app.spatius.ai), open the Personal Avatar, and find the **Background** card below the preview.
Select **Download**. The image is the 16:9 master stage without the Avatar.
Bundle the image with the client or upload it to infrastructure your application controls.
`AvatarManager` loads avatar assets, but not this background image. If your app supports multiple Avatars, map each `avatar-id` to its background in your application.
## Render the two layers
Place a transparent Avatar view above the background inside the same stage:
```text theme={null}
Display window (clips overflow)
└── 16:9 stage
├── Background image
└── Transparent Avatar view
```
Both layers must fill the stage and share the same coordinate system. This composition is entirely client-side and works with every integration.
Implementation guides: [Web](/sdk-reference/web-sdk/reference#render-over-an-avatar-background) · [iOS](/sdk-reference/ios-sdk/api-reference#render-over-an-avatar-background) · [Android](/sdk-reference/android-sdk/api-reference#render-over-an-avatar-background) · [Flutter](/sdk-reference/flutter-sdk/api-reference#render-over-an-avatar-background)
## Crop for other aspect ratios
Use aspect-fill and center-crop for square, portrait, or custom containers. Keep the 16:9 stage centered and clip its overflow; do not stretch the background or move the Avatar independently.
The Avatar and background must be transformed together. Independent scaling or positioning breaks their spatial relationship.
## Practical rules
* Store the image as a normal Web, iOS, Android, or Flutter application asset, or serve it from your CDN.
* Preload it while the Avatar loads so both layers can appear together.
* Switch the Avatar and its mapped background in the same visual update.
* If the image is unavailable, fall back to an application-provided scene without blocking AvatarKit.
## Next steps
# How It Works
Source: https://docs.spatius.ai/concepts/how-it-works
Spatius architecture: Motion Server receives avatar speech audio, returns motion data, and AvatarKit renders locally.
Spatius does not stream finished Avatar video to the client. Instead, it **renders the Avatar locally on the client**. Motion Server returns motion data rather than video; delivery depends on the selected integration.
## Overall Flow
```mermaid actions={false} theme={null}
---
config:
"look": "handDrawn"
"theme": "base"
"themeVariables":
"background": "#ffffff"
"textColor": "#111827"
"lineColor": "#64748b"
"primaryColor": "#e8f4fd"
"primaryTextColor": "#111827"
"primaryBorderColor": "#2196F3"
"secondaryColor": "#f3e5f5"
"secondaryTextColor": "#111827"
"secondaryBorderColor": "#9C27B0"
"tertiaryColor": "#fff3e0"
"tertiaryTextColor": "#111827"
"tertiaryBorderColor": "#FF9800"
"clusterBkg": "#f8fafc"
"clusterBorder": "#cbd5e1"
"edgeLabelBackground": "#ffffff"
---
graph LR
A["avatar speech audio
usually TTS output"]
B["Motion Server
generates motion data"]
C["AvatarKit
local playback and rendering"]
D["your app
observes state and handles recovery"]
A --> B
A -->|"audio"| C
B -->|"motion data"| C
C --> D
```
> For one Avatar response: the audio the Avatar should speak is sent to Motion Server, Motion Server generates motion data, AvatarKit plays the audio locally while rendering synchronized lip and body motion, and your app observes the process through state callbacks.
## Core Components
### AvatarKit Client SDK
The client SDK covers Web, iOS, Android, and Flutter. It renders the Avatar on the client and keeps avatar speech audio and motion data synchronized.
Reference: [Web](/sdk-reference/web-sdk/reference) | [iOS](/sdk-reference/ios-sdk/api-reference) | [Android](/sdk-reference/android-sdk/api-reference) | [Flutter](/sdk-reference/flutter-sdk/api-reference)
### Motion Server
> Cloud service provided by Spatius.
* **Input**: avatar speech audio, usually the TTS output from an ASR -> LLM -> TTS pipeline.
* **Output**: motion data, \~10–15 KB/s, much lower than a video stream.
* **Role**: provides motion data that AvatarKit uses to animate the Avatar. **AvatarKit synchronizes audio and motion data playback, so application developers don't need to manage it.**
## Next steps
# Client Lifecycle
Source: https://docs.spatius.ai/concepts/lifecycle
The AvatarKit client lifecycle from initialization and asset loading through rendering, interaction, and cleanup.
AvatarKit follows the same client lifecycle across integrations. What changes is **who owns the connection and audio path**, not how the Avatar is loaded and rendered.
```mermaid actions={false} theme={null}
---
config:
"look": "handDrawn"
"theme": "base"
"themeVariables":
"background": "#ffffff"
"textColor": "#111827"
"lineColor": "#64748b"
"primaryColor": "#e8f4fd"
"primaryTextColor": "#111827"
"primaryBorderColor": "#2196F3"
---
graph LR
A["Initialize
configure AvatarKit"]
B["Load
download avatar assets"]
C["Render
mount the Avatar view"]
D["Connect
start or join the data path"]
E["Interact
play, pause, interrupt"]
F["Cleanup
release client resources"]
A --> B --> C --> D --> E --> F
```
## Initialize
Initialize AvatarKit once when the application starts. This establishes shared configuration before any Avatar is loaded.
## Load
Load the selected `avatar-id` before creating its view. AvatarKit downloads and caches the avatar assets; loading the same Avatar again can reuse that cache.
See [Avatar](/concepts/avatar) for the asset mental model.
## Render
Mount the loaded Avatar into an Avatar view. The view owns the render surface and its controller, and can show idle animation before a conversation begins.
Treat the view and its controller as one lifecycle unit. Recreate or release them together when the Avatar changes.
## Connect
Connect only after the Avatar is ready to render. The selected integration determines the connection owner:
* In Direct Mode, AvatarKit connects to Motion Server.
* In the recommended integrations, the client joins the configured RTC room while the agent runtime owns the Motion Server session.
* In Backend Mode, your backend owns the Motion Server session and delivers data to the client.
Follow the setup and Client pages for your path from [Integrations](/integrations/overview). Do not reuse a Direct Mode connection sequence in another integration.
## Interact
During a response, AvatarKit keeps avatar speech audio and motion data synchronized. Your UI can observe conversation state and control playback, including pause, resume, and interruption.
See [Audio](/concepts/audio) for input timing and [Client State & Events](/concepts/state-events) for observable state.
## Cleanup
Release the Avatar view, controller, and any transport listeners when the screen or session ends. Cleanup behavior differs slightly by client platform; use the platform reference for the exact API:
[Web](/sdk-reference/web-sdk/reference#avatarview) · [iOS](/sdk-reference/ios-sdk/api-reference#avatarview) · [Android](/sdk-reference/android-sdk/api-reference#avatarview) · [Flutter](/sdk-reference/flutter-sdk/api-reference#render-and-control-playback)
Do not leave an old view or transport subscription alive after switching Avatars. Stale callbacks can update a view that is no longer visible.
# Client State & Events
Source: https://docs.spatius.ai/concepts/state-events
The two client state dimensions to observe: Avatar conversation playback and Direct Mode connectivity.
Treat Avatar playback and network connectivity as two separate dimensions. One can change without the other.
## `ConversationState`
`ConversationState` answers: **What is the Avatar doing now?**
Use it to drive speaking indicators, pause controls, and idle UI. A conversation typically moves between idle, active playback, and paused behavior, but the exact enum cases and intermediate states vary by platform.
Do not infer network health from conversation state. An Avatar can be idle while its data path remains connected.
Platform definitions: [Web](/sdk-reference/web-sdk/reference#conversationstate) · [iOS](/sdk-reference/ios-sdk/api-reference#conversationstate) · [Android](/sdk-reference/android-sdk/api-reference#conversationstate) · [Flutter](/sdk-reference/flutter-sdk/api-reference#render-and-control-playback)
## `ConnectionState`
`ConnectionState` answers: **Can AvatarKit reach Motion Server?**
This AvatarKit state applies to **Direct Mode**, where the client owns that connection. It is useful for connection UI, retry decisions, and distinguishing startup from an established session.
LiveKit Agents, Agora Convo AI, and Backend Mode own connectivity outside the core AvatarKit controller. Observe the RTC provider or your application transport for those paths instead.
Platform definitions: [Web](/sdk-reference/web-sdk/reference#connectionstate) · [iOS](/sdk-reference/ios-sdk/api-reference#connectionstate) · [Android](/sdk-reference/android-sdk/api-reference#connectionstate) · [Flutter](/sdk-reference/flutter-sdk/api-reference#render-and-control-playback)
## Errors and recovery
Handle errors according to the lifecycle phase that produced them. See [Client Error Handling](/resources/client-error) for recovery guidance and the platform SDK references for exact error types.
# Direct Mode Client
Source: https://docs.spatius.ai/direct-mode/client
Connect AvatarKit directly to Motion Server from Web, iOS, Android, or Flutter.
## Before you start
Copy the App ID and API Key from [Apps](https://app.spatius.ai/apps) and the Avatar ID from the [Avatar Library](https://app.spatius.ai/avatars/library). Keep the API Key on your backend and send the client only the App ID, Avatar ID, and a short-lived [Session Token](/api-reference/auth).
## Choose your client
Every client follows the same path: initialize Direct Mode and set the Session Token → load and mount the Avatar → connect and send avatar speech audio → close and dispose.
Install AvatarKit:
```bash theme={null}
npm install @spatius/avatarkit
```
Configure the WebAssembly assets for [Vite or Next.js](/sdk-reference/web-sdk/toolchain) before initializing the SDK.
[Web demo](https://github.com/spatius-ai/spatius-integration-demo/tree/main/direct-mode/clients/web/reference) · [Web SDK Reference](/sdk-reference/web-sdk/reference)
Install AvatarKit with Swift Package Manager, CocoaPods, or `AvatarKit.xcframework`. See [iOS installation](/sdk-reference/ios-sdk/api-reference#installation).
[iOS demo](https://github.com/spatius-ai/spatius-integration-demo/tree/main/direct-mode/clients/ios) · [iOS SDK Reference](/sdk-reference/ios-sdk/api-reference)
Add the latest standard `ai.spatius:avatarkit` release to your app. See [Android installation](/sdk-reference/android-sdk/api-reference#installation).
[Android demo](https://github.com/spatius-ai/spatius-integration-demo/tree/main/direct-mode/clients/android) · [Android SDK Reference](/sdk-reference/android-sdk/api-reference)
Add AvatarKit:
```bash theme={null}
flutter pub add spatius_avatarkit
```
[Flutter demo](https://github.com/spatius-ai/spatius-integration-demo/tree/main/direct-mode/clients/flutter) · [Flutter SDK Reference](/sdk-reference/flutter-sdk/api-reference)
## Next steps
# Welcome to Spatius
Source: https://docs.spatius.ai/getting-started
What Spatius is, the core components, and how to start integrating.
Spatius turns avatar speech audio into real-time motion data. Your client renders the avatar locally, so you do not need to stream finished avatar video.
## Core components
| Term | What it is |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Spatius** | The platform — [Spatius Studio](https://app.spatius.ai) for managing apps and avatars, Motion Server in the cloud, and the AvatarKit SDKs you ship in your app. |
| **AvatarKit** | The client SDK family that downloads avatar assets, renders the avatar locally, and plays synchronized audio. Available for Web, iOS, Android, and Flutter. |
| **Motion Server** | The Spatius cloud service that receives avatar speech audio and returns lip-sync motion data, \~10–15 KB/s. |
## Next steps
# Integrations
Source: https://docs.spatius.ai/integrations/overview
Explore Spatius data flows, then compare delivery latency, development effort, and ideal use cases.
## Compare integrations
| Integration | Delivery latency | Dev effort | Best for |
| ---------------------------------------------------------------------------- | ---------------- | ---------- | ----------------------------- |
| **[LiveKit Agents](/livekit-agents/overview)**
Recommended | Low ⚡ | Low 🟢 | LiveKit voice agents |
| **[Agora Convo AI](/agora-convoai/overview)**
Recommended | Low ⚡ | Low 🟢 | Agora Convo AI or TEN |
| **[Direct Mode](/direct-mode/client)** | Variable | Medium 🟡 | Audio available on the client |
| **[Backend Mode](/backend-mode/server-sdk)** | Low ⚡ | High 🔴 | Backend-owned audio pipeline |
*Relative delivery latency after audio is produced; Direct Mode varies with the client network.*
# LiveKit Agents Integration Client
Source: https://docs.spatius.ai/livekit-agents/client
Connect a Web client to a LiveKit Agents room and render the avatar.
## Before you start
* Complete the [Agent setup](/livekit-agents/server).
* Create a backend endpoint that returns a short-lived LiveKit token, server URL, and room name for the client.
* Use the same Spatius App ID and Avatar ID as the worker. Keep the Spatius API Key on the worker.
This packaged LiveKit client path is currently available on Web. The native RTC SDKs expose an `RTCProvider` extension point, but do not bundle a native `LiveKitProvider`.
## Install
```bash theme={null}
npm install @spatius/avatarkit @spatius/avatarkit-rtc livekit-client
```
Configure your build tool to load the AvatarKit WebAssembly assets. See [Toolchain Setup](/sdk-reference/web-sdk/toolchain).
## Connect and render
```typescript theme={null}
import { AvatarPlayer, LiveKitProvider } from '@spatius/avatarkit-rtc'
import { AvatarSDK, AvatarView, AvatarManager, DrivingServiceMode } from '@spatius/avatarkit'
interface LiveKitSession {
url: string
token: string
roomName: string
}
export async function startAvatar(
container: HTMLElement,
session: LiveKitSession,
) {
await AvatarSDK.initialize('your-spatius-app-id', {
drivingServiceMode: DrivingServiceMode.rtc,
})
const avatar = await AvatarManager.shared.load('your-spatius-avatar-id')
const avatarView = new AvatarView(avatar, container)
const provider = new LiveKitProvider()
const player = new AvatarPlayer(provider, avatarView)
await player.connect(session)
await player.startPublishing()
return {
player,
async dispose() {
await player.stopPublishing()
await player.disconnect()
avatarView.dispose()
},
}
}
```
`startPublishing()` requests microphone permission, so call `startAvatar()` from a user action. Run the returned `dispose()` function when the experience ends.
The adapter creates and owns the LiveKit `Room` in this path. If your app must supply an existing room, follow the advanced `attach()` flow in the [RTC Adapter reference](/sdk-reference/web-sdk/rtc-adapter#host-owned-rtc-clients).
## Next steps
# LiveKit Agents Integration
Source: https://docs.spatius.ai/livekit-agents/overview
Use LiveKit Agents and Spatius to build voice avatars.
## Architecture
## Runtime boundary
Your LiveKit Agents worker owns the voice agent and sends its speech audio to Motion Server through `livekit-plugins-spatius`. The Web client joins the same room, while AvatarKit renders the avatar locally from the synchronized audio and motion data.
## Next steps
# Set Up LiveKit Agents Integration
Source: https://docs.spatius.ai/livekit-agents/server
Add livekit-plugins-spatius to a LiveKit Agents worker.
## Before you start
Use an existing LiveKit Agents worker and a room that your client can join. Keep its `LIVEKIT_URL`, `LIVEKIT_API_KEY`, and `LIVEKIT_API_SECRET` configured. Get the Spatius App ID and API Key from [Apps](https://app.spatius.ai/apps) and the Avatar ID from the [Avatar Library](https://app.spatius.ai/avatars/library).
## Install
```bash theme={null}
pip install livekit-plugins-spatius
```
## Configure
Keep the API Key in the worker. The current plugin selects a region automatically when `SPATIUS_REGION` is unset. Only set it when you need to pin `us-west`, `ap-northeast`, or `cn-beijing`. See [Regions](/api-reference/regions) for endpoint details.
```bash title=".env" theme={null}
SPATIUS_API_KEY=your-api-key
SPATIUS_APP_ID=your-app-id
SPATIUS_AVATAR_ID=your-avatar-id
```
Load these values through your worker's existing environment configuration before creating `AvatarSession`. If you use a local `.env` file, keep it out of version control.
## Add the Avatar session
Connect the worker to the LiveKit room, start `AvatarSession`, then start the agent session. The avatar publishes into the same room.
`AvatarSession.start()` registers playback RPC handlers on the worker's local participant. Call `await ctx.connect()` before `await avatar.start(...)`; otherwise startup fails with `cannot access local participant before connecting`.
Call this helper from your existing LiveKit Agents entrypoint with your configured `AgentSession` and `Agent`. It preserves your choice of STT, LLM, and TTS.
```python theme={null}
from livekit.agents import Agent, AgentSession, JobContext
from livekit.agents.voice.room_io import RoomOptions
from livekit.plugins.spatius import AvatarSession
async def start_avatar_session(
ctx: JobContext, session: AgentSession, agent: Agent
) -> None:
await ctx.connect()
# Read Spatius credentials and the avatar ID from the environment.
avatar = AvatarSession()
await avatar.start(session, room=ctx.room)
await session.start(
agent=agent,
room=ctx.room,
room_options=RoomOptions(audio_output=False),
)
```
Merge this sequence into your entrypoint rather than adding a second `session.start()` call. If you already connect to the room, keep that connection before avatar startup.
The plugin sends the agent's speech audio to Motion Server and publishes synchronized audio and motion data into the room. `audio_output=False` disables the agent's direct room audio output so the avatar provides playback. Keep any existing input or text settings when merging `RoomOptions`.
### Choose the avatar per session
`SPATIUS_AVATAR_ID` is a convenient default for a fixed-avatar worker. To choose an avatar per conversation, pass the ID explicitly instead:
```python theme={null}
avatar = AvatarSession(avatar_id=validated_avatar_id)
```
Your business layer can supply the ID in LiveKit dispatch metadata (`ctx.job.metadata`). Parse and validate your application's metadata contract before creating the avatar session. In that flow, omit `SPATIUS_AVATAR_ID` from the worker environment; `SPATIUS_APP_ID` and `SPATIUS_API_KEY` remain worker configuration. The plugin does not require a particular metadata schema.
## Run locally
With dependencies installed and your environment configured, run the current [LiveKit CLI](https://docs.livekit.io/reference/developer-tools/livekit-cli/#setup) from the Python agent project directory:
```bash theme={null}
lk agent dev
```
The CLI detects `agent.py` or `src/agent.py` and reloads the worker when files change. For another entrypoint, pass its path, for example `lk agent dev worker.py`. See [Agent commands](https://docs.livekit.io/reference/developer-tools/livekit-cli/agent/#dev).
The Python CLI's `dev` mode is deprecated and no longer provides in-process hot reload. Use `lk agent dev` instead of `python agent.py dev`.
## Troubleshooting
* **`cannot access local participant before connecting`:** Connect with `await ctx.connect()` before starting the avatar. A registered worker can receive jobs before its job process joins the room. The plugin's final credential or network error message can wrap this connection-order error; check the underlying traceback first.
* **`no warmed process available`:** The development worker may create its job process on demand. If the next logs show successful process initialization, this warning indicates a cold start, not an avatar failure.
## Next steps
# Android Quickstart
Source: https://docs.spatius.ai/quickstarts/android-sdk
Run the native Android scenario demo with AvatarKit.
Run four complete AvatarKit scenes on Android. Current demo integration: Agora Convo AI.
## Prerequisites
* Android Studio with its bundled JDK
* An Android device or emulator that supports the AvatarKit native libraries
* Python 3
* Spatius API Key and App ID from [Spatius Studio](https://app.spatius.ai/apps)
* Agora App ID, App Certificate, and published Agent Pipeline ID
## Run it
```bash theme={null}
git clone https://github.com/spatius-ai/spatius-scenario-demo.git
cd spatius-scenario-demo/backend
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
```
Set `TRANSPORT=agora` and fill the Spatius and Agora credentials in `.env`. Follow the [backend README](https://github.com/spatius-ai/spatius-scenario-demo/tree/main/backend) to align the ASR resource IDs and TTS sample rate with your published Agora agent, then start the backend:
```bash theme={null}
python server.py
```
The demo backend is unauthenticated. Run it only on your local machine or a trusted LAN; do not expose it publicly.
Keep the printed LAN address—you will enter it on the Android config screen.
In a second terminal:
```bash theme={null}
cd spatius-scenario-demo/android
./gradlew :app:installDebug
```
You can also open `android/` in Android Studio and run the app from there.
On the config screen, enter the backend LAN address, confirm the credentials, and choose a scene.
## Next steps
# iOS Quickstart
Source: https://docs.spatius.ai/quickstarts/ios-sdk
Run the native iOS scenario demo with AvatarKit.
Run four complete AvatarKit scenes on an iOS device. Current demo integration: Agora Convo AI.
## Prerequisites
* macOS with Xcode and [XcodeGen](https://github.com/yonaskolb/XcodeGen)
* A physical iOS device; AvatarKit and Agora do not include simulator slices in this demo
* Python 3
* Spatius API Key and App ID from [Spatius Studio](https://app.spatius.ai/apps)
* Agora App ID, App Certificate, and published Agent Pipeline ID
## Run it
```bash theme={null}
git clone https://github.com/spatius-ai/spatius-scenario-demo.git
cd spatius-scenario-demo/backend
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
```
Set `TRANSPORT=agora` and fill the Spatius and Agora credentials in `.env`. Follow the [backend README](https://github.com/spatius-ai/spatius-scenario-demo/tree/main/backend) to align the ASR resource IDs and TTS sample rate with your published Agora agent, then start the backend:
```bash theme={null}
python server.py
```
The demo backend is unauthenticated. Run it only on your local machine or a trusted LAN; do not expose it publicly.
Keep the printed LAN address—you will enter it on the iOS config screen.
In a second terminal:
```bash theme={null}
cd spatius-scenario-demo/ios
xcodegen generate
open SpatiusScenes.xcodeproj
```
Build to a connected iOS device. On the config screen, enter the backend LAN address, confirm the credentials, and choose a scene.
## Next steps
# Web Quickstart
Source: https://docs.spatius.ai/quickstarts/web-sdk
Run the Web scenario demo with LiveKit Agents or Agora Convo AI.
Run four complete avatar scenes in the browser.
The Web demo supports both recommended integrations. Choose `livekit` to run the conversation through LiveKit Agents on your machine, or `agora` to use Agora Convo AI.
## Prerequisites
* Python 3 and Node.js with pnpm
* Spatius API Key and App ID from [Spatius Studio](https://app.spatius.ai/apps)
* One realtime provider:
* LiveKit URL, API Key, and API Secret from [LiveKit Cloud](https://cloud.livekit.io)
* Or Agora App ID, App Certificate, and published Agent Pipeline ID
## Run it
```bash theme={null}
git clone https://github.com/spatius-ai/spatius-scenario-demo.git
cd spatius-scenario-demo
```
```bash theme={null}
cd backend
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
```
In `.env`, set `TRANSPORT=livekit` or `TRANSPORT=agora`, then fill the Spatius and matching provider credentials. The [backend README](https://github.com/spatius-ai/spatius-scenario-demo/tree/main/backend) lists every field. For Agora, also follow its required ASR resource-ID and TTS sample-rate setup.
```bash theme={null}
python server.py
```
Keep it running. It listens on port `8787` and starts the LiveKit agent worker automatically when `TRANSPORT=livekit`.
The demo backend is unauthenticated. Run it only on your local machine or a trusted LAN; do not expose it publicly.
In a second terminal:
```bash theme={null}
cd spatius-scenario-demo/web
pnpm install
pnpm dev
```
Open `http://localhost:5180/spatius-scenario-demo/`, confirm the backend configuration, then choose a scene.
## Next steps
# Reference Overview
Source: https://docs.spatius.ai/reference/overview
A lookup map for Spatius Client SDKs, Server SDKs, Session Token API pages, regions, endpoints, and error codes.
Use the Reference tab when you need exact names: classes, methods, enums, request fields, endpoint regions, and error codes. Use Guides when you need to choose an integration or follow a step-by-step setup.
## Reference sections
| Section | What it contains | Use it when |
| ------------------- | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| Client SDKs | Web, iOS, Android, Flutter, and Web RTC Adapter reference pages. | You need AvatarKit classes, methods, callbacks, enums, rendering APIs, or client lifecycle details. |
| Server SDKs | Python and Go SDK references. | Your backend connects to Motion Server in Backend Mode or a packaged integration runtime. |
| Session Token API | Console API pages for Session Token creation and Direct Mode auth flow. | Your backend mints short-lived Session Tokens for Direct Mode clients. |
| Regions & Endpoints | Supported region names and composed Console API / Motion Server endpoints. | Any integration needs a deployment region or endpoint reference. |
| Error handling | Client and server recovery guidance. | You are debugging failed sessions, asset loading, WebSocket connection, or Server SDK errors. |
## Client SDK lookup
| Platform or package | Reference page | Main use |
| ---------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Web `@spatius/avatarkit` | [AvatarKit Web SDK Reference](/sdk-reference/web-sdk/reference) | Web AvatarKit initialization, loading, rendering, Direct Mode, Backend Mode client feed, state, enums, and errors. |
| Web toolchain | [Toolchain Setup](/sdk-reference/web-sdk/toolchain) | Configure bundlers to serve AvatarKit WebAssembly assets correctly. |
| Web RTC Adapter `@spatius/avatarkit-rtc` | [RTC Adapter](/sdk-reference/web-sdk/rtc-adapter) | Render avatar output from LiveKit or Agora RTC providers on Web. |
| iOS AvatarKit | [iOS SDK Reference](/sdk-reference/ios-sdk/api-reference) | Native iOS AvatarKit methods, enums, callbacks, and rendering controls. |
| Android AvatarKit | [Android SDK Reference](/sdk-reference/android-sdk/api-reference) | Native Android AvatarKit methods, enums, callbacks, and rendering controls. |
| Flutter `spatius_avatarkit` | [Flutter SDK Reference](/sdk-reference/flutter-sdk/api-reference) | Flutter setup, Direct Mode, Backend Mode client feed, state callbacks, and demos. |
## Server lookup
| Surface | Reference page | Main use |
| ----------------- | -------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| Python Server SDK | [Python SDK](/sdk-reference/python-sdk/python-sdk) | Backend Mode sessions, audio input, egress, encoded output callbacks, options, and errors. |
| Go Server SDK | [Go SDK](/sdk-reference/go-sdk/go-sdk) | Go Backend Mode sessions, audio input, egress, encoded output callbacks, options, and errors. |
| Session Token API | [Session Token API](/api-reference/api-reference) | Mint Direct Mode Session Tokens from your backend. |
| Auth Flow | [Session Token Auth Flow](/api-reference/auth) | Understand how a Direct Mode client obtains and uses a Session Token. |
## Shared lookup
| Need | Page |
| ------------------------------------- | ---------------------------------------------------- |
| Which SDK supports which feature | [SDK Capability Matrix](/reference/sdk-capabilities) |
| Region names and endpoint composition | [Regions](/api-reference/regions) |
| Client errors | [Client Error Handling](/resources/client-error) |
| Server SDK errors | [Server Error Handling](/resources/server-error) |
| Demo repo by integration | [Demo Matrix](/resources/demo-projects) |
## Next steps
# SDK Capability Matrix
Source: https://docs.spatius.ai/reference/sdk-capabilities
Documented Spatius SDK capability matrix across Web, iOS, Android, Flutter, Python Server SDK, and Go Server SDK.
This matrix shows capabilities that are documented in the public reference pages. It is a lookup aid, not a replacement for the platform reference pages.
If a cell says **Not documented**, the current docs do not expose a public reference for that capability on that SDK. It does not prove the runtime cannot support it.
## Client SDK capabilities
| Capability | Web | iOS | Android | Flutter |
| ---------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ---------------------------------------- |
| Direct Mode Integration | Documented | Documented | Documented | Documented |
| Backend Mode client feed | `yieldAudioData()` / `yieldFramesData()` | `yieldAudioData(_:end:)` / `yieldFramesData(_:conversationID:)` | `yieldAudioData(audioData, end)` / `yieldFramesData(animations, conversationId)` | `yieldAudioData()` / `yieldAnimations()` |
| RTC client/provider support | `@spatius/avatarkit-rtc` | `AvatarKitRTC` with a public `RTCProvider` abstraction; Agora provider included | `ai.spatius:avatarkit-rtc` with a public `RTCProvider` abstraction; Agora provider included | Not documented |
| Region config | `region` in `AvatarSDK.initialize()` | `region` in `Configuration` | `region` in `Configuration` | `region` in `Configuration` |
| Region default | Automatic selection when unset | Automatic selection when unset | Automatic selection when unset | Automatic selection when unset |
| Opus audio input | `inputAudioFormat: 'opus'` | `inputAudioFormat: .opus` | `inputAudioFormat = AudioCodec.OPUS` | `inputAudioFormat: AudioCodec.opus` |
| Direct Mode Opus uplink | On by default (`opusUplinkEnabled`) | On by default (`opusUplinkEnabled`) | On by default (`opusUplinkEnabled`) | On by default (`opusUplinkEnabled`) |
| Driving mode enum | `DrivingServiceMode.direct` / `.backend` / `.rtc` | `.direct` / `.backend` / `.rtc` | `DIRECT` / `BACKEND` / `RTC` | `direct` / `backend` |
| Render quality | `RenderQuality` and `setRenderQuality()` | `RenderQuality` and `setRenderQuality(_:)` | `RenderQuality` and `setRenderQuality()` | `RenderQuality` and `setRenderQuality()` |
| Render resolution cap | `setRenderResolutionCap()` | `setRenderResolutionCap(enabled:maxHeight:)` | `setRenderResolutionCap(enabled,maxHeight)` | Not documented |
| Device capability check | `isDeviceSupported()` / `deviceScore()` | `isDeviceSupported()` / `deviceScore()` | `isDeviceSupported()` / `deviceScore()` | Not documented |
| Frame starvation behavior | `FrameStarvationMode` | `FrameStarvationMode` | `FrameStarvationMode` | Not documented |
| Strict sync mode | `FrameStarvationMode.strictSync` | `FrameStarvationMode` enum | `STRICT_SYNC` enum value | Not documented |
| Avatar bitmap export | `AvatarView.exportBitmap()` | Not documented | Not documented | Not documented |
| Avatar transform / bounds | `avatarTransform`, `getBoundingRect()` | `AvatarView.avatarTransform`, `getBoundingRect()` | `AvatarView.avatarTransform`, `getBoundingRect()` | Not documented |
| `ConnectionState` callback | Direct Mode only | Documented | Documented | Documented |
| `ConversationState` callback | Documented | Documented | Documented | Documented |
| Interrupt playback | `interrupt()` | `interrupt()` | `interrupt()` | Not documented |
| Pause / resume rendering | `AvatarView.pauseRendering()` / `resumeRendering()` | `AvatarView.pauseRendering()` / `resumeRendering()` | `AvatarView.pauseRendering()` / `resumeRendering()` | Not documented |
| Cache management | `AvatarManager` cache methods | `AvatarManager` cache methods | `AvatarManager` cache methods | Not documented |
## Web package split
| Package | What it does | Reference |
| ------------------------ | ----------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| `@spatius/avatarkit` | Core Web AvatarKit SDK for loading, rendering, Direct Mode, and Backend Mode client feed. | [Web SDK Reference](/sdk-reference/web-sdk/reference) |
| `@spatius/avatarkit-rtc` | Web RTC transport adapter for LiveKit and Agora providers. | [RTC Adapter](/sdk-reference/web-sdk/rtc-adapter) |
## Server SDK capabilities
| Capability | Python Server SDK | Go Server SDK |
| ------------------------- | ----------------------------- | -------------------------- |
| Backend Mode session | Documented | Documented |
| Region config | `region` option | `WithRegion(region)` |
| Console endpoint override | `console_endpoint_url` | `WithConsoleEndpointURL()` |
| Ingress endpoint override | `ingress_endpoint_url` | `WithIngressEndpointURL()` |
| Audio input | Documented | Documented |
| Motion data callback | Documented | Documented |
| Encoded audio callback | Documented | Documented |
| LiveKit egress | Documented | Documented |
| Agora egress | Documented | `WithAgoraEgress()` |
| Interrupt | `interrupt()` for egress mode | `Interrupt()` |
| Structured SDK errors | `AvatarSDKError` fields | Documented error handling |
LiveKit and Agora egress are low-level Server SDK capabilities. They are not additional integration paths.
## Integration support
| Integration | Web | iOS | Android | Flutter | Server-side component |
| -------------------------- | ------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------ | -------------- | -------------------------------------- |
| Direct Mode Integration | Yes | Yes | Yes | Yes | Session Token endpoint |
| LiveKit Agents Integration | Bundled `LiveKitProvider` | `RTCProvider` extension point; native LiveKit provider not bundled | `RTCProvider` extension point; native LiveKit provider not bundled | Not documented | `livekit-plugins-spatius` |
| Agora Convo AI Integration | Web RTC client | AvatarKit RTC client | AvatarKit RTC client | Not documented | Agora avatar provider or TEN extension |
| Backend Mode Integration | Client feed | Client feed | Client feed | Client feed | Python or Go Server SDK |
## Region support
All documented SDKs and integrations use the same region names:
| Region | Status |
| -------------- | --------- |
| `us-west` | Supported |
| `ap-northeast` | Supported |
| `cn-beijing` | Supported |
See [Regions](/api-reference/regions) for endpoint composition and advanced override notes.
# Client Error Handling
Source: https://docs.spatius.ai/resources/client-error
How to classify and recover from AvatarKit loading, connection, audio, playback, and runtime errors.
Handle a client error according to **where it occurred in the lifecycle**. Exact error names and associated values differ by platform; use the SDK Reference as the source of truth.
| Lifecycle area | Representative signal | First action |
| -------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| Avatar loading | Unknown ID, metadata failure, asset download failure | Verify the `avatar-id`, then retry transient downloads. |
| Direct Mode authentication | Invalid or expired Session Token | Request a fresh token from your backend before reconnecting. |
| Direct Mode connection | Handshake, WebSocket, timeout, or unexpected close | Check region and network reachability, then reconnect with backoff. |
| Audio input | Input does not match the configured codec or sample rate | Convert the source and keep one configured format for the session. |
| Playback | Audio context or render player is not ready | Restore the required client lifecycle before retrying playback. |
| Backend Mode feed | Invalid or mismatched audio or motion data | Verify the configured format and ordering by turn ID. |
| Server response | Motion Server reports an application, billing, or runtime failure | Log the complete error and fix non-transient configuration failures before retrying. |
Spatius API Keys must never be used for client recovery. In Direct Mode, the backend exchanges the API Key for a short-lived Session Token and returns only that token to the client.
## Recovery flow
1. Log the error, integration, SDK version, region, and current lifecycle phase.
2. Retry only network and download failures that are likely to be transient.
3. Refresh credentials for authentication failures instead of repeatedly reconnecting with the same token.
4. Treat format, asset compatibility, and application configuration failures as developer errors.
5. Stop automatic retries after a bounded number of attempts and surface a recoverable UI state.
## Exact platform errors
* [Web SDK error handling](/sdk-reference/web-sdk/reference#error-handling)
* [iOS `AvatarError`](/sdk-reference/ios-sdk/api-reference#avatarerror)
* [Android SDK reference](/sdk-reference/android-sdk/api-reference)
* [Flutter SDK reference](/sdk-reference/flutter-sdk/api-reference)
* [Web RTC Adapter](/sdk-reference/web-sdk/rtc-adapter)
For backend failures, see [Server Error Handling](/resources/server-error).
# Demo Projects
Source: https://docs.spatius.ai/resources/demo-projects
Choose a complete scenario quickstart or a runnable integration demo.
## Scenario quickstarts
Complete scenarios for Web, iOS, and Android.
| Platform | Start here |
| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| **Web** | [Quickstart](/quickstarts/web-sdk) (LiveKit or Agora) · [Source](https://github.com/spatius-ai/spatius-scenario-demo/tree/main/web) |
| **iOS** | [Quickstart](/quickstarts/ios-sdk) (current demo: Agora) · [Source](https://github.com/spatius-ai/spatius-scenario-demo/tree/main/ios) |
| **Android** | [Quickstart](/quickstarts/android-sdk) (current demo: Agora) · [Source](https://github.com/spatius-ai/spatius-scenario-demo/tree/main/android) |
## Integration demos
Runnable servers and clients for every integration path.
| Need | Integration demo |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Direct Mode Web** | [React, Vue, vanilla, and Next.js clients](https://github.com/spatius-ai/spatius-integration-demo/tree/main/direct-mode/clients/web/reference) |
| **Direct Mode native** | [iOS](https://github.com/spatius-ai/spatius-integration-demo/tree/main/direct-mode/clients/ios) · [Android](https://github.com/spatius-ai/spatius-integration-demo/tree/main/direct-mode/clients/android) · [Flutter](https://github.com/spatius-ai/spatius-integration-demo/tree/main/direct-mode/clients/flutter) |
| **Backend Mode** | [Backend and client references](https://github.com/spatius-ai/spatius-integration-demo/tree/main/backend-mode) |
| **LiveKit Agents** | [Agent quickstart](https://github.com/spatius-ai/spatius-integration-demo/tree/main/platform-integrations/livekit-agents-demo/livekit-agent-quickstart) · [Full Web reference implementation](https://github.com/spatius-ai/spatius-integration-demo/tree/main/platform-integrations/livekit-agents-demo/livekit-agents-reference-demo) · [RTC clients and server](https://github.com/spatius-ai/spatius-integration-demo/tree/main/rtc-mode) |
| **Agora Convo AI** | [RTC clients for Web, iOS, and Android](https://github.com/spatius-ai/spatius-integration-demo/tree/main/rtc-mode) |
# FAQ
Source: https://docs.spatius.ai/resources/faq
Common questions about Avatars, audio input, integrations, regions, and developer support.
## Avatars and scenes
Use **Create Avatar** in [Spatius Studio](https://app.spatius.ai). The creation flow shows the current image requirements. After creation, use the Avatar's `avatar-id` in your app.
Yes. AvatarKit renders the Avatar with a transparent background, so your application can place an image, video, WebGL scene, or native view behind it.
A Personal Avatar may also have an optional 16:9 background available from Studio. See [Avatar Background](/concepts/avatar-background) for the download and layout flow.
## Audio
No. Motion Server receives the audio the Avatar should speak and generates motion data from it. If your application starts with text, first produce speech audio with TTS or a speech-to-speech model.
The canonical input is mono 16-bit PCM (`s16le`) at the sample rate configured for the session. Some SDK entry points also accept Opus, and RTC integrations receive audio through their provider path.
The configured format must match the bytes you send. See [Audio](/concepts/audio) and the SDK Reference for platform-specific formats.
Send new TTS chunks as they are generated instead of pacing them at playback speed. Motion Server needs enough buffered audio to prepare the next motion window before the current one finishes.
If only 1x paced audio is available, pre-buffer it at the start of each speech turn. See [Audio: Send timing](/concepts/audio#send-timing).
## Integrations
No. Spatius turns avatar speech audio into motion data and renders the Avatar. Your voice-agent platform or application owns microphone input, turn handling, ASR, LLM, and speech generation.
For a packaged agent path, use [LiveKit Agents Integration](/livekit-agents/overview) or [Agora Convo AI Integration](/agora-convoai/overview).
Use a recommended integration for LiveKit Agents or Agora Convo AI. Use Direct Mode when the client owns the audio source, or Backend Mode when your application owns client delivery.
See the concise comparison in [Integrations](/integrations/overview).
AvatarKit supports Web, iOS, Android, and Flutter. The packaged RTC-provider coverage varies by integration and platform; the core iOS and Android SDKs are not limited to Agora.
See [SDK Capability Matrix](/reference/sdk-capabilities) for the current package boundaries.
## Regions and support
Spatius operates in `us-west`, `ap-northeast`, and `cn-beijing`.
See [Regions & Endpoints](/api-reference/regions) for endpoint and client-selection details.
See the [Spatius pricing page](https://www.spatius.ai/pricing/) for current public pricing and plan information.
Join the [Spatius Discord](https://discord.com/invite/9HGhZfHZh9). Include the platform, integration, SDK version, region, error output, and a short screen recording when reporting a visual issue.
## Next steps
# Server Error Handling
Source: https://docs.spatius.ai/resources/server-error
Error handling boundaries for the Python and Go Server SDKs.
Server SDK error handling is language-specific. Do not assume that Python's structured fields or error-code enum also exist in Go.
## Python
The Python SDK exposes structured `SessionTokenError` and `AvatarSDKError` instances. Use their stable `code` for program logic and log context fields such as failure phase, HTTP status, server details, correlation IDs, and WebSocket close information when present.
The complete field list and `AvatarSDKErrorCode` values live in the [Python SDK error reference](/sdk-reference/python-sdk/python-sdk#error-handling).
## Go
The Go SDK returns standard Go `error` values from session operations and sends asynchronous failures to the callback configured with `WithOnError`. Log the full error and preserve the operation that produced it.
See the [Go SDK reference](/sdk-reference/go-sdk/go-sdk) for its current public surface. Do not branch on Python `AvatarSDKErrorCode` values in Go code unless the installed Go SDK explicitly exports an equivalent typed error.
## Recovery policy
| Failure area | Recommended handling |
| ---------------------------------------- | ------------------------------------------------------------------------------ |
| Credentials or application configuration | Stop retrying and surface the problem to operators. |
| Session Token creation | Check API Key, App ID, expiry, account state, and selected region. |
| WebSocket connection | Retry transient network failures with bounded exponential backoff. |
| Audio or request validation | Fix the payload before retrying. |
| Unexpected runtime close | Log close details and correlation IDs, then create a new session if retryable. |
| Repeated upstream or server failure | Stop the retry loop and alert operators or contact Spatius support. |
Never log the API Key or complete Session Token while collecting diagnostic context.
# Android SDK Reference
Source: https://docs.spatius.ai/sdk-reference/android-sdk/api-reference
Browse the Android AvatarKit SDK API reference.
## Installation
Use the latest standard release from [Maven Central](https://central.sonatype.com/artifact/ai.spatius/avatarkit). Compatibility-suffixed artifacts expose a reduced API and are not a drop-in replacement.
```kotlin title="build.gradle.kts" theme={null}
dependencies {
implementation("ai.spatius:avatarkit:")
}
```
### AvatarSDK
The core management class of the SDK, responsible for initialization and global configuration.
```kotlin theme={null}
object AvatarSDK
```
##### Properties
The session token used to authenticate avatars with Motion Server.
```kotlin theme={null}
var sessionToken: String
```
The user identifier.
```kotlin theme={null}
var userId: String
```
Returns the version of AvatarKit.
```kotlin theme={null}
val version: String
```
##### Methods
Initialize AvatarKit.
```kotlin theme={null}
fun initialize(
context: Context,
appId: String,
configuration: Configuration
)
```
**Parameters:**
* `context`: Application context
* `appId`: Your application identifier
* `configuration`: The configuration for AvatarKit
Returns a Boolean value that indicates whether AvatarKit supports the current device.
```kotlin theme={null}
suspend fun isDeviceSupported(): Boolean
```
Update the global rendering quality tier.
```kotlin theme={null}
fun setRenderQuality(quality: RenderQuality)
```
Cap render backing-buffer height while preserving view layout size.
```kotlin theme={null}
fun setRenderResolutionCap(enabled: Boolean, maxHeight: Int = 1440)
```
Measures the device's computational performance for avatar rendering.
```kotlin theme={null}
suspend fun deviceScore(): AvatarSDK.DeviceScore
```
The result contains separate `cpuScore` and `gpuScore` values.
### AvatarManager
Avatar resource manager, responsible for downloading, caching, and loading avatar data.
```kotlin theme={null}
object AvatarManager
```
##### Methods
Initialize AvatarManager. Must be called before use.
```kotlin theme={null}
fun initialize(context: Context)
```
**Parameters:**
* `context`: Application context
Loads an avatar by ID.
```kotlin theme={null}
suspend fun load(
id: String,
useCompressedModel: Boolean = false,
onProgress: ((LoadProgress) -> Unit)? = null
): Avatar
```
**Parameters:**
* `id`: The avatar identifier
* `useCompressedModel`: Load the smaller compressed model variant
* `onProgress`: Optional progress callback
**Returns:** The loaded `Avatar` instance.
Clears cached data for a specific avatar.
```kotlin theme={null}
suspend fun clear(id: String)
```
**Parameters:**
* `id`: The avatar identifier to clear
Clears all cached avatar data.
```kotlin theme={null}
suspend fun clearAll()
```
Gets the cache size for a specific avatar.
```kotlin theme={null}
suspend fun getCacheSize(id: String): Long
```
**Parameters:**
* `id`: The avatar identifier
**Returns:** The cache size in bytes.
Gets the total cache size for all avatars.
```kotlin theme={null}
suspend fun getAllCacheSize(): Long
```
**Returns:** The total cache size in bytes.
### AvatarController
Real-time communication controller that handles connections, audio, and motion data.
```kotlin theme={null}
class AvatarController
```
##### Properties
Callback for connection state changes.
```kotlin theme={null}
var onConnectionState: ((ConnectionState) -> Unit)?
```
Callback for conversation state changes.
```kotlin theme={null}
var onConversationState: ((ConversationState) -> Unit)?
```
Callback for error events.
```kotlin theme={null}
var onError: ((AvatarError) -> Unit)?
```
Sets or gets the playback volume (0.0 to 1.0).
```kotlin theme={null}
fun setVolume(volume: Float)
fun getVolume(): Float
```
##### Methods
Starts the avatar driving service connection.
```kotlin theme={null}
fun start()
```
Close connection.
```kotlin theme={null}
fun close()
```
Pause Avatar playback.
```kotlin theme={null}
fun pause()
```
Resume Avatar playback.
```kotlin theme={null}
fun resume()
```
Stops playback and terminates the current conversation.
```kotlin theme={null}
fun interrupt()
```
Sends audio to the avatar driving service.
For audio source and timing guidance, see [Audio](/concepts/audio).
```kotlin theme={null}
suspend fun send(audioData: ByteArray, end: Boolean = false): String
```
**Parameters:**
* `audioData`: The audio data to send
* `end`: Whether this is the end of the audio stream
**Returns:** A conversation ID string.
Provides response audio received from your backend in Backend Mode.
```kotlin theme={null}
suspend fun yieldAudioData(
audioData: ByteArray,
end: Boolean = false
): String
```
**Parameters:**
* `audioData`: The audio data
* `end`: Whether this is the end of the audio stream
**Returns:** A conversation ID string.
Provides motion data payloads received from your backend in Backend Mode.
```kotlin theme={null}
fun yieldFramesData(animations: List, conversationId: String): Boolean
```
**Parameters:**
* `animations`: List of encoded motion data payloads
* `conversationId`: The identifier returned by the matching `yieldAudioData()` call
**Returns:** Whether the payload completed the current conversation.
### AvatarView
3D rendering view that automatically creates and manages AvatarController.
```kotlin theme={null}
class AvatarView : FrameLayout
```
##### Initializers
Creates a new avatar view.
```kotlin theme={null}
constructor(context: Context)
```
##### Properties
The controller for the avatar.
```kotlin theme={null}
val controller: AvatarController
```
Transform for Avatar position and scale within the view.
```kotlin theme={null}
var avatarTransform: Transform
```
##### Methods
Initialize view with avatar.
```kotlin theme={null}
fun init(avatar: Avatar, scope: CoroutineScope)
```
**Parameters:**
* `avatar`: The avatar to display
* `scope`: Coroutine scope, usually obtained via `activity.lifecycleScope`
Pause rendering for this view.
```kotlin theme={null}
fun pauseRendering()
```
Resume rendering for this view.
```kotlin theme={null}
fun resumeRendering()
```
Returns the rendered Avatar bounds when available.
```kotlin theme={null}
fun getBoundingRect(): RectF?
```
Clean up all resources. Should be called when no longer in use.
```kotlin theme={null}
fun dispose()
```
#### Render over an Avatar Background
Download the optional 16:9 background from Spatius Studio and add it as a drawable resource. In Compose, place the background `Image` and `AvatarView` in the same `Box`. `AvatarView` renders on a transparent surface by default, so the background shows through.
```kotlin theme={null}
@Composable
fun AvatarStage(
avatar: Avatar,
scope: CoroutineScope,
) {
Box(
modifier = Modifier
.fillMaxWidth()
.aspectRatio(16f / 9f)
.clipToBounds(),
) {
Image(
painter = painterResource(R.drawable.avatar_background),
contentDescription = null,
contentScale = ContentScale.Crop,
modifier = Modifier.fillMaxSize(),
)
AndroidView(
factory = { context ->
AvatarView(context).also { avatarView ->
avatarView.init(avatar, scope)
}
},
modifier = Modifier.fillMaxSize(),
)
}
}
```
For square or portrait display windows, center this 16:9 stage inside a clipped outer container. See [Avatar Background](/concepts/avatar-background) for the shared cropping rules.
***
### Avatar
Avatar data class containing core avatar information.
```kotlin theme={null}
class Avatar
```
| Property | Type | Description |
| ------------- | --------- | ----------------------------------------- |
| `id` | `String` | The avatar identifier. |
| `isFromCache` | `Boolean` | Whether the avatar was loaded from cache. |
### Configuration
SDK configuration class.
```kotlin theme={null}
data class Configuration
```
##### Initializers
Creates a new configuration with the specified parameters.
```kotlin theme={null}
data class Configuration(
val region: String = "auto",
val audioFormat: AudioFormat = AudioFormat(16000),
val drivingServiceMode: DrivingServiceMode = DrivingServiceMode.DIRECT,
val logLevel: LogLevel = LogLevel.OFF,
val renderQuality: RenderQuality = RenderQuality.ULTRA
)
```
**Parameters:**
* `region`: The region to connect to. Defaults to `"auto"`: the SDK selects the closest serving region at initialization and reuses the cached choice on later launches. Pass `"us-west"`, `"ap-northeast"`, or `"cn-beijing"` to force that region. If automatic selection cannot be reached, the SDK falls back to a default region and continues initializing.
* `audioFormat`: The audio format configuration
* `drivingServiceMode`: The driving service mode
* `logLevel`: The log level
* `renderQuality`: Render quality tier
### AudioFormat
Audio format configuration for AvatarKit.
```kotlin theme={null}
class AudioFormat
```
##### Initializers
Creates a new audio format.
```kotlin theme={null}
class AudioFormat(
sampleRate: Int,
val opusBitrate: Int = 48000,
val inputAudioFormat: AudioCodec = AudioCodec.PCM,
val opusUplinkEnabled: Boolean = true,
)
```
**Parameters:**
* `sampleRate`: The audio sample rate in Hz. Supported sample rates are: 8000, 16000, 22050, 24000, 32000, 44100, 48000. Not used when `inputAudioFormat` is `AudioCodec.OPUS`: Opus always decodes at 48 kHz, so the session runs at 48000.
* `inputAudioFormat`: The codec of the audio you feed into the SDK via `send` or `yieldAudioData`. `AudioCodec.PCM` (default) for raw PCM16, `AudioCodec.OPUS` for Opus. Fixed for the whole session; audio that does not match is reported as `AvatarError.InvalidAudioInput`.
* `opusUplinkEnabled`: Whether the SDK compresses the direct mode uplink to Opus. On by default, which reduces the upload to roughly an eighth at the cost of encoding on the client. Set `false` to keep a raw PCM uplink. Not used in `BACKEND` mode, which has no uplink, or with Opus input, which is already compressed. With PCM input, `sampleRate` must be 8000, 16000, 24000, or 48000; on any other rate the SDK logs a warning and sends raw PCM instead of failing to initialize.
* `opusBitrate`: Target bitrate in bits/sec for the uplink the SDK encodes. Defaults to 48000. Higher means better quality and a larger upload.
### DrivingServiceMode
Driving service modes for AvatarKit.
```kotlin theme={null}
enum class DrivingServiceMode
```
| Case | Description |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DIRECT` | The SDK handles the Motion Server connection directly. |
| `BACKEND` | The host application provides response audio and motion data. |
| `RTC` | The avatar is driven through the companion RTC SDK, `ai.spatius:avatarkit-rtc`. It does not connect to a channel by itself; see [Agora Convo AI Client](/agora-convoai/client). |
### AudioCodec
Codec of the audio the host feeds into the SDK.
```kotlin theme={null}
enum class AudioCodec
```
| Case | Description |
| ------ | --------------------------------------------------------------------------- |
| `PCM` | Raw PCM16, mono. The default. |
| `OPUS` | Opus. Decoded to PCM locally; in Direct Mode, forwarded upstream unchanged. |
### RenderQuality
Render quality tiers for AvatarKit.
```kotlin theme={null}
enum class RenderQuality
```
| Case | Description |
| ---------- | ------------------------------------ |
| `STANDARD` | Lower rendering cost. |
| `HIGH` | Balanced quality and rendering cost. |
| `ULTRA` | Highest quality. Default. |
### FrameStarvationMode
Controls playback behavior when motion data cannot keep up with the audio clock.
```kotlin theme={null}
enum class FrameStarvationMode
```
| Case | Description |
| ------------------- | -------------------------------------------------------------------------------- |
| `AUDIO_INDEPENDENT` | Audio keeps playing while motion data catches up. Default. |
| `STRICT_SYNC` | Audio pauses when motion data runs out and resumes when new motion data arrives. |
### LogLevel
Log levels for AvatarKit.
```kotlin theme={null}
enum class LogLevel
```
| Case | Description |
| --------- | --------------------- |
| `ALL` | Log all messages. |
| `WARNING` | Log warning messages. |
| `ERROR` | Log error messages. |
| `OFF` | Disable logging. |
### ConnectionState
Connection state sealed class.
```kotlin theme={null}
sealed class ConnectionState
```
| Case | Description |
| ------------------- | ---------------------------------------- |
| `Connecting` | The connection is being established. |
| `Connected` | The connection is active. |
| `Disconnected` | The connection has been closed. |
| `Failed(Exception)` | The connection failed with an exception. |
### ConversationState
Avatar conversation state enum.
```kotlin theme={null}
enum class ConversationState
```
| Case | Description |
| --------- | ---------------------------------------- |
| `Idle` | Idle state, showing breathing animation. |
| `Playing` | The avatar is playing audio/animation. |
| `Paused` | Playback is paused. |
### LoadProgress
Load progress sealed class.
```kotlin theme={null}
sealed class LoadProgress
```
| Case | Description |
| -------------------- | -------------------------------- |
| `Downloading(Float)` | Downloading with progress (0-1). |
| `Completed` | Loading completed. |
| `Failed(Throwable)` | Loading failed with error. |
# AvatarKit Flutter SDK Reference
Source: https://docs.spatius.ai/sdk-reference/flutter-sdk/api-reference
AvatarKit Flutter SDK reference for Direct Mode, Backend Mode client feed, state callbacks, and demos.
## Installation
Add the Flutter package from [pub.dev](https://pub.dev/packages/spatius_avatarkit):
```bash theme={null}
flutter pub add spatius_avatarkit
```
The package supports iOS and Android Flutter apps.
## Import
```dart theme={null}
import 'package:spatius_avatarkit/spatius_avatarkit.dart';
```
## Initialize
Initialize AvatarKit once before loading avatars or creating an avatar view.
```dart theme={null}
await AvatarSDK.initialize(
appID: appId,
configuration: Configuration(
audioFormat: const AudioFormat(sampleRate: 16000),
drivingServiceMode: DrivingServiceMode.direct,
logLevel: LogLevel.all,
),
);
await AvatarSDK.setSessionToken(sessionToken);
```
Use `DrivingServiceMode.direct` for [Direct Mode](/direct-mode/client). Use `DrivingServiceMode.backend` when your Flutter app receives response audio and motion data from [Backend Mode Client](/backend-mode/client-sdk).
`initialize` fails fast when `appID` is missing, rather than returning silently. Get an App ID from [app.spatius.ai](https://app.spatius.ai/).
### Configuration
| Field | What it does |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `region` | Defaults to `'auto'`: the SDK selects the closest serving region at initialization and reuses the cached choice on later launches. Pass `'us-west'`, `'ap-northeast'`, or `'cn-beijing'` to force that region. If automatic selection cannot be reached, the SDK falls back to a default region and continues initializing. |
| `audioFormat` | See [AudioFormat](#audioformat) below. |
| `drivingServiceMode` | `direct` or `backend`, as described above. |
| `logLevel` | SDK log verbosity. |
| `renderQuality` | Rendering quality tier. Defaults to `RenderQuality.ultra`. |
### RenderQuality
`RenderQuality.ultra` is the default and highest-quality tier. Use `high` or `standard` only when you intentionally trade visual quality for lower rendering cost.
```dart theme={null}
enum RenderQuality { standard, high, ultra }
await AvatarSDK.setRenderQuality(RenderQuality.ultra);
```
### AudioFormat
```dart theme={null}
const AudioFormat({
int sampleRate = 16000,
AudioCodec inputAudioFormat = AudioCodec.pcm,
bool opusUplinkEnabled = true,
int opusBitrate = 48000,
})
```
| Field | What it does |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sampleRate` | Sample rate in Hz. Supported: 8000, 16000, 22050, 24000, 32000, 44100, 48000. Not used when `inputAudioFormat` is `AudioCodec.opus`: Opus always decodes at 48 kHz, so the session runs at 48000. |
| `inputAudioFormat` | The codec of the audio you feed into the SDK via `send` / `yieldAudioData`. `AudioCodec.pcm` (default) for raw PCM16, `AudioCodec.opus` for Opus. Fixed for the whole session. |
| `opusUplinkEnabled` | Whether the SDK compresses the Direct Mode uplink to Opus. On by default, which reduces the upload to roughly an eighth at the cost of encoding on the client. Set `false` to keep a raw PCM uplink. Not used in Backend Mode, which has no uplink, or with Opus input, which is already compressed. With PCM input, `sampleRate` must be 8000, 16000, 24000, or 48000; on any other rate the SDK logs a warning and sends raw PCM instead of failing to initialize. |
| `opusBitrate` | Target bitrate in bits/sec for the uplink the SDK encodes. Defaults to 48000. Higher means better quality and a larger upload. |
Audio that does not match the declared `inputAudioFormat` is reported through `AvatarController.onError` as `AvatarError.invalidAudioInput`.
## Load an avatar
```dart theme={null}
final avatar = await AvatarManager.shared.load(
id: avatarId,
onProgress: (progress) {
// Update loading UI.
},
);
```
## Render and control playback
The Flutter view creates an `AvatarController` when the platform view is ready. Keep that controller and use it for lifecycle, state, and audio operations.
```dart theme={null}
void onAvatarViewCreated(AvatarController controller) {
controller.onConnectionState = (state, errorMessage) {
// Observe Direct Mode connection state.
};
controller.onConversationState = (state) {
// Observe avatar playback state.
};
controller.onError = (error) {
// Log or surface AvatarError.
};
}
```
### Render over an Avatar Background
Download the optional 16:9 background from Spatius Studio, add it to your Flutter assets, and declare it in `pubspec.yaml`.
```yaml theme={null}
flutter:
assets:
- assets/avatar-background.webp
```
Place the background and `AvatarWidget` in the same `Stack`, with the image first.
```dart theme={null}
AspectRatio(
aspectRatio: 16 / 9,
child: ClipRect(
child: Stack(
fit: StackFit.expand,
children: [
Image.asset(
'assets/avatar-background.webp',
fit: BoxFit.cover,
),
AvatarWidget(
avatar: avatar,
onPlatformViewCreated: onAvatarViewCreated,
),
],
),
),
)
```
For square or portrait display windows, center this 16:9 stage inside a clipped outer widget. See [Avatar Background](/concepts/avatar-background) for the shared cropping rules.
For Direct Mode audio input, start the connection and send PCM chunks:
```dart theme={null}
await controller.start();
controller.send(audioBytes, end: isLastChunk);
```
For audio source and timing guidance, see [Audio](/concepts/audio).
For Backend Mode input, feed the response audio and motion data received from your backend:
```dart theme={null}
final conversationId = await controller.yieldAudioData(audioBytes, end: isLastChunk);
controller.yieldAnimations(framesData, conversationID: conversationId);
```
The per-call `audioFormat` parameter of `yieldAudioData` is deprecated and ignored. The format comes from `Configuration.audioFormat` passed to `initialize`. Drop the argument if you still pass it; it will be removed in a future release. The iOS and Android SDKs have already removed theirs.
## Demos
# Go SDK
Source: https://docs.spatius.ai/sdk-reference/go-sdk/go-sdk
Go Server SDK reference for Backend Mode sessions, audio input, callbacks, LiveKit egress, options, and errors.
The Go SDK creates Spatius avatar sessions from backend services. It requests a short-lived session token from the Console API, opens the ingress WebSocket, sends audio, and returns motion data payloads through callbacks.
## Install
```bash theme={null}
go get github.com/spatius-ai/spatius-sdk-go
```
Repository: [spatius-ai/spatius-sdk-go](https://github.com/spatius-ai/spatius-sdk-go)
## Quick start
```go theme={null}
package main
import (
"context"
"log"
"time"
spatius "github.com/spatius-ai/spatius-sdk-go"
)
func main() {
ctx := context.Background()
session := spatius.NewAvatarSession(
spatius.WithAPIKey("your-api-key"),
spatius.WithAppID("your-app-id"),
spatius.WithAvatarID("your-avatar-id"),
spatius.WithExpireAt(time.Now().Add(5*time.Minute).UTC()),
spatius.WithTransportFrames(func(data []byte, last bool) {
// Handle motion data payloads.
}),
spatius.WithOnError(func(err error) {
// Handle async session errors.
}),
spatius.WithOnClose(func() {
// Handle connection close.
}),
)
if err := session.Init(ctx); err != nil {
log.Fatal(err)
}
defer session.Close()
connectionID, err := session.Start(ctx)
if err != nil {
log.Fatal(err)
}
log.Printf("connection id: %s", connectionID)
audioBytes := []byte{} // Replace with mono PCM audio bytes.
reqID, err := session.SendAudio(audioBytes, true)
if err != nil {
log.Fatal(err)
}
log.Printf("request id: %s", reqID)
}
```
## Region configuration
The SDK defaults to `us-west` when no region is provided, or when `WithRegion` receives an empty or whitespace-only value. See [Regions](/api-reference/regions) for the current region list. To set a region:
```go theme={null}
session := spatius.NewAvatarSession(
spatius.WithRegion("us-west"),
)
```
## Session lifecycle
1. Create a session with `NewAvatarSession`.
2. Call `Init(ctx)` to exchange the API key and expiry time for a session token.
3. Call `Start(ctx)` to open the ingress WebSocket and configure the avatar session.
4. Call `SendAudio(audio, end)` for each audio chunk.
5. Call `Interrupt()` when playback should stop.
6. Call `Close()` when the session is done.
`SendAudio` returns the request ID associated with the audio. The callback passed to `WithTransportFrames` receives motion data payloads and a `last` flag.
For audio source and timing guidance, see [Audio](/concepts/audio).
## Audio format
PCM 16-bit little-endian is the default input format:
```go theme={null}
session := spatius.NewAvatarSession(
spatius.WithSampleRate(24000),
spatius.WithAudioFormat(spatius.AudioFormatPCMS16LE),
)
```
The SDK also supports Ogg Opus sessions:
```go theme={null}
session := spatius.NewAvatarSession(
spatius.WithAudioFormat(spatius.AudioFormatOggOpus),
spatius.WithSampleRate(24000),
spatius.WithBitrate(32000),
spatius.WithOggOpusEncoder(nil),
)
```
Passing `nil` to `WithOggOpusEncoder` enables the default encoder config.
## Authentication mode
By default, WebSocket authentication is sent in headers:
```go theme={null}
session := spatius.NewAvatarSession(
spatius.WithUseQueryAuth(false),
)
```
For web-style query authentication, enable:
```go theme={null}
session := spatius.NewAvatarSession(
spatius.WithUseQueryAuth(true),
)
```
## LiveKit egress
Server SDK egress is a low-level capability, not an additional integration path. Use [LiveKit Agents Integration](/livekit-agents/overview) for the supported LiveKit Agents workflow.
Use LiveKit egress when Spatius should publish avatar output directly into a LiveKit room:
```go theme={null}
session := spatius.NewAvatarSession(
spatius.WithLiveKitEgress(&spatius.LiveKitEgressConfig{
URL: "wss://your-livekit-server.com",
APIToken: "livekit-access-token",
RoomName: "room-name",
PublisherID: "avatar-publisher",
ExtraAttributes: map[string]string{
"role": "avatar",
},
IdleTimeout: 60,
}),
)
```
Prefer `APIToken` for new integrations. `APIKey` and `APISecret` remain available for older setups.
## Options
Core options:
* `WithAPIKey(apiKey string)` - Console API key.
* `WithAppID(appID string)` - application ID.
* `WithAvatarID(avatarID string)` - avatar ID.
* `WithExpireAt(expireAt time.Time)` - session token expiry.
* `WithRegion(region string)` - region used to select the Spatius region. Empty or whitespace-only values fall back to `us-west`.
* `WithConsoleEndpointURL(endpointURL string)` - explicit Console API URL.
* `WithIngressEndpointURL(endpointURL string)` - explicit ingress WebSocket URL.
Audio options:
* `WithSampleRate(sampleRate int)` - input sample rate in Hz.
* `WithBitrate(bitrate int)` - target bitrate for encoded sessions.
* `WithAudioFormat(audioFormat AudioFormat)` - `AudioFormatPCMS16LE` or `AudioFormatOggOpus`.
* `WithOggOpusEncoder(config *OggOpusEncoderConfig)` - enable client-side PCM to Ogg Opus encoding.
Callbacks:
* `WithTransportFrames(func([]byte, bool))` - receives motion data payloads.
* `WithOnEncodedAudio(func(string, []byte))` - receives internally encoded audio by request ID.
* `WithOnError(func(error))` - receives async session errors.
* `WithOnClose(func())` - runs when the session closes.
Egress options:
* `WithLiveKitEgress(config *LiveKitEgressConfig)` - stream output to LiveKit.
* `WithAgoraEgress(config *AgoraEgressConfig)` - stream output to Agora.
## Changelog
See [GitHub releases](https://github.com/spatius-ai/spatius-sdk-go/releases).
# iOS SDK Reference
Source: https://docs.spatius.ai/sdk-reference/ios-sdk/api-reference
Browse the iOS AvatarKit SDK API reference.
## Installation
In Xcode, open **File > Add Package Dependencies**, enter `https://github.com/spatius-ai/avatarkit-ios-release.git`, and select the latest stable release.
```ruby theme={null}
pod 'SpatiusAvatarKit'
```
The pod is named `SpatiusAvatarKit`, but the module name is unchanged: import it as `import AvatarKit`.
Download the latest [AvatarKit.xcframework](https://github.com/spatius-ai/avatarkit-ios-release/releases/latest) from the releases page, unzip, and drag it into your Xcode project. Set **Embed & Sign** in your target's **Frameworks, Libraries, and Embedded Content**.
### AvatarSDK
Main initialization and configuration interface for AvatarKit.
```swift theme={null}
@MainActor enum AvatarSDK
```
##### Type Properties
The app identifier.
```swift theme={null}
static var appID: String { get }
```
The app configuration.
```swift theme={null}
static var configuration: Configuration { get }
```
The session token used to authenticate avatars with Motion Server.
```swift theme={null}
static var sessionToken: String
```
The user identifier provided by the developer to identify the user in the AvatarKit Log Service.
```swift theme={null}
static var userID: String
```
Returns the version of AvatarKit.
```swift theme={null}
static var version: String { get }
```
##### Type Methods
Initialize AvatarKit.
```swift theme={null}
static func initialize(appID: String, configuration: Configuration)
```
**Parameters:**
* `appID`: The app identifier
* `configuration`: The configuration for AvatarKit
Update the global rendering quality tier.
```swift theme={null}
static func setRenderQuality(_ quality: RenderQuality)
```
Cap render backing-buffer height while preserving the view's CSS/layout size.
```swift theme={null}
static func setRenderResolutionCap(enabled: Bool, maxHeight: Int = 1440)
```
Check whether the current device can run AvatarKit.
```swift theme={null}
static func isDeviceSupported() async -> Bool
```
Measures the device's computational performance for avatar rendering.
```swift theme={null}
static func deviceScore() async -> AvatarSDK.DeviceScore
```
### AvatarManager
Manage avatar asset loading, caching, and retrieval.
```swift theme={null}
final class AvatarManager
```
##### Type Properties
The shared avatar manager instance.
```swift theme={null}
static let shared: AvatarManager
```
##### Instance Methods
Loads an avatar by ID.
```swift theme={null}
func load(
id: String,
useCompressedModel: Bool = false,
onProgress: ProgressHandler? = nil
) async throws -> Avatar
```
**Parameters:**
* `id`: The avatar identifier
* `useCompressedModel`: Load the smaller compressed model variant
* `onProgress`: Optional progress callback
**Returns:** The loaded `Avatar` instance.
Cancels the loading of avatar by ID.
```swift theme={null}
func cancelLoading(id: String) async
```
**Parameters:**
* `id`: The avatar identifier
Cancels all loading of avatars.
```swift theme={null}
func cancelAllLoading() async
```
Retrieves a cached avatar by ID.
```swift theme={null}
func retrieve(id: String) -> Avatar?
```
**Parameters:**
* `id`: The avatar identifier
**Returns:** The cached `Avatar` if available, otherwise `nil`.
Derives an avatar from a local asset path.
```swift theme={null}
func derive(assetPath: String) throws -> Avatar
```
**Parameters:**
* `assetPath`: The path to the avatar asset
**Returns:** The derived `Avatar` instance.
Clears cached data for a specific avatar.
```swift theme={null}
func clear(id: String) throws
```
**Parameters:**
* `id`: The avatar identifier to clear
Clears all cached avatar data.
```swift theme={null}
func clearAll() throws
```
Clears least recently used avatars, keeping the specified count.
```swift theme={null}
func clearLRU(keepCount: Int) throws
```
**Parameters:**
* `keepCount`: Number of most recently used avatars to keep
Gets the cache size for a specific avatar.
```swift theme={null}
func getCacheSize(id: String) throws -> Int
```
**Parameters:**
* `id`: The avatar identifier
**Returns:** The cache size in bytes.
Gets the total cache size for all avatars.
```swift theme={null}
func getAllCacheSize() throws -> Int
```
**Returns:** The total cache size in bytes.
### AvatarController
The main controller for managing avatar driving service connections and interactions.
```swift theme={null}
@MainActor final class AvatarController
```
##### Instance Properties
Callback for connection state changes.
```swift theme={null}
var onConnectionState: ((ConnectionState) -> Void)?
```
Callback for conversation state changes.
```swift theme={null}
var onConversationState: ((ConversationState) -> Void)?
```
Callback for error events.
```swift theme={null}
var onError: ((AvatarError) -> Void)?
```
Starts the avatar driving service connection.
```swift theme={null}
func start()
```
Closes the avatar driving service.
```swift theme={null}
func close()
```
Pauses Avatar playback.
```swift theme={null}
func pause()
```
Resumes Avatar playback.
```swift theme={null}
func resume()
```
Stops playback and terminates the current conversation.
```swift theme={null}
func interrupt()
```
Sends audio to the avatar driving service.
For audio source and timing guidance, see [Audio](/concepts/audio).
```swift theme={null}
func send(_ data: Data, end: Bool) -> String
```
**Parameters:**
* `data`: The audio data to send
* `end`: Whether this is the end of the audio stream
**Returns:** A conversation ID string.
Provides response audio received from your backend in Backend Mode.
```swift theme={null}
func yieldAudioData(_ data: Data, end: Bool) -> String
```
**Parameters:**
* `data`: The audio data
* `end`: Whether this is the end of the audio stream
**Returns:** A conversation ID string.
Provides motion data payloads received from your backend in Backend Mode.
```swift theme={null}
func yieldFramesData(_ frames: [Data], conversationID: String) -> Bool
```
**Parameters:**
* `frames`: Array of encoded motion data payloads
* `conversationID`: The identifier returned by the matching `yieldAudioData()` call
**Returns:** Whether the payload completed the current conversation.
The point count of the current avatar.
```swift theme={null}
var pointCount: Int { get }
```
The volume of playback.
```swift theme={null}
var volume: Float
```
### AvatarView
A UIView subclass for rendering avatars.
```swift theme={null}
@MainActor final class AvatarView: UIView
```
##### Initializers
Creates a new avatar view with the specified avatar.
```swift theme={null}
init(avatar: Avatar)
```
**Parameters:**
* `avatar`: The avatar to display
##### Instance Properties
The controller for the avatar.
```swift theme={null}
var controller: AvatarController { get }
```
Callback when the Avatar is first rendered.
```swift theme={null}
var onFirstRendering: (() -> Void)?
```
Transform for avatar content position and scale within the view.
```swift theme={null}
var avatarTransform: Transform
```
##### Instance Methods
Pauses Avatar rendering.
```swift theme={null}
func pauseRendering()
```
Resumes Avatar rendering.
```swift theme={null}
func resumeRendering()
```
Returns whether rendering is enabled.
```swift theme={null}
func isRenderingEnabled() -> Bool
```
Returns the rendered Avatar bounds when available.
```swift theme={null}
func getBoundingRect() -> CGRect?
```
#### Render over an Avatar Background
Download the optional 16:9 background from Spatius Studio and add it to the Asset Catalog. Place the image and `AvatarView` in the same `ZStack`, with the image first. `AvatarView` is transparent by default, so the background shows through.
```swift theme={null}
struct AvatarLayer: UIViewRepresentable {
let avatar: Avatar
func makeUIView(context: Context) -> AvatarView {
AvatarView(avatar: avatar)
}
func updateUIView(_ avatarView: AvatarView, context: Context) {}
}
struct AvatarStage: View {
let avatar: Avatar
var body: some View {
ZStack {
Image("AvatarBackground")
.resizable()
.aspectRatio(contentMode: .fill)
AvatarLayer(avatar: avatar)
}
.aspectRatio(16 / 9, contentMode: .fit)
.clipped()
}
}
```
For square or portrait display windows, center this 16:9 stage inside a clipped outer view. See [Avatar Background](/concepts/avatar-background) for the shared cropping rules.
***
### Avatar
Represents an avatar instance.
```swift theme={null}
struct Avatar
```
| Instance Property | Type | Description |
| ----------------- | -------- | --------------------------------------------- |
| `id` | `String` | The avatar identifier. |
| `isFromCache` | `Bool` | Whether the instance of avatar is from cache. |
### Configuration
Configuration for AvatarKit.
```swift theme={null}
struct Configuration
```
##### Initializers
Creates a new configuration with the specified parameters.
```swift theme={null}
init(
region: String = "auto",
audioFormat: AudioFormat = .init(),
drivingServiceMode: DrivingServiceMode = .direct,
logLevel: LogLevel = .off,
renderQuality: RenderQuality = .ultra
)
```
**Parameters:**
* `region`: The region to connect to. Defaults to `"auto"`: the SDK selects the closest serving region at initialization and reuses the cached choice on later launches. Pass `"us-west"`, `"ap-northeast"`, or `"cn-beijing"` to force that region. If automatic selection cannot be reached, the SDK falls back to a default region and continues initializing.
* `audioFormat`: The audio format configuration
* `drivingServiceMode`: The driving service mode
* `logLevel`: The log level
* `renderQuality`: Render quality tier
| Instance Property | Type | Description |
| -------------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `region` | `String` | The region to connect to. Defaults to `"auto"`: the SDK selects the closest serving region at initialization and reuses the cached choice on later launches. Pass `"us-west"`, `"ap-northeast"`, or `"cn-beijing"` to force that region. If automatic selection cannot be reached, the SDK falls back to a default region and continues initializing. |
| `audioFormat` | `AudioFormat` | Audio format for AvatarKit. |
| `drivingServiceMode` | `DrivingServiceMode` | Driving service mode for AvatarKit. |
| `logLevel` | `LogLevel` | Log level for AvatarKit. |
| `renderQuality` | `RenderQuality` | Render quality tier for AvatarKit. |
### AudioFormat
Audio format configuration for AvatarKit.
```swift theme={null}
struct AudioFormat
```
##### Initializers
Creates a new audio format.
```swift theme={null}
init(
sampleRate: Int = 16000,
inputAudioFormat: AudioCodec = .pcm,
opusBitrate: Int = 48000,
opusUplinkEnabled: Bool = true
)
```
**Parameters:**
* `sampleRate`: The audio sample rate in Hz. Defaults to 16000. Supported: 8000, 16000, 22050, 24000, 32000, 44100, 48000. Not used when `inputAudioFormat` is `.opus`: Opus always decodes at 48 kHz, so the session runs at 48000.
* `inputAudioFormat`: The codec of the audio you feed into the SDK via `send` or `yieldAudioData`. `.pcm` (default) for raw PCM16, `.opus` for Opus. Fixed for the whole session; audio that does not match is reported as `AvatarError.invalidAudioInput`.
* `opusBitrate`: Target bitrate in bits/sec for the uplink the SDK encodes. Defaults to 48000. Higher means better quality and a larger upload.
* `opusUplinkEnabled`: Whether the SDK compresses the direct mode uplink to Opus. On by default, which reduces the upload to roughly an eighth at the cost of encoding on the client. Set `false` to keep a raw PCM uplink. Not used in `.backend` mode, which has no uplink, or with Opus input, which is already compressed. With PCM input, `sampleRate` must be 8000, 16000, 24000, or 48000; on any other rate the SDK logs a warning and sends raw PCM instead of failing to initialize.
| Instance Property | Type | Description |
| ------------------- | ------------ | ------------------------------------------------------------------------------ |
| `sampleRate` | `Int` | The audio sample rate in Hz. Reports 48000 when `inputAudioFormat` is `.opus`. |
| `channelCount` | `Int` | The number of audio channels. Fixed to 1 for mono. |
| `inputAudioFormat` | `AudioCodec` | The codec the host feeds into the SDK. |
| `opusBitrate` | `Int` | Target bitrate for the SDK-encoded uplink. |
| `opusUplinkEnabled` | `Bool` | Whether the direct-mode uplink is Opus-compressed. |
### DrivingServiceMode
Driving service modes for AvatarKit.
```swift theme={null}
enum DrivingServiceMode
```
| Case | Description |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `direct` | The SDK handles the Motion Server connection directly. |
| `backend` | The host application provides response audio and motion data. |
| `rtc` | The avatar is driven through the companion RTC SDK, `AvatarKitRTC`. It does not connect to a channel by itself; see [Agora Convo AI Client](/agora-convoai/client). |
### AudioCodec
Codec of the audio the host feeds into the SDK.
```swift theme={null}
enum AudioCodec
```
| Case | Description |
| ------ | --------------------------------------------------------------------------- |
| `pcm` | Raw PCM16, mono. The default. |
| `opus` | Opus. Decoded to PCM locally; in Direct Mode, forwarded upstream unchanged. |
### RenderQuality
Render quality tiers for AvatarKit.
```swift theme={null}
enum RenderQuality
```
| Case | Description |
| ---------- | ------------------------------------ |
| `standard` | Lower rendering cost. |
| `high` | Balanced quality and rendering cost. |
| `ultra` | Highest quality. Default. |
### FrameStarvationMode
Controls playback behavior when motion data cannot keep up with the audio clock.
```swift theme={null}
enum FrameStarvationMode
```
| Case | Description |
| ------------------ | -------------------------------------------------------------------------------- |
| `audioIndependent` | Audio keeps playing while motion data catches up. Default. |
| `strictSync` | Audio pauses when motion data runs out and resumes when new motion data arrives. |
### LogLevel
Log levels for AvatarKit.
```swift theme={null}
enum LogLevel
```
| Case | Description |
| --------- | ----------------------------- |
| `all` | Log all messages. |
| `warning` | Log warnings and errors only. |
| `error` | Log errors only. |
| `off` | Disable logging. |
### ConnectionState
Connection states for AvatarKit.
```swift theme={null}
enum ConnectionState
```
| Case | Description |
| --------------- | ------------------------------------ |
| `connecting` | The connection is being established. |
| `connected` | The connection is active. |
| `disconnected` | The connection has been closed. |
| `failed(Error)` | The connection failed with an error. |
### ConversationState
Conversation states for AvatarKit.
```swift theme={null}
enum ConversationState
```
| Case | Description |
| --------- | -------------------------------------- |
| `idle` | No active conversation. |
| `playing` | The avatar is playing audio/animation. |
| `paused` | The conversation is paused. |
### Transform
Transform for avatar content rendering.
```swift theme={null}
struct Transform
```
##### Initializers
Creates a new transform with the specified translation and scale.
```swift theme={null}
init(x: Float, y: Float, scale: Float)
```
**Parameters:**
* `x`: The x-axis translation
* `y`: The y-axis translation
* `scale`: The scale factor
| Type Property | Type | Description |
| ------------- | ----------- | ----------------------- |
| `identity` | `Transform` | The identity transform. |
| Instance Property | Type | Description |
| ----------------- | ------- | ----------------------- |
| `x` | `Float` | The x-axis translation. |
| `y` | `Float` | The y-axis translation. |
| `scale` | `Float` | The scale factor. |
### AvatarError
Avatar errors for AvatarKit.
```swift theme={null}
enum AvatarError: LocalizedError
```
| Case | Description |
| ------------------------------ | --------------------------------- |
| `appIDUnrecognized` | The app ID is not recognized. |
| `avatarIDUnrecognized` | The avatar ID is not recognized. |
| `avatarAssetMissing` | The avatar asset is missing. |
| `failedToDownloadAvatarAssets` | Failed to download avatar assets. |
| `failedToFetchAvatarMetadata` | Failed to fetch avatar metadata. |
| `sessionTokenExpired` | The session token has expired. |
| `sessionTokenInvalid` | The session token is invalid. |
| Instance Property | Type | Description |
| ------------------ | --------- | ------------------------------------- |
| `errorDescription` | `String?` | A localized description of the error. |
| `failureReason` | `String?` | The reason for the failure. |
# Python SDK
Source: https://docs.spatius.ai/sdk-reference/python-sdk/python-sdk
Python Server SDK reference for Backend Mode sessions, audio input, motion data callbacks, egress modes, and errors.
The Spatius Python SDK creates server-side avatar sessions over WebSocket. It handles session-token creation, WebSocket authentication, audio upload, motion data payload callbacks, and optional egress to LiveKit or Agora.
## Repository
The Python SDK is available on GitHub: [spatius-ai/spatius-sdk-python](https://github.com/spatius-ai/spatius-sdk-python) and [PyPI](https://pypi.org/project/spatius/).
```bash theme={null}
pip install spatius
```
Install the optional Ogg Opus encoder support when you want the SDK to encode raw PCM before sending:
```bash theme={null}
pip install "spatius[opus]"
```
The optional encoder uses `opuslib`, which requires a working `libopus` runtime on the host system.
## Requirements
* `api_key`
* `app_id`
* `avatar_id`
* `expire_at`
* audio bytes in a supported input format
The SDK defaults to `auto`: bootstrap selects a region and falls back to `us-west` when automatic selection is unavailable. See [Regions](/api-reference/regions).
## Telemetry
The SDK exports OpenTelemetry metrics and traces by default; it does not export logs. Telemetry resources include the App ID and resolved region. Disable export before creating a session if your application does not want SDK telemetry:
```python theme={null}
from spatius import configure_telemetry
configure_telemetry("")
```
See the [SDK repository](https://github.com/spatius-ai/spatius-sdk-python#telemetry) for endpoint configuration and shutdown behavior.
## Region configuration
Passing a region is enough for normal production use:
```python theme={null}
session = new_avatar_session(
api_key="your-api-key",
app_id="your-app-id",
avatar_id="your-avatar-id",
region="us-west",
expire_at=expire_at,
)
```
## Quick Start
```python theme={null}
import asyncio
from datetime import datetime, timedelta, timezone
from spatius import new_avatar_session
async def main():
session = new_avatar_session(
api_key="your-api-key",
app_id="your-app-id",
avatar_id="your-avatar-id",
expire_at=datetime.now(timezone.utc) + timedelta(minutes=5),
transport_frames=lambda payload, last: print(
f"Received payload: {len(payload)} bytes, last={last}"
),
on_error=lambda err: print(f"Session error: {err}"),
on_close=lambda: print("Session closed"),
)
await session.init()
connection_id = await session.start()
print(f"Connected: {connection_id}")
audio_data = b"..." # mono PCM s16le audio bytes
request_id = await session.send_audio(audio_data, end=True)
print(f"Sent audio request: {request_id}")
await asyncio.sleep(10)
await session.close()
if __name__ == "__main__":
asyncio.run(main())
```
## Session Configuration
Use `new_avatar_session()` to configure and create a session:
```python theme={null}
from datetime import datetime, timedelta, timezone
from spatius import AudioFormat, new_avatar_session
session = new_avatar_session(
avatar_id="avatar-123",
api_key="your-api-key",
app_id="your-app-id",
use_query_auth=False,
expire_at=datetime.now(timezone.utc) + timedelta(minutes=5),
region="us-west",
sample_rate=16000,
audio_format=AudioFormat.PCM_S16LE,
transport_frames=on_frame_received,
on_error=on_error,
on_close=on_close,
)
```
`use_query_auth=False` sends WebSocket credentials in headers. Set `use_query_auth=True` for web-style auth that sends `appId` and `sessionKey` in the WebSocket query string.
## Session Lifecycle
```python theme={null}
# 1. Initialize and request a session token
await session.init()
# 2. Start the WebSocket connection
connection_id = await session.start()
# 3. Send audio data
request_id = await session.send_audio(audio_bytes, end=True)
# 4. Receive motion data payloads through transport_frames
# 5. Close the session
await session.close()
```
For audio source and timing guidance, see [Audio](/concepts/audio).
## Audio Format
The SDK supports two session-level input formats:
* `AudioFormat.PCM_S16LE` - mono 16-bit PCM bytes
* `AudioFormat.OGG_OPUS` - one continuous Ogg Opus stream per request ID
### PCM Input
* Sample rate: one of `8000`, `16000`, `22050`, `24000`, `32000`, `44100`, `48000`
* Channels: `1` (mono)
* Bit depth: `16-bit`
* Format: raw PCM bytes
```python theme={null}
from spatius import AudioFormat, new_avatar_session
session = new_avatar_session(
...,
sample_rate=16000,
audio_format=AudioFormat.PCM_S16LE,
)
with open("audio.pcm", "rb") as f:
audio_data = f.read()
await session.send_audio(audio_data, end=True)
```
### Ogg Opus Input
* Sample rate: one of `8000`, `12000`, `16000`, `24000`, `48000`
* Channels: `1` (mono)
* Format: Ogg Opus pages/chunks
* Request contract: each request ID must carry one continuous Ogg Opus stream across one or more `send_audio()` calls, and the final chunk must use `end=True`
```python theme={null}
from spatius import AudioFormat, new_avatar_session
session = new_avatar_session(
...,
sample_rate=24000,
bitrate=32000,
audio_format=AudioFormat.OGG_OPUS,
)
with open("audio.ogg", "rb") as f:
while chunk := f.read(4096):
await session.send_audio(chunk, end=False)
await session.send_audio(b"", end=True)
```
### Built-In PCM to Ogg Opus Encoder
If you want the session to negotiate `AudioFormat.OGG_OPUS` but still provide raw PCM bytes to `send_audio()`, enable the optional internal encoder.
```python theme={null}
from spatius import AudioFormat, OggOpusEncoderConfig, new_avatar_session
encoded_outputs = []
session = new_avatar_session(
...,
sample_rate=24000,
bitrate=32000,
audio_format=AudioFormat.OGG_OPUS,
ogg_opus_encoder=OggOpusEncoderConfig(frame_duration_ms=20),
on_encoded_audio=lambda req_id, payload: encoded_outputs.append((req_id, payload)),
)
with open("audio_24000.pcm", "rb") as f:
pcm_audio = f.read()
await session.send_audio(pcm_audio, end=True)
```
Notes:
* The internal encoder is optional; if you do not install `spatius[opus]`, keep using PCM or provide pre-encoded Ogg Opus bytes yourself.
* `on_encoded_audio` fires when internal encoding completes for a request and receives `(req_id, encoded_audio_bytes)`.
* If `audio_format=AudioFormat.OGG_OPUS` and `ogg_opus_encoder` is unset, `send_audio()` forwards your pre-encoded Ogg Opus bytes unchanged.
## LiveKit Egress Mode
LiveKit and Agora egress are low-level Server SDK capabilities. They are not additional integration paths; LiveKit Agents and Agora Convo AI remain the supported packaged integrations.
When configured with `livekit_egress`, audio and motion data are streamed to a LiveKit room instead of being returned through the WebSocket connection.
```python theme={null}
from datetime import datetime, timedelta, timezone
from spatius import LiveKitEgressConfig, new_avatar_session
session = new_avatar_session(
avatar_id="avatar-123",
api_key="your-api-key",
app_id="your-app-id",
region="us-west",
expire_at=datetime.now(timezone.utc) + timedelta(minutes=5),
livekit_egress=LiveKitEgressConfig(
url="wss://livekit.example.com",
api_token="livekit-token",
room_name="my-room",
publisher_id="avatar-publisher",
),
)
```
`api_key` and `api_secret` remain supported for backward compatibility, but they are deprecated. Prefer `api_token` for new integrations.
When LiveKit egress is enabled:
* the server streams output to the specified LiveKit room
* the `transport_frames` callback is not invoked
* audio and motion data are published under the configured publisher ID
## Agora Egress Mode
When configured with `agora_egress`, audio and motion data are streamed to an Agora channel instead of being returned through the WebSocket connection.
```python theme={null}
from datetime import datetime, timedelta, timezone
from spatius import AgoraEgressConfig, new_avatar_session
session = new_avatar_session(
avatar_id="avatar-123",
api_key="your-api-key",
app_id="your-app-id",
region="us-west",
expire_at=datetime.now(timezone.utc) + timedelta(minutes=5),
agora_egress=AgoraEgressConfig(
channel_name="my-channel",
token="agora-token",
uid=0,
publisher_id="avatar-publisher",
),
)
```
## Interrupt
The `interrupt()` method sends an interrupt signal to stop current audio processing. This is only available when using egress mode.
```python theme={null}
request_id = await session.send_audio(audio_data, end=True)
# Later, interrupt the most recent in-flight request.
interrupted_id = await session.interrupt()
print(f"Interrupted request: {interrupted_id}")
```
The interrupt uses the most recent request ID, even after `end=True` has been sent.
## Callbacks
### `transport_frames`
Receives motion data payloads from the server:
```python theme={null}
def on_frame_received(payload: bytes, is_last: bool):
print(f"Received payload: {len(payload)} bytes")
if is_last:
print("This is the last payload")
```
### `on_error`
Handles structured SDK errors from the session:
```python theme={null}
from spatius import AvatarSDKError
def on_error(error: Exception):
print(f"Session error: {error}")
if isinstance(error, AvatarSDKError):
print(" code:", error.code.value)
print(" phase:", error.phase)
print(" http_status:", error.http_status)
print(" server_code:", error.server_code)
print(" server_detail:", error.server_detail)
```
The SDK reports structured `AvatarSDKError` instances for token creation failures, WebSocket upgrade rejections, handshake failures, runtime `ServerError` messages, and unexpected connection drops.
### `on_close`
Called when the session closes:
```python theme={null}
def on_close():
print("Session has been closed")
```
## Error Handling
Use `SessionTokenError` for token creation failures and `AvatarSDKError` for all other structured SDK errors:
```python theme={null}
from spatius import AvatarSDKError, SessionTokenError
try:
await session.init()
await session.start()
except SessionTokenError as error:
print("token failed", error.code.value, error.server_detail)
except AvatarSDKError as error:
print("sdk error", error.code.value, error.phase, error.server_detail)
```
`AvatarSDKError` and `SessionTokenError` expose these fields:
* `code` - stable SDK error code
* `message` - human-readable message
* `phase` - failure phase such as `session_token`, `websocket_connect`, `websocket_handshake`, `websocket_runtime`, or `websocket_send`
* `http_status` - HTTP status for token or WebSocket upgrade failures
* `server_code` - server-provided error code, including runtime protobuf `ServerError.code`
* `server_title` / `server_detail` - parsed server error details when available
* `connection_id` / `req_id` - server correlation identifiers when available
* `raw_body` - raw HTTP rejection body for token or WebSocket upgrade failures
* `close_code` / `close_reason` - WebSocket close details for unexpected disconnects
### Common `AvatarSDKErrorCode` Values
* `sessionTokenExpired` - session token expired or unauthorized
* `sessionTokenInvalid` - invalid or empty session token
* `appIDUnrecognized` - App ID is not recognized by the server
* `appIDMismatch` - session token belongs to a different app
* `avatarNotFound` - avatar does not exist
* `billingRequired` - session denied by billing checks
* `creditsExhausted` - runtime or connect-time credits exhausted
* `sessionDurationExceeded` - billing-enforced session timeout reached
* `unsupportedSampleRate` - handshake rejected unsupported audio sample rate
* `invalidEgressConfig` - egress config is invalid
* `egressUnavailable` - egress service is unavailable or not configured
* `idleTimeout` - server closed the session after input inactivity
* `upstreamError` - internal upstream service failed
* `protocolError` - invalid protobuf or unexpected message sequence
* `connectionFailed` - transport-level connection failure
* `connectionClosed` - unexpected WebSocket close
* `serverError` - server-side failure that did not match a more specific mapping
* `invalidRequest` - other client-side request validation errors
* `unknown` - fallback when the SDK cannot classify the failure
For server-side recovery guidance, see [Server Error Handling](/resources/server-error).
## SDK Reference
This page includes usage guidance. The SDK source also keeps developer-facing docstrings on public classes and functions so generated API reference pages can stay close to the code.
### `new_avatar_session(...)`
Creates an `AvatarSession` from typed keyword parameters. It is the primary public factory.
### `AvatarSession`
Main class for managing avatar sessions.
#### Methods
* `async init()` - initialize the session and obtain a token
* `async start() -> str` - start the WebSocket connection and return a `connection_id`
* `async send_audio(audio: bytes, end: bool = False) -> str` - send audio data and return a request ID
* `async interrupt() -> str` - interrupt current audio processing in egress mode
* `async close()` - close the session and clean up resources
* `config -> SessionConfig` - current session configuration
### `SessionConfig`
Configuration dataclass for avatar sessions.
#### Fields
* `avatar_id: str` - avatar identifier
* `api_key: str` - API key for authentication
* `app_id: str` - application identifier
* `use_query_auth: bool` - send WebSocket auth in query params instead of headers
* `expire_at: datetime` - session expiration time
* `sample_rate: int` - audio sample rate, default `16000`
* `bitrate: int` - audio bitrate, default `0`
* `audio_format: AudioFormat` - negotiated input format
* `ogg_opus_encoder: Optional[OggOpusEncoderConfig]` - optional PCM to Ogg Opus encoder config
* `on_encoded_audio: Optional[Callable[[str, bytes], None]]` - callback for encoded Ogg output
* `transport_frames: Callable[[bytes, bool], None]` - frame callback
* `on_error: Callable[[Exception], None]` - error callback
* `on_close: Callable[[], None]` - close callback
* `region: str` - Spatius region. Defaults to `auto`; see [Regions](/api-reference/regions).
* `console_endpoint_url: str` - optional explicit Console API URL
* `ingress_endpoint_url: str` - optional explicit ingress WebSocket URL
* `livekit_egress: Optional[LiveKitEgressConfig]` - LiveKit egress configuration
* `agora_egress: Optional[AgoraEgressConfig]` - Agora egress configuration
### `LiveKitEgressConfig`
Configuration for streaming to a LiveKit room.
#### Fields
* `url: str` - LiveKit server URL, for example `wss://livekit.example.com`
* `api_key: str` - deprecated LiveKit API key
* `api_secret: str` - deprecated LiveKit API secret
* `api_token: str` - preferred pre-generated LiveKit access token
* `room_name: str` - room name to join
* `publisher_id: str` - publisher identity in the room
* `extra_attributes: dict[str, str]` - extra participant attributes
* `idle_timeout: int` - idle timeout in seconds, `0` uses server defaults
### `AgoraEgressConfig`
Configuration for streaming to an Agora channel.
#### Fields
* `channel_name: str` - Agora channel name
* `token: str` - Agora token
* `uid: int` - publisher UID; `0` lets Agora assign one
* `publisher_id: str` - publisher identity/name
### Utility Functions
* `generate_log_id() -> str` - generate a unique log ID in the format `YYYYMMDDHHMMSS_`
### Exceptions
* `AvatarSDKError` - structured SDK error with stable code and context fields
* `SessionTokenError` - subclass of `AvatarSDKError` raised when session token creation fails
## Changelog
[spatius-sdk-python releases](https://github.com/spatius-ai/spatius-sdk-python/releases)
# AvatarKit Web SDK Reference
Source: https://docs.spatius.ai/sdk-reference/web-sdk/reference
Complete API reference for @spatius/avatarkit (Web).
## Quick Reference
| Class | Purpose | Key methods |
| ------------------ | ------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `AvatarSDK` | SDK initialization and global configuration | `initialize()`, `setSessionToken()`, `setRenderQuality()`, `deviceScore()` |
| `AvatarManager` | Avatar asset loading and caching | `load()`, `cancelLoad()`, `retrieve()`, `clear()`, `clearAll()` |
| `AvatarView` | 3D rendering view | constructor, `dispose()`, `exportBitmap()`, `avatarTransform`, `getBoundingRect()` |
| `AvatarController` | Runtime communication and playback control | `start()`, `send()`, `yieldAudioData()`, `yieldFramesData()`, `pause()`, `resume()`, `interrupt()` |
**App ID** is required on every integration and identifies your Spatius application (it scopes which avatars you can load). **Session Token** is only required for **Direct Mode** (`DrivingServiceMode.direct`), where it authenticates the Motion Server WebSocket; LiveKit Agents, Agora Convo AI, and Backend Mode do not need it.
For the minimum end-to-end integration walkthrough, open the Web tab in [Direct Mode Client](/direct-mode/client).
***
## AvatarSDK
Main entry point for SDK initialization and global configuration.
```typescript theme={null}
import { AvatarSDK } from '@spatius/avatarkit'
```
### Static properties
| Property | Type | Description |
| ---------------------------- | ----------------------- | ---------------------------------------------------------- |
| `appId` | `string \| null` | Current App ID |
| `configuration` | `Configuration \| null` | Current SDK configuration |
| `sessionToken` | `string \| null` | Current Session Token |
| `userId` | `string \| null` | Current user ID (telemetry) |
| `version` | `string` | SDK version |
| `renderResolutionCapEnabled` | `boolean` | Whether render output resolution capping is enabled |
| `renderResolutionMaxHeight` | `number` | Maximum backing-buffer height used when capping is enabled |
### Static methods
#### initialize(appId, configuration)
Initialize the SDK. Must be called before any other operation.
```typescript theme={null}
await AvatarSDK.initialize('your-app-id', {
region: 'us-west', // optional, default: closest region
drivingServiceMode: DrivingServiceMode.direct, // optional, default: direct
logLevel: LogLevel.warning, // optional, default: off
renderQuality: RenderQuality.ultra, // optional, default: ultra
audioFormat: { // optional
channelCount: 1,
sampleRate: 16000,
},
})
```
#### setSessionToken(token)
Set the Session Token used to authenticate the Motion Server WebSocket. **Only required in Direct Mode** (`DrivingServiceMode.direct`); `AvatarController.start()` opens that WebSocket and authenticates with the token.
```typescript theme={null}
AvatarSDK.setSessionToken('your-session-token')
```
* Required for `DrivingServiceMode.direct` before calling `AvatarController.start()`.
* **Not required** for `DrivingServiceMode.backend` (Backend Mode) or `DrivingServiceMode.rtc` (the recommended RTC integrations). Those paths receive audio and motion data externally and do not open a Motion Server WebSocket from the client; `AvatarManager.load()` fetches avatar metadata over an App-ID-scoped public endpoint.
* The token must be obtained from your backend (see [Session token API](/api-reference/api-reference)).
* Maximum 24-hour validity.
* Token must be paired with the App ID used in `initialize()`.
`setSessionToken()` can be called before or after `initialize()`. If called before, the token is applied automatically during initialization.
#### setUserId(userId)
Set a user identifier for logging and telemetry.
```typescript theme={null}
AvatarSDK.setUserId('user-123')
```
#### setRenderQuality(quality)
Update the global rendering quality tier. The default is `RenderQuality.ultra`; no setter call is required unless you change the tier at runtime. The change takes effect on the next rendered frame across active `AvatarView` instances.
```typescript theme={null}
AvatarSDK.setRenderQuality(RenderQuality.ultra)
```
#### setRenderResolutionCap(enabled, maxHeight?)
Cap the render backing-buffer height while keeping the same CSS size. This is useful on high-DPI displays where rendering above the source avatar asset resolution adds cost without visible benefit.
```typescript theme={null}
AvatarSDK.setRenderResolutionCap(true, 1080)
```
#### deviceScore()
Run a short CPU/GPU benchmark and return device scores.
```typescript theme={null}
const { cpuScore, gpuScore } = await AvatarSDK.deviceScore()
```
#### isDeviceSupported()
Check whether the current device can run AvatarKit. This runs the same benchmark path used by `deviceScore()`.
```typescript theme={null}
const supported = await AvatarSDK.isDeviceSupported()
```
#### cleanup()
Release all SDK resources. Call when the SDK is no longer needed.
```typescript theme={null}
AvatarSDK.cleanup()
```
***
## AvatarManager
Handles avatar asset loading and caching. Access via the singleton `AvatarManager.shared`.
```typescript theme={null}
import { AvatarManager } from '@spatius/avatarkit'
```
### Static properties
| Property | Type | Description |
| -------- | --------------- | ------------------ |
| `shared` | `AvatarManager` | Singleton instance |
### Instance methods
#### load(id, onProgress?, useCompressedModel?)
Load an avatar by ID. Downloads and caches the avatar's assets.
```typescript theme={null}
const avatar = await AvatarManager.shared.load('avatar-id', (progress) => {
switch (progress.type) {
case 'downloading': {
// progress.progress is in 0..1; multiply by 100 to render as a percentage.
const percent = Math.round((progress.progress ?? 0) * 100)
console.log(`Loading: ${percent}%`)
break
}
case 'completed':
console.log('Load complete')
break
case 'failed':
console.error('Load failed:', progress.error)
break
}
})
```
| Parameter | Type | Description |
| -------------------- | -------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `id` | `string` | Avatar ID |
| `onProgress` | `(progress: LoadProgressInfo) => void` | Optional progress callback |
| `useCompressedModel` | `boolean` | Optional. Loads a smaller compressed model asset with minor quality tradeoff. Defaults to `false`. |
**Returns:** `Promise`
#### cancelLoad(id)
Cancel a pending or running avatar load task.
```typescript theme={null}
const cancelled = AvatarManager.shared.cancelLoad('avatar-id')
```
#### retrieve(id)
Return a cached avatar instance, if available.
```typescript theme={null}
const cachedAvatar = AvatarManager.shared.retrieve('avatar-id')
```
#### clear(id)
Clear a specific avatar from cache.
```typescript theme={null}
AvatarManager.shared.clear('avatar-id')
```
#### clearAll()
Clear all cached avatar resources.
```typescript theme={null}
AvatarManager.shared.clearAll()
```
***
## AvatarView
3D rendering view. Automatically creates a Canvas element and an associated `AvatarController`.
```typescript theme={null}
import { AvatarView } from '@spatius/avatarkit'
```
### Constructor
```typescript theme={null}
const avatarView = new AvatarView(avatar, container)
```
| Parameter | Type | Description |
| ----------- | ------------- | --------------------------------------------------- |
| `avatar` | `Avatar` | Loaded avatar object |
| `container` | `HTMLElement` | Container element (canvas auto-fills the container) |
**Container requirement:** the container element must have non-zero `width` and `height`. The canvas fills the container and auto-resizes via `ResizeObserver`.
### Render over an Avatar Background
Download the optional 16:9 background from Spatius Studio and store it as an application asset. Set it on the same stage element that you pass to `AvatarView`; the SDK-created canvas has a transparent background.
```html theme={null}
```
```css theme={null}
#avatar-stage {
position: relative;
width: 100%;
aspect-ratio: 16 / 9;
overflow: hidden;
background: url('/backgrounds/my-avatar.webp') center / cover no-repeat;
}
```
```typescript theme={null}
const stage = document.querySelector('#avatar-stage')!
const avatar = await AvatarManager.shared.load('avatar-id')
const avatarView = new AvatarView(avatar, stage)
```
For square or portrait display windows, keep this inner stage at 16:9, center it, and clip it with an outer container. See [Avatar Background](/concepts/avatar-background) for the shared cropping rules.
### Instance properties
| Property | Type | Description |
| ------------------ | ------------------- | -------------------------------------------- |
| `controller` | `AvatarController` | Communication controller (read-only) |
| `avatarTransform` | `{ x, y, scale }` | Avatar position and scale (see below) |
| `renderSize` | `{ width, height }` | Current canvas backing-buffer size in pixels |
| `onFirstRendering` | `() => void` | Callback fired when the first frame renders |
**Transform coordinates**
| Field | Range | Description |
| ------- | ------- | ---------------------------------------------------- |
| `x` | -1 to 1 | Horizontal offset (-1 = left, 0 = center, 1 = right) |
| `y` | -1 to 1 | Vertical offset (-1 = bottom, 0 = center, 1 = top) |
| `scale` | > 0 | Scale factor (1.0 = original size) |
### Instance methods
#### dispose()
Release all view resources. Call when the view is no longer needed (see [Lifecycle management](#lifecycle-management) for details).
```typescript theme={null}
avatarView.dispose()
```
#### exportBitmap()
Capture the current rendered avatar frame as a PNG `Blob`. Returns `null` if the canvas is not initialized or not currently rendering.
```typescript theme={null}
const png = await avatarView.exportBitmap()
if (png) {
// Use the Blob for download, upload, or preview.
}
```
#### getCameraConfig() / updateCameraConfig(cameraConfig)
Read or update the camera used by the renderer.
```typescript theme={null}
const camera = avatarView.getCameraConfig()
if (camera) {
avatarView.updateCameraConfig({
...camera,
fov: 35,
})
}
```
#### pauseRendering() / resumeRendering()
Pause or resume GPU/canvas rendering without stopping audio playback.
```typescript theme={null}
avatarView.pauseRendering()
avatarView.resumeRendering()
```
#### isRenderingEnabled()
Check whether the render loop is currently active.
```typescript theme={null}
const rendering = avatarView.isRenderingEnabled()
```
#### getBoundingRect()
Return the approximate avatar bounds in CSS pixels, or `null` if the view is not ready. Recalculate after container size or `avatarTransform` changes.
```typescript theme={null}
const rect = avatarView.getBoundingRect()
if (rect) {
console.log(rect.x, rect.y, rect.width, rect.height)
}
```
***
## AvatarController
Handles runtime communication with Motion Server and playback control.
```typescript theme={null}
// Accessed via AvatarView
const controller = avatarView.controller
```
### Event callbacks
```typescript theme={null}
// Connection state changes
controller.onConnectionState = (state: ConnectionState) => {
// 'disconnected' | 'connecting' | 'connected' | 'failed'
}
// Conversation state changes
controller.onConversationState = (state: ConversationState) => {
// 'idle' | 'playing' | 'paused'
}
// Error events
controller.onError = (error: AvatarError) => {
console.error('Error:', error.code, error.message)
}
// Animation type changes
controller.onAnimationState = (type: AnimationType) => {
// 'idle' | 'mono'
}
// Only fires when frameStarvationMode is FrameStarvationMode.strictSync
controller.onPlaybackStall = (stalled: boolean) => {
console.log('Playback stalled:', stalled)
}
```
### Instance properties
| Property | Type | Description |
| ------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `frameStarvationMode` | `FrameStarvationMode` | Controls what happens when motion data cannot keep up with the audio clock. Defaults to `FrameStarvationMode.audioIndependent`. |
| `onFrameRateInfo` | `(info: FrameRateInfo) => void` | Optional callback for aggregated frame-rate metrics. |
| `frameRateMonitorEnabled` | `boolean` | Enables frame-rate monitoring. Defaults to `false`. |
### `DrivingServiceMode.direct` methods
Available when `drivingServiceMode` is `DrivingServiceMode.direct` (the Direct Mode path).
#### initializeAudioContext()
Initialize the audio context. **Must be called inside a user-gesture handler** (e.g. a `click` listener).
```typescript theme={null}
button.addEventListener('click', async () => {
await controller.initializeAudioContext()
})
```
#### start()
Connect to Motion Server.
```typescript theme={null}
await controller.start()
```
#### send(audioData, end)
Send avatar speech audio. Returns the `conversationId` for the current round.
For audio source and timing guidance, see [Audio](/concepts/audio).
```typescript theme={null}
const conversationId = controller.send(
audioData, // PCM16, mono, ArrayBuffer
end, // true = end of conversation round
)
```
| Parameter | Type | Description |
| ----------- | ------------- | ---------------------------------------------------- |
| `audioData` | `ArrayBuffer` | PCM16 mono audio data |
| `end` | `boolean` | Whether this is the final chunk of the current round |
**`send()` behavior:**
* `end: false` — continues the current conversation round.
* `end: true` — marks the end of audio input for the current round. The avatar plays the remaining animation, then returns to idle (notified via `onConversationState`). Sending new audio after this starts a new round and interrupts any ongoing playback.
#### close()
Close the Motion Server connection.
```typescript theme={null}
controller.close()
```
### `DrivingServiceMode.backend` methods
Available when `drivingServiceMode` is `DrivingServiceMode.backend` (the Backend Mode path).
#### yieldAudioData(audioData, end?)
Provide audio data when your backend owns the Motion Server connection.
```typescript theme={null}
const conversationId = controller.yieldAudioData(audioData, false)
```
| Parameter | Type | Description |
| ----------- | ------------ | ------------------------------------------------------------------------ |
| `audioData` | `Uint8Array` | Encoded audio payload from your backend transport |
| `end` | `boolean` | Optional. Whether this is the final audio payload for the current round. |
**Returns:** `string | null` — conversation ID for this audio round, or `null` if audio playback could not start.
#### yieldFramesData(motionDataPayloads, conversationId)
Provide motion data payloads when using `DrivingServiceMode.backend`. The `conversationId` must match the one returned by the corresponding `yieldAudioData()` call.
```typescript theme={null}
const ended = controller.yieldFramesData(motionDataPayloads, conversationId)
```
| Parameter | Type | Description |
| -------------------- | ------------------------------- | ------------------------------------------------ |
| `motionDataPayloads` | `(Uint8Array \| ArrayBuffer)[]` | Motion data payloads from your backend transport |
| `conversationId` | `string` | Conversation ID returned by `yieldAudioData()` |
**Returns:** `boolean` — `true` when the final motion data payload for that conversation has been received, otherwise `false`.
### Common methods
Available in both `DrivingServiceMode.direct` and `DrivingServiceMode.backend`.
```typescript theme={null}
controller.pause() // Pause audio + animation
await controller.resume() // Resume playback
controller.interrupt() // Stop current playback
controller.getAudioTime() // Current audio playback time in seconds
// Volume control (avatar audio only, not system volume)
controller.setVolume(0.5) // 0.0 to 1.0
controller.getVolume() // returns current volume
```
***
## Types and Enums
### Configuration
```typescript theme={null}
interface Configuration {
region?: string // Default: automatic selection
drivingServiceMode?: DrivingServiceMode // Default: DrivingServiceMode.direct
logLevel?: LogLevel // Default: LogLevel.off
audioFormat?: AudioFormat // Default: { channelCount: 1, sampleRate: 16000 }
customEndpoint?: string // Advanced override
renderQuality?: RenderQuality // Default: RenderQuality.ultra
}
```
`region` selects the Spatius deployment region. Supported values are `'us-west'`, `'ap-northeast'`, and `'cn-beijing'`. When unset, the SDK selects the closest serving region at initialization; passing a value forces that region. If automatic selection cannot be reached, the SDK falls back to a default region and continues initializing. See [Regions](/api-reference/regions) for endpoint details and override options.
`initialize` fails fast when `appId` is missing, rather than returning silently. Get an App ID from [app.spatius.ai](https://app.spatius.ai/).
### DrivingServiceMode
```typescript theme={null}
enum DrivingServiceMode {
direct = 'direct', // Client SDK handles the Motion Server connection (Direct Mode)
backend = 'backend', // Your backend handles the Motion Server connection (Backend Mode)
rtc = 'rtc', // Driven through @spatius/avatarkit-rtc (RTC Adapter)
}
```
Use `rtc` only when driving the avatar through `@spatius/avatarkit-rtc`. It unlocks no API and does not connect to a room by itself. See the [RTC Adapter](/sdk-reference/web-sdk/rtc-adapter) and the relevant [LiveKit Agents Client](/livekit-agents/client) or [Agora Convo AI Client](/agora-convoai/client).
### RenderQuality
```typescript theme={null}
enum RenderQuality {
standard = 'standard',
high = 'high',
ultra = 'ultra',
}
```
`ultra` is the default and highest-quality tier. Use `high` or `standard` only when you intentionally trade visual quality for lower rendering cost.
### FrameStarvationMode
```typescript theme={null}
enum FrameStarvationMode {
audioIndependent = 'audioIndependent',
strictSync = 'strictSync',
}
```
`audioIndependent` keeps audio playing while motion data catches up. `strictSync` pauses audio when motion data runs out and resumes when new motion data arrives; use `onPlaybackStall` to observe those transitions.
### AnimationType
```typescript theme={null}
enum AnimationType {
idle = 'idle',
mono = 'mono',
}
```
### LogLevel
```typescript theme={null}
enum LogLevel {
off = 'off', // No logging (default)
error = 'error', // Errors only
warning = 'warning', // Errors + warnings
all = 'all', // All logs
}
```
### ConnectionState
Reported via `onConnectionState`. Only emitted when `drivingServiceMode` is `DrivingServiceMode.direct`.
```typescript theme={null}
enum ConnectionState {
disconnected = 'disconnected',
connecting = 'connecting',
connected = 'connected',
failed = 'failed',
}
```
### ConversationState
Reported via `onConversationState`.
```typescript theme={null}
enum ConversationState {
idle = 'idle', // Breathing animation, waiting for input
playing = 'playing', // Active conversation playback
paused = 'paused', // Paused during playback
}
```
State transitions are notified immediately when the transition starts, not when the animation completes. For example, `playing` is reported as soon as the transition from `idle` begins.
### LoadProgressInfo
```typescript theme={null}
type LoadProgressInfo = {
type: 'downloading' | 'completed' | 'failed'
progress?: number // 0..1
error?: Error
}
```
Multiply by 100 when rendering as a percentage in your UI.
### AudioFormat
The SDK takes audio as **mono PCM16** by default, and can also take Opus.
```typescript theme={null}
interface AudioFormat {
readonly channelCount: 1 // Fixed to mono
readonly sampleRate: number // Default: 16000
readonly inputAudioFormat?: 'pcm' | 'opus' // Default: 'pcm'
readonly opusUplinkEnabled?: boolean // Default: true
readonly opusBitrate?: number // Default: 48000
}
```
| Property | Value |
| --------------- | -------------------------------------------------------------------------------------- |
| **Format** | PCM16 (16-bit signed integer, little-endian) |
| **Channels** | Mono (1 channel) |
| **Sample rate** | Configurable: 8000 / 16000 / 22050 / 24000 / 32000 / 44100 / 48000 Hz (default: 16000) |
| **Data type** | `ArrayBuffer` or `Uint8Array` |
| Field | What it does |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `inputAudioFormat` | The codec of the audio you feed into the SDK via `send()` / `yieldAudioData()`. Fixed for the whole session; do not vary it per call. Under `'opus'` the configured `sampleRate` does not apply: Opus always decodes at 48 kHz, and the SDK reports that rate for the session. |
| `opusUplinkEnabled` | Whether the SDK compresses the Direct Mode uplink to Opus. On by default, which reduces the upload to roughly an eighth at the cost of encoding on the client. Set `false` to keep a raw PCM uplink. Not used in Backend Mode, which has no uplink, or with Opus input, which is already compressed. With PCM input, `sampleRate` must be 8000, 16000, 24000, or 48000; on any other rate the SDK logs a warning and sends raw PCM instead of failing to initialize. |
| `opusBitrate` | Target bitrate in bits/sec for the uplink the SDK encodes. Defaults to 48000. Higher means better quality and a larger upload. |
Under `inputAudioFormat: 'opus'` the SDK accepts either shape TTS providers produce, and detects which from the bytes:
* **Ogg Opus**, as file-style APIs return it. Any chunk boundary is fine.
* **Bare Opus packets, one per call**, as streaming APIs push them over a WebSocket. These cannot be batched: an Opus packet carries no length of its own, so several concatenated into one buffer have no recoverable boundaries.
Opus audio must be mono and must keep one shape for the whole conversation. WebM, MP4, WAV and CAF are not demuxed, even when the Opus inside them is valid. Audio that does not meet these requirements is reported through `onError` as `ErrorCode.invalidAudioInput`.
**Data size:** 1 second at 16 kHz = 16,000 samples × 2 bytes = 32,000 bytes.
```typescript theme={null}
async function mp3ToPcm16(mp3File: File, targetSampleRate: number): Promise {
const arrayBuffer = await mp3File.arrayBuffer()
const audioContext = new AudioContext({ sampleRate: targetSampleRate })
const audioBuffer = await audioContext.decodeAudioData(arrayBuffer.slice(0))
const length = audioBuffer.length
const channels = audioBuffer.numberOfChannels
const pcm16Buffer = new ArrayBuffer(length * 2)
const pcm16View = new DataView(pcm16Buffer)
// Mix to mono if stereo
const mono = channels === 1
? audioBuffer.getChannelData(0)
: (() => {
const mixed = new Float32Array(length)
const left = audioBuffer.getChannelData(0)
const right = audioBuffer.getChannelData(1)
for (let i = 0; i < length; i++) mixed[i] = (left[i] + right[i]) / 2
return mixed
})()
// Float32 → Int16
for (let i = 0; i < length; i++) {
const s = Math.max(-1, Math.min(1, mono[i]))
pcm16View.setInt16(i * 2, s < 0 ? s * 0x8000 : s * 0x7FFF, true)
}
audioContext.close()
return pcm16Buffer
}
```
***
## Error Handling
### AvatarError
```typescript theme={null}
import { AvatarError } from '@spatius/avatarkit'
try {
await avatarView.controller.start()
} catch (error) {
if (error instanceof AvatarError) {
console.error('SDK error:', error.message, error.code)
}
}
```
### Error callback
```typescript theme={null}
avatarView.controller.onError = (error: AvatarError) => {
console.error('Controller error:', error.code, error.message)
}
```
### ErrorCode
`AvatarError.code` is one of the SDK string enum values below.
```typescript theme={null}
enum ErrorCode {
appIDUnrecognized = 'appIDUnrecognized',
sessionTokenInvalid = 'sessionTokenInvalid',
sessionTokenExpired = 'sessionTokenExpired',
insufficientBalance = 'insufficientBalance',
concurrentLimitExceeded = 'concurrentLimitExceeded',
avatarIDUnrecognized = 'avatarIDUnrecognized',
failedToFetchAvatarMetadata = 'failedToFetchAvatarMetadata',
invalidAvatarMetadata = 'invalidAvatarMetadata',
failedToDownloadAvatarAssets = 'failedToDownloadAvatarAssets',
unsupportedAvatarAsset = 'unsupportedAvatarAsset',
websocketError = 'websocketError',
websocketClosedAbnormally = 'websocketClosedAbnormally',
websocketClosedUnexpected = 'websocketClosedUnexpected',
sessionTimeout = 'sessionTimeout',
connectionInProgress = 'connectionInProgress',
networkLayerNotAvailable = 'networkLayerNotAvailable',
playbackStartFailed = 'playbackStartFailed',
playbackInitFailed = 'playbackInitFailed',
audioOnlyInitFailed = 'audioOnlyInitFailed',
noAudio = 'noAudio',
invalidAudioInput = 'invalidAudioInput',
audioContextNotInitialized = 'audioContextNotInitialized',
animationPlayerNotInitialized = 'animationPlayerNotInitialized',
serverError = 'serverError',
}
```
See [Client Error Handling](/resources/client-error) for recovery guidance and [Server Error Handling](/resources/server-error) for Console API and Motion Server errors.
***
## Lifecycle Management
### Avatar switching
```typescript theme={null}
// 1. Dispose the current view
currentAvatarView.dispose()
// 2. Load a new avatar
const newAvatar = await AvatarManager.shared.load('new-avatar-id')
// 3. Create a new view (reuse the same container)
currentAvatarView = new AvatarView(newAvatar, container)
// 4. Reconnect
await currentAvatarView.controller.initializeAudioContext()
await currentAvatarView.controller.start()
```
### Resource cleanup
`dispose()` automatically cleans up:
* WebSocket connections
* Audio playback data and animation resources
* Canvas elements and the render system
* Event listeners and callbacks
Always call `dispose()` when the view is no longer needed. Failing to do so may cause memory leaks.
### Fallback mechanism
If the WebSocket connection fails within 15 seconds, the SDK automatically enters **audio-only fallback mode** — audio continues playing without animation. This keeps playback uninterrupted when Motion Server is unreachable.
* Fallback mode is interruptible like normal playback.
* `onConnectionState` reports `failed` when the connection times out.
***
## Browser Compatibility
| Browser | Minimum version | Rendering |
| -------------- | --------------- | ------------------ |
| Chrome / Edge | 90+ | WebGPU (preferred) |
| Firefox | 90+ | WebGL |
| Safari | 14+ | WebGL |
| iOS Safari | 14+ | WebGL |
| Android Chrome | 90+ | WebGL |
***
## Common Issues
| Issue | Cause | Solution |
| --------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------ |
| Audio not working | `initializeAudioContext()` not in a user gesture | Call it inside a `click` or `touchstart` handler. |
| Avatar not rendering | Container has zero dimensions | Set explicit `width` and `height` on the container. |
| WASM MIME type error | Build tool misconfigured | Use the Vite plugin or Next.js wrapper. See [Toolchain Setup](/sdk-reference/web-sdk/toolchain). |
| Session Token invalid | Token expired or not set | Refresh token from backend; call `setSessionToken()` before `start()`. |
| WebSocket connection failed | Network or auth issue | Check network connectivity and Session Token validity. |
***
## Complete Usage Example
```typescript theme={null}
import {
AvatarSDK,
AvatarManager,
AvatarView,
ConnectionState,
ConversationState,
} from '@spatius/avatarkit'
class AvatarApp {
private avatarView: AvatarView | null = null
async init(appId: string, sessionToken: string, avatarId: string, container: HTMLElement) {
// Initialize the SDK
await AvatarSDK.initialize(appId, {})
AvatarSDK.setSessionToken(sessionToken)
// Load the avatar
const avatar = await AvatarManager.shared.load(avatarId)
if (!avatar) throw new Error('Failed to load avatar')
// Create the view
this.avatarView = new AvatarView(avatar, container)
this.avatarView.onFirstRendering = () => {
console.log('First frame rendered')
}
// Wire up handlers
this.avatarView.controller.onConnectionState = (state) => {
console.log('Connection:', state)
}
this.avatarView.controller.onConversationState = (state) => {
console.log('Conversation:', state)
}
this.avatarView.controller.onError = (error) => {
console.error('Error:', error)
}
}
// Must be called inside a user-gesture handler
async start() {
await this.avatarView?.controller.initializeAudioContext()
await this.avatarView?.controller.start()
}
send(audioData: ArrayBuffer, isEnd: boolean) {
this.avatarView?.controller.send(audioData, isEnd)
}
interrupt() {
this.avatarView?.controller.interrupt()
}
dispose() {
this.avatarView?.controller.close()
this.avatarView?.dispose()
this.avatarView = null
}
}
```
# RTC Adapter
Source: https://docs.spatius.ai/sdk-reference/web-sdk/rtc-adapter
Reference for @spatius/avatarkit-rtc, the Web transport adapter for LiveKit and Agora RTC providers.
`@spatius/avatarkit-rtc` feeds motion data from a supported RTC room into AvatarKit for local rendering. In its managed path, it also creates the RTC client and plays the remote audio.
## Package boundary
| Package | Role |
| ------------------------ | ------------------------------------------------- |
| `@spatius/avatarkit` | Loads and renders the avatar in the browser. |
| `@spatius/avatarkit-rtc` | Connects AvatarKit to LiveKit or Agora RTC rooms. |
The RTC Adapter is part of the Web SDK family. Native Agora clients use `AvatarKitRTC` on iOS or `ai.spatius:avatarkit-rtc` on Android.
| Integration | Use the Web RTC Adapter? |
| -------------------------- | -------------------------------------------------------- |
| LiveKit Agents Integration | Yes, with `LiveKitProvider`. |
| Agora Convo AI Integration | Yes, with `AgoraProvider`. |
| Direct Mode Integration | No. Use `@spatius/avatarkit`. |
| Backend Mode Integration | No. Feed the backend output into the core AvatarKit SDK. |
## Install
Install the peer for the provider your application uses.
```bash theme={null}
npm install @spatius/avatarkit @spatius/avatarkit-rtc livekit-client
```
```bash theme={null}
npm install @spatius/avatarkit @spatius/avatarkit-rtc agora-rtc-sdk-ng
```
### Package compatibility
`@spatius/avatarkit-rtc` declares peer dependencies on AvatarKit and the selected RTC client. Install the packages together, and treat peer-dependency warnings as compatibility errors. For exact current ranges, check the package's published [`peerDependencies`](https://www.npmjs.com/package/@spatius/avatarkit-rtc?activeTab=code) instead of relying on a version copied into this page.
## Runtime contract
Initialize AvatarKit in RTC mode before constructing `AvatarPlayer`.
```typescript theme={null}
import { AvatarSDK, DrivingServiceMode } from '@spatius/avatarkit'
await AvatarSDK.initialize(appId, {
drivingServiceMode: DrivingServiceMode.rtc,
})
```
The adapter owns the RTC-to-renderer data flow. Do not manually call `yieldAudioData()` or `yieldFramesData()` while it is active.
## Providers
```typescript theme={null}
import {
AgoraProvider,
AvatarPlayer,
LiveKitProvider,
} from '@spatius/avatarkit-rtc'
const provider = new LiveKitProvider()
const player = new AvatarPlayer(provider, avatarView)
```
| Provider | Connection config | Used by |
| ----------------- | ------------------------- | -------------- |
| `LiveKitProvider` | `LiveKitConnectionConfig` | LiveKit Agents |
| `AgoraProvider` | `AgoraConnectionConfig` | Agora Convo AI |
### `LiveKitProviderOptions`
Most applications use `new LiveKitProvider()` without options. Use these options only when the room publishes motion data under custom track names.
| Option | Type | Description |
| --------------------------- | -------- | --------------------------------------------------------------- |
| `animationTrackName` | `string` | Exact publication track name to treat as the motion data track. |
| `animationTrackNamePattern` | `RegExp` | Pattern used to match motion data track names. |
```typescript theme={null}
const provider = new LiveKitProvider({
animationTrackNamePattern: /^spatius-motion-/,
})
```
## Connection configuration
```typescript theme={null}
interface LiveKitConnectionConfig {
url: string
token: string
roomName: string
}
interface AgoraConnectionConfig {
appId: string
channel: string
token?: string
uid?: number
}
```
Use short-lived room credentials returned by your backend. Never put a LiveKit API secret, Agora App Certificate, or Spatius API Key in the browser.
## `AvatarPlayer`
### Constructor
```typescript theme={null}
new AvatarPlayer(
provider: RTCProvider,
avatarView: AvatarView,
options?: AvatarPlayerOptions,
)
```
### `AvatarPlayerOptions`
| Option | Type | Default | Description |
| -------------------- | ------------------------------------------ | ----------- | ------------------------------------------------------ |
| `logLevel` | `'info' \| 'warning' \| 'error' \| 'none'` | `'warning'` | SDK log level. |
| `enableJitterBuffer` | `boolean` | `true` | Buffers and orders motion data for smoother playback. |
| `maxBufferDelayMs` | `number` | `80` | Maximum jitter-buffer delay when buffering is enabled. |
### Properties
| Property | Type | Description |
| ---------------- | ---------------- | --------------------------------------------------------- |
| `isConnected` | `boolean` | Whether the player is connected or attached. |
| `sessionSummary` | SDK-owned object | Cumulative playback diagnostics for this player lifetime. |
Treat the shape of `sessionSummary` as diagnostics data, not application state.
### Methods
| Method | Returns | Description |
| --------------------------- | --------------- | ----------------------------------------------------------------------------------------- |
| `prepareConnection(config)` | `Promise` | Best-effort LiveKit connection pre-warm. Does not join the room. |
| `connect(config)` | `Promise` | Create and connect the provider-owned RTC client. |
| `disconnect()` | `Promise` | Disconnect the provider-owned client and release SDK-owned resources. |
| `attach(client)` | `Promise` | Attach to a host-owned RTC client. See [Host-owned RTC clients](#host-owned-rtc-clients). |
| `detach()` | `Promise` | Remove adapter listeners without disconnecting the host-owned client. |
| `startPublishing()` | `Promise` | Request microphone access and publish it. |
| `stopPublishing()` | `Promise` | Unpublish and release the SDK-owned microphone track. |
| `publishAudio(track)` | `Promise` | Publish a supplied audio `MediaStreamTrack`. |
| `unpublishAudio()` | `Promise` | Unpublish a supplied track without stopping it. |
| `reconnect()` | `Promise` | Reconnect with the last managed connection config. |
| `getConnectionState()` | `string` | Return the provider connection state. |
| `getNativeClient()` | `unknown` | Return the underlying LiveKit Room or Agora client. |
| `on(event, handler)` | `void` | Add an event handler. |
| `off(event, handler)` | `void` | Remove the same handler reference. |
### Audio publishing
Use `startPublishing()` for the microphone or supply any browser audio track with `publishAudio()`.
```typescript theme={null}
const stream = await navigator.mediaDevices.getUserMedia({ audio: true })
const track = stream.getAudioTracks()[0]
await player.publishAudio(track)
// Later
await player.unpublishAudio()
track.stop()
```
`unpublishAudio()` does not stop a host-supplied track. The application that created it must release it.
## State and events
The package exports `ConnectionState`:
```typescript theme={null}
enum ConnectionState {
Disconnected = 'disconnected',
Connecting = 'connecting',
Connected = 'connected',
Reconnecting = 'reconnecting',
Failed = 'failed',
}
```
| Event | Handler argument | Meaning |
| ---------------------------- | ----------------------- | --------------------------------------------------------------- |
| `'connected'` | None | A managed provider connected. |
| `'disconnected'` | None | A managed provider disconnected. |
| `'error'` | `Error` | The provider reported an error. |
| `'connection-state-changed'` | Connection state string | The provider state changed. |
| `'stalled'` | None | The motion data stream stalled and the avatar returned to idle. |
```typescript theme={null}
const recover = () => {
void player.reconnect()
}
player.on('stalled', recover)
player.off('stalled', recover)
```
## Host-owned RTC clients
The default `connect()` path lets the adapter create and own the RTC client. Use `attach()` only when your application already owns that client.
### LiveKit Room
Create the room with `singlePeerConnection: false`, attach before connecting, and use `detach()` before your application disconnects the room. In this flow, your application also owns remote audio playback.
```typescript theme={null}
import { Room } from 'livekit-client'
import { AvatarPlayer, LiveKitProvider } from '@spatius/avatarkit-rtc'
const room = new Room({ singlePeerConnection: false })
const player = new AvatarPlayer(new LiveKitProvider(), avatarView)
await player.attach(room)
await room.connect(url, token)
// Later
await player.detach()
await room.disconnect()
```
### Agora client
Attach before joining the channel. The host-owned client must use the H.264 codec and remains responsible for joining, subscribing, audio playback, and leaving.
Do not mix `connect()` with `attach()` on the same player.
## Native client access
For provider-specific features in the managed path, read the client owned by the provider:
```typescript theme={null}
import type { AgoraClient, LiveKitRoom } from '@spatius/avatarkit-rtc'
const room: LiveKitRoom | null = liveKitProvider.getNativeClient()
const agoraClient: AgoraClient | null = agoraProvider.getNativeClient()
```
`player.getNativeClient()` returns the same object as `unknown`; keep the provider reference when you want the provider-specific type.
## Browser requirements
The package targets browser applications. AvatarKit's WebAssembly and rendering requirements still apply; see [Web Toolchain Setup](/sdk-reference/web-sdk/toolchain).
`LiveKitProvider` additionally requires `RTCRtpScriptTransform` to receive motion data. Check for the capability at runtime instead of relying on a fixed browser-version table:
```typescript theme={null}
const liveKitMotionSupported =
typeof globalThis.RTCRtpScriptTransform !== 'undefined'
```
Browser autoplay policies can also require a user gesture before remote audio plays or microphone capture begins.
## Next steps
# Toolchain Setup
Source: https://docs.spatius.ai/sdk-reference/web-sdk/toolchain
Configure your web build tool to load AvatarKit WebAssembly assets
AvatarKit uses WebAssembly for avatar rendering. Your build tool must serve `.wasm` files with the correct MIME type and keep them as external assets instead of inlining them into JavaScript.
**Required:** Complete this setup before initializing the Web SDK. Missing WASM configuration commonly appears as a `.wasm` 404, an incorrect MIME type error, or a failed SDK initialization.
Use the official AvatarKit Vite plugin:
```typescript title="vite.config.ts" theme={null}
import { defineConfig } from 'vite'
import { avatarkitVitePlugin } from '@spatius/avatarkit/vite'
export default defineConfig({
plugins: [
avatarkitVitePlugin(),
],
})
```
The Vite plugin handles the WASM requirements for both development and production:
* Serves `.wasm` files as `application/wasm` in the Vite dev server
* Copies AvatarKit WASM files into `dist/assets/` during builds
* Generates a `_headers` file for Cloudflare Pages deployments
* Configures Vite options such as `optimizeDeps`, `assetsInclude`, and `assetsInlineLimit`
Wrap your Next.js config with `withAvatarkit`:
```javascript title="next.config.mjs" theme={null}
import { withAvatarkit } from '@spatius/avatarkit/next'
export default withAvatarkit({
// ...your existing Next.js config
})
```
The Next.js wrapper handles the Webpack and response-header work needed for AvatarKit WASM assets:
* Serves runtime assets from `${basePath}/_avatarkit/`
* Copies `.wasm` files into `.next/static/chunks/` as a standalone fallback
* Adds the `application/wasm` response header for `/_avatarkit/:path*.wasm`
* Preserves any existing `webpack` and `headers` configuration
If your project already uses another Next.js config wrapper, apply `withAvatarkit` to the final wrapped config:
```javascript title="next.config.mjs" theme={null}
import { withAvatarkit } from '@spatius/avatarkit/next'
import withOtherPlugin from 'other-plugin'
export default withAvatarkit(withOtherPlugin({
// ...your existing Next.js config
}))
```
## Verify Setup
After configuring your toolchain, run your app and confirm the AvatarKit `.wasm` request succeeds:
* The request should return `200`
* The response header should include `Content-Type: application/wasm`
* The file should be loaded as a separate `.wasm` asset, not inlined into JavaScript
If the WASM request fails, use the Vite plugin or Next.js wrapper above before debugging SDK initialization code.