Agora Convo AI Client

Before you start

Start the Convo AI agent or TEN graph on your backend, then return this runtime configuration to the app:

Value	Source
Spatius App ID and Avatar ID	The values used by the agent setup
Agora App ID and channel	Your Agora project and agent session
Client UID and RTC token	Generated by your backend for this participant

Use a different Agora UID for the client, conversational agent, and avatar publisher. Never send the Spatius API Key or Agora App Certificate to the app. This path does not use a Spatius Session Token.

Connect the client

Web
iOS
Android

Install AvatarKit, the Web RTC Adapter, and Agora’s Web SDK:

pnpm add @spatius/avatarkit @spatius/avatarkit-rtc agora-rtc-sdk-ng

Configure the AvatarKit WebAssembly assets for Vite or Next.js, then connect:

import { AvatarManager, AvatarSDK, AvatarView, DrivingServiceMode } from '@spatius/avatarkit'
import { AgoraProvider, AvatarPlayer } from '@spatius/avatarkit-rtc'

await AvatarSDK.initialize(spatiusAppId, {
  drivingServiceMode: DrivingServiceMode.rtc,
})

const avatar = await AvatarManager.shared.load(avatarId)
const avatarView = new AvatarView(
  avatar,
  document.getElementById('avatar-container')!,
)
const player = new AvatarPlayer(new AgoraProvider(), avatarView, {
  logLevel: 'info',
})

await player.connect({
  appId: agoraAppId,
  channel,
  token: clientToken,
  uid: clientUid,
})

await player.startPublishing()

async function disconnectAvatar() {
  await player.stopPublishing()
  await player.disconnect()
  avatarView.dispose()
}

Call the connect flow from a user action so the browser can request microphone permission.The default connect() path lets AgoraProvider create and own the Agora client. If your app already owns one, attach the player to it before joining the channel instead. The client must use the H.264 codec. The two paths are mutually exclusive.

import AgoraRTC from 'agora-rtc-sdk-ng'
import { AgoraProvider, AvatarPlayer } from '@spatius/avatarkit-rtc'

const client = AgoraRTC.createClient({ mode: 'rtc', codec: 'h264' })
const player = new AvatarPlayer(new AgoraProvider(), avatarView)

// Attach before joining the channel.
await player.attach(client)

// Your app subscribes to and plays the agent's audio.
client.on('user-published', async (user, mediaType) => {
  if (mediaType !== 'audio') return
  await client.subscribe(user, 'audio')
  user.audioTrack?.play()
})

await client.join(agoraAppId, channel, clientToken, clientUid)

async function disconnectAvatar() {
  await player.detach()
  await client.leave()
  avatarView.dispose()
}

Your app keeps owning the client: it joins and leaves the channel, subscribes to remote tracks, and plays remote audio. detach() hands it back without leaving the channel. attach() turns on two Agora Web SDK parameters, ENABLE_VIDEO_SEI and ENABLE_FULL_LINK_AV_SYNC, which the avatar needs. They are set on the AgoraRTC module, so they apply to every Agora client on the page, and detach() does not turn them off.

Web RTC Adapter Reference

Install AvatarKitRTC with Swift Package Manager or CocoaPods:In Xcode, open File > Add Package Dependencies, enter https://github.com/spatius-ai/avatarkit-ios-rtc.git, and select the latest stable release.

pod 'AvatarKitRTC'

Initialize AvatarKit in RTC mode, load the Avatar, and connect:

import AvatarKit
import AvatarKitRTC

AvatarSDK.initialize(
    appID: spatiusAppID,
    configuration: Configuration(drivingServiceMode: .rtc)
)

let avatar = try await AvatarManager.shared.load(id: avatarID)
let avatarView = AvatarView(avatar: avatar)
let player = AvatarPlayer(
    provider: AgoraProvider(),
    avatarView: avatarView,
    options: AvatarPlayerOptions(logLevel: .info)
)

try await player.connect(AgoraConnectionConfig(
    appId: agoraAppID,
    channel: channel,
    token: clientToken,
    uid: clientUID
))

try await player.publishAudio()

func disconnectAvatar() async {
    await player.unpublishAudio()
    await player.disconnect()
}

Add NSMicrophoneUsageDescription to the app and request microphone permission before calling publishAudio().The default connect(_:) path lets AgoraProvider own the engine. If your iOS app already owns an AgoraRtcEngineKit, attach the player to it before joining the channel instead. The two paths are mutually exclusive; do not create a second engine only for avatar playback.

import AgoraRtcKit

let engineConfig = AgoraRtcEngineConfig()
engineConfig.appId = agoraAppID
engineConfig.channelProfile = .liveBroadcasting
let engine = AgoraRtcEngineKit.sharedEngine(with: engineConfig, delegate: yourEngineDelegate)

let player = AvatarPlayer(provider: AgoraProvider(), avatarView: avatarView)

// Attach before joining the channel.
try player.attach(to: engine)

engine.setClientRole(.broadcaster)
let options = AgoraRtcChannelMediaOptions()
options.clientRoleType = .broadcaster
options.autoSubscribeAudio = true
options.autoSubscribeVideo = true
options.publishCameraTrack = false
options.publishMicrophoneTrack = true
engine.joinChannel(byToken: clientToken, channelId: channel, uid: clientUID, mediaOptions: options)

func disconnectAvatar() async {
    await player.detach()
    engine.leaveChannel(nil)
    AgoraRtcEngineKit.destroy()
}

Your app keeps owning the engine: it creates it, sets the client role, joins and leaves the channel, and destroys it. detach() hands it back without leaving the channel. attach(to:) does change two engine settings, and detach() does not restore them: it enables SEI (rtc.video.enable_sei), which the avatar needs, and sets the audio scenario to .chorus. If your app needs a different audio scenario, set it after calling attach(to:).

Add the Android RTC package and Agora’s Android SDK:

implementation("ai.spatius:avatarkit-rtc:<latest-version>")
implementation("io.agora.rtc:full-sdk:4.5.2")

Use the latest stable release from Maven Central. The RTC package does not bring in Agora’s SDK, so your app must declare it. Use Agora 4.5.2: Agora 4.6.x is not supported.Initialize AvatarKit in RTC mode, load the Avatar, then connect the player:

import ai.spatius.avatarkit.AvatarSDK
import ai.spatius.avatarkit.AvatarView
import ai.spatius.avatarkit.Configuration
import ai.spatius.avatarkit.DrivingServiceMode
import ai.spatius.avatarkit.assets.AvatarManager
import ai.spatius.avatarkit.rtc.AgoraConnectionConfig
import ai.spatius.avatarkit.rtc.AvatarPlayer
import ai.spatius.avatarkit.rtc.AvatarPlayerOptions
import ai.spatius.avatarkit.rtc.RTCLogLevel
import ai.spatius.avatarkit.rtc.providers.AgoraProvider

AvatarSDK.initialize(
    context = context,
    appId = spatiusAppId,
    configuration = Configuration(
        drivingServiceMode = DrivingServiceMode.RTC,
    ),
)

AvatarManager.initialize(context)
val avatar = AvatarManager.load(avatarId) ?: error("Avatar load failed")
val avatarView = AvatarView(context)
avatarView.init(avatar, lifecycleScope)

val player = AvatarPlayer(
    provider = AgoraProvider(context),
    avatarView = avatarView,
    options = AvatarPlayerOptions(logLevel = RTCLogLevel.INFO),
)

player.connect(
    AgoraConnectionConfig(
        appId = agoraAppId,
        channel = channel,
        token = clientToken,
        uid = clientUid,
    )
)

player.publishAudio()

suspend fun disconnectAvatar() {
    player.unpublishAudio()
    player.disconnect()
}

Declare android.permission.RECORD_AUDIO and request the runtime permission before calling publishAudio().The default connect(...) path lets AgoraProvider own the engine. If your Android app already owns an RtcEngine, attach the player to it before joining the channel instead. The two paths are mutually exclusive; do not create a second engine only for avatar playback.

import io.agora.rtc2.ChannelMediaOptions
import io.agora.rtc2.Constants
import io.agora.rtc2.RtcEngine
import io.agora.rtc2.RtcEngineConfig

val engine = RtcEngine.create(RtcEngineConfig().apply {
    mContext = context
    mAppId = agoraAppId
    mEventHandler = yourEngineEventHandler
    mChannelProfile = Constants.CHANNEL_PROFILE_LIVE_BROADCASTING
})

val player = AvatarPlayer(
    provider = AgoraProvider(context),
    avatarView = avatarView,
)

// Attach before joining the channel.
player.attach(engine)

engine.setClientRole(Constants.CLIENT_ROLE_BROADCASTER)
engine.joinChannel(clientToken, channel, clientUid.toInt(), ChannelMediaOptions().apply {
    clientRoleType = Constants.CLIENT_ROLE_BROADCASTER
    autoSubscribeAudio = true
    autoSubscribeVideo = true
    publishCameraTrack = false
    publishMicrophoneTrack = true
})

suspend fun disconnectAvatar() {
    player.detach()
    engine.leaveChannel()
    RtcEngine.destroy()
}

Your app keeps owning the engine: it creates it, sets the client role, joins and leaves the channel, and destroys it. detach() hands it back without leaving the channel. attach(engine) does change two engine settings, and detach() does not restore them: it enables SEI (rtc.video.enable_sei), which the avatar needs, and sets the audio scenario to AUDIO_SCENARIO_CHORUS. If your app needs a different audio scenario, set it after calling attach(engine).

The packaged AgoraProvider coordinates audio and motion data. On the default connect path, it also owns the RTC lifecycle. Do not also call Backend Mode feed methods.

Show when the Avatar is speaking

AvatarPlayer reports whether the Avatar is idle or speaking on screen. Use it to drive speaking indicators and idle UI. The state turns to speaking on the first rendered frame of a response and returns to idle once the transition back to idle has finished playing, or when the stream stalls or the player disconnects or detaches. The event fires once per change, never per frame, and the current value is always readable from the player. AvatarKit’s own conversation state callback does not fire in RTC mode, so read the state from AvatarPlayer instead.

Web
iOS
Android

import { RTCConversationState } from '@spatius/avatarkit-rtc'

player.on('conversation-state-changed', (state: RTCConversationState) => {
  setSpeaking(state === RTCConversationState.Speaking)
})

// Current value: player.conversationState

player.subscribe { event in
    if case .conversationStateChanged(let state) = event {
        setSpeaking(state == .speaking)
    }
}

// Current value: player.conversationState (.idle or .speaking)

import ai.spatius.avatarkit.rtc.AvatarPlayerEvent
import ai.spatius.avatarkit.rtc.RTCConversationState

player.subscribe { event ->
    if (event is AvatarPlayerEvent.ConversationStateChanged) {
        setSpeaking(event.state == RTCConversationState.SPEAKING)
    }
}

// Current value: player.conversationState (IDLE or SPEAKING)

AvatarPlayerEvent is a sealed class. If your app handles it with an exhaustive when, add a ConversationStateChanged branch or an else branch.

Start Here

Quickstarts

Integrations

Concepts

Pricing

Examples & Support

Agora Convo AI Client

Before you start

Connect the client

Show when the Avatar is speaking

Next steps

Scenario demo

Integration demo

Client lifecycle

​Before you start

​Connect the client

​Show when the Avatar is speaking

​Next steps

Scenario demo

Integration demo

Client lifecycle

Before you start

Connect the client

Show when the Avatar is speaking

Next steps