Skip to main content
Start here for new OpenAI GPT-Live integrations. createOpenAILiveAdapter owns microphone capture, WebRTC negotiation, audio playback, input/output metering, and session cleanup. Your server owns the OpenAI API key, voice model, instructions, and delegation configuration. GPT-Live uses a different API from Realtime. Use the Realtime adapter for existing Realtime sessions; changing its model or endpoint does not make it compatible with Live.

Create a session on your server

The browser sends an SDP offer to your authenticated application endpoint. The server exchanges it with POST /v1/live/sessions and returns the session ID and SDP answer. This example uses gpt-live-1 for speech and a Responses backend for delegated work.
Keep OPENAI_API_KEY server-only and protect session creation with your application’s authentication and request limits. WebRTC negotiates the audio format; omit audio.format. Live sessions are billed by duration, including an initialization minimum; see the official WebRTC guide.

Create the adapter

createSession runs for each start and returns { session: { id }, transport: { type: 'webrtc', sdp } }. Forward the provided AbortSignal to your fetch. No client SDK or Realtime client secret is required. start() resolves after both SDP negotiation and session.started; opening the data channel alone does not mean Live is ready.

Full-duplex state and volume

  • Connection setup → connecting
  • session.started and quiet assistant output → listening
  • Audible remote-track activity → speaking
  • Provider, playback, or connection failure → error
  • Confirmed session.closed or cancelled startup → idle
Live can listen and speak at the same time. Both volume fields remain active, and microphone activity does not suppress the speaking indicator. The single orb state shows assistant playback when both sides are talking. Live has no per-utterance audio completion event. The adapter estimates speech activity from the remote audio track, with a short silence hold; captions and delegated response.event completion do not start or stop speech. It does not infer thinking from backend events, since backend work and speech run independently. Use application state alongside the orb for task progress. The adapter starts with the shared OpenAI WebRTC RMS calibration, not a separately measured Live voice profile. inputVolumeCalibration, outputVolumeCalibration, onInputVolumeSample, and onOutputVolumeSample support directional tuning. If Web Audio metering is unavailable or disabled, volume and inferred speaking state are unavailable; session events still work.

Captions and delegation

Pass onEvent(event) to receive all JSON server events unchanged, including transcript timing, delegation IDs, nested Responses events, errors, usage updates, and session.closed. After startup, use adapter.send({ type: ..., ... }) for application commands allowed by your server’s frontend permissions. Handle custom tools on your trusted backend or sideband connection. The adapter does not execute tools or manage backend task state. For client delegation, supply conversation context to your own agent and return results with session.commentary.append. For Responses delegation, read inner response.output_item.done events, return function outputs with response.item.create, and continue with response.create. See the delegation guide for full event schemas, frontend permissions, and backend responsibilities. Do not send session.start on WebRTC: the HTTP request already starts the session. Use adapter.stop() to send session.close. It waits for session.closed before releasing media, so onEvent can collect final usage. Finish required delegated work before calling stop. On timeout or connection loss it releases resources and rejects with an error indicating that final usage is unconfirmed. Muting the microphone does not end a billed Live session.

Migrating from Realtime

  1. Replace createOpenAIRealtimeAdapter({ getClientSecret }) with createOpenAILiveAdapter({ createSession }).
  2. Replace your client-secret endpoint with the server-side JSON session exchange above.
  3. Split voice instructions from backend instructions and choose Responses or client delegation.
  4. Replace Realtime captions and tool handling with Live events. Remove manual audio commits and voice-response triggers. Keep existing Realtime users on the unchanged Realtime adapter.
This is an additive orb-ui API; no existing imports need to change. See OpenAI’s migration guide for the provider behavior changes.

Runtime options and troubleshooting

mediaStreamConstraints, getUserMedia, createPeerConnection, createAudioElement, and createAudioContext allow custom browser runtimes and tests. startTimeoutMs defaults to 30 seconds; closeTimeoutMs defaults to 15 seconds and may need increasing for stored sessions. No audio: Start from a user action on HTTPS or localhost. Check microphone permission and autoplay; playback failures surface as an error. If you provide an audio element, keep it unmuted. Startup fails: Verify project access to gpt-live-1, backend model access, your authenticated session endpoint, and its JSON response. The adapter waits for ICE gathering and session.started. Speaking never ends: Do not drive playback from transcript deltas or Responses completion. Inspect remote audio activity and calibration; Live has no Realtime-style output buffer end event. Stop fails: The connection ended or timed out before session.closed. Resources are released, but final usage is unconfirmed. Start a fresh session to reconnect.
Last modified on September 21, 2026