createOpenAIRealtimeAdapter owns the browser voice path: microphone capture, WebRTC negotiation,
remote audio playback, input/output metering, interruption-aware state, and cleanup. Your server
still owns the standard OpenAI API key and creates a short-lived Realtime client secret.
The adapter targets the GA Realtime API. New browser integrations should use
/v1/realtime/client_secrets and /v1/realtime/calls, not the older beta session flow.
The simple setup
Your browser provides one function: getClientSecret. orb-ui owns everything after that callback,
including microphone permission, WebRTC negotiation, remote playback, conversation state, audio
metering, and cleanup. Most applications do not need any other adapter option.
Create a client secret on your server
This example uses the current general voice-agent model and the marin voice. Keep
OPENAI_API_KEY server-only.
Add authentication and rate limiting appropriate for your app before exposing a token endpoint.
Create the adapter
getClientSecret must mint a fresh secret for every start. Do not put a standard OpenAI API
key in browser code or a VITE_*/NEXT_PUBLIC_* variable.
State and volume mapping
- WebRTC negotiation ->
connecting
input_audio_buffer.speech_started -> listening
input_audio_buffer.speech_stopped and response.created -> thinking when no output is playing
output_audio_buffer.started -> speaking
- completed/interrupted output ->
listening
- Realtime or WebRTC failure ->
error
- explicit stop or closed connection ->
idle
The adapter meters the local microphone into inputVolume and the remote WebRTC track into
outputVolume, keeping both envelopes warm during active conversation states. It also attaches
and plays the model audio automatically.
WebRTC playback start, stop, and clear events take priority over volume-based state inference.
This prevents a listening flash before the first audio packet or a return to speaking while the
output envelope fades after playback ends. Detected user speech keeps listening priority during
interruption. Audio-level inference remains available when playback events are absent.
Directional calibration
OpenAI Realtime ships with separate input and output defaults that map normal speech to the same
portable 0–1 distribution as the other adapters. Most applications should use those defaults. If
your audio path needs different amplitude anchors or envelope timing, pass a partial directional
override:
Pass a getter instead of an object to update calibration while a session is active. The optional
onInputVolumeSample and onOutputVolumeSample callbacks report raw, mapped, and normalized values
for diagnostics. See volume calibration for the exact contract and
guided profile generator.
Runtime overrides
callsUrl, mediaStreamConstraints, getUserMedia, createPeerConnection, fetch,
createAudioElement, and createAudioContext are available for custom browser wrappers and tests.
Most applications only need getClientSecret.
ChatGPT Live
This adapter targets the public OpenAI Realtime API. It does not wrap consumer ChatGPT voice
features that do not expose a corresponding developer API.
Troubleshooting
The browser receives 401 or cannot connect. Verify that the server endpoint creates a fresh
short-lived client secret for every start and that the browser sends that value to the adapter.
Never substitute a standard API key in client code.
The orb connects but no remote audio plays. Check browser autoplay and audio-output behavior,
then inspect the WebRTC connection and remote track. The adapter creates and plays the media element
unless a custom runtime override replaces that behavior.
Listening does not react to speech. Confirm microphone permission and the requested media
constraints. Test the raw input diagnostics before changing calibration; a silent or wrong input
device cannot be fixed with amplitude mapping.
Speaking motion is too quiet or too aggressive. Start with the shipped defaults and use the
guided calibration runner to capture silence, quiet, normal, and energetic speech. Avoid tuning the
provider profile to compensate for one theme’s visual range.
Production checklist
- A server-only standard API key creates short-lived client secrets.
- The token endpoint is authenticated and rate-limited for the application’s threat model.
- Microphone denial, WebRTC failure, and token failure have distinct recovery messages.
- A fresh client secret is created after stop, error, or expiration.
- Start, interruption, stop, and restart are tested on supported browsers.
- A visible text status and labeled controls accompany the animation.
- Custom runtime overrides release media tracks, audio contexts, and peer connections.
Last modified on September 8, 2026