createOpenAIRealtimeAdapter owns the browser voice path: microphone capture, WebRTC negotiation,
remote audio playback, input/output metering, interruption-aware state, and cleanup. Your server
still owns the standard OpenAI API key and creates a short-lived Realtime client secret.
The adapter targets the GA Realtime API. New browser integrations should use
/v1/realtime/client_secrets and /v1/realtime/calls, not the older beta session flow.
The simple setup
Your browser provides one function: getClientSecret. orb-ui owns everything after that callback,
including microphone permission, WebRTC negotiation, remote playback, conversation state, audio
metering, and cleanup. Most applications do not need any other adapter option.
Create a client secret on your server
This example uses the current general voice-agent model and the marin voice. Keep
OPENAI_API_KEY server-only.
Add authentication and rate limiting appropriate for your app before exposing a token endpoint.
Create the adapter
getClientSecret must mint a fresh secret for every start. Do not put a standard OpenAI API
key in browser code or a VITE_*/NEXT_PUBLIC_* variable.
State and volume mapping
- WebRTC negotiation ->
connecting
input_audio_buffer.speech_started -> listening
input_audio_buffer.speech_stopped and response.created -> thinking
- output audio activity ->
speaking
- completed/interrupted output ->
listening
- Realtime or WebRTC failure ->
error
- explicit stop or closed connection ->
idle
The adapter meters the local microphone into inputVolume while listening and the remote WebRTC
track into outputVolume while speaking. It also attaches and plays the model audio automatically.
Output calibration
OpenAI Realtime ships with tuned output defaults for noise floor, gain, curve, attack, and release.
Most applications should use those defaults. If your audio environment needs different shaping,
pass a partial outputVolumeCalibration object:
Pass a getter instead of an object to update calibration while a session is active. The optional
onOutputVolumeSample callback reports raw RMS, shaped, and smoothed values for diagnostics.
Runtime overrides
callsUrl, mediaStreamConstraints, getUserMedia, createPeerConnection, fetch,
createAudioElement, and createAudioContext are available for custom browser wrappers and tests.
Most applications only need getClientSecret.
ChatGPT Live
This adapter targets the public OpenAI Realtime API. It does not wrap consumer ChatGPT voice
features that do not expose a corresponding developer API.
Troubleshooting
The browser receives 401 or cannot connect. Verify that the server endpoint creates a fresh
short-lived client secret for every start and that the browser sends that value to the adapter.
Never substitute a standard API key in client code.
The orb connects but no remote audio plays. Check browser autoplay and audio-output behavior,
then inspect the WebRTC connection and remote track. The adapter creates and plays the media element
unless a custom runtime override replaces that behavior.
Listening does not react to speech. Confirm microphone permission and the requested media
constraints. Test the raw input diagnostics before changing visual calibration; a silent or wrong
input device cannot be fixed with gain.
Speaking motion is too quiet or too aggressive. Start with the tuned defaults, inspect samples
with onOutputVolumeSample, and change one calibration field at a time. Validate normal speech and
silence rather than tuning to a single loud clip.
Production checklist
- A server-only standard API key creates short-lived client secrets.
- The token endpoint is authenticated and rate-limited for the application’s threat model.
- Microphone denial, WebRTC failure, and token failure have distinct recovery messages.
- A fresh client secret is created after stop, error, or expiration.
- Start, interruption, stop, and restart are tested on supported browsers.
- A visible text status and labeled controls accompany the animation.
- Custom runtime overrides release media tracks, audio contexts, and peer connections.
Last modified on July 22, 2026