Skip to main content
createOpenAIRealtimeAdapter owns the browser voice path: microphone capture, WebRTC negotiation, remote audio playback, input/output metering, interruption-aware state, and cleanup. Your server still owns the standard OpenAI API key and creates a short-lived Realtime client secret. The adapter targets the GA Realtime API. New browser integrations should use /v1/realtime/client_secrets and /v1/realtime/calls, not the older beta session flow.

The simple setup

Your browser provides one function: getClientSecret. orb-ui owns everything after that callback, including microphone permission, WebRTC negotiation, remote playback, conversation state, audio metering, and cleanup. Most applications do not need any other adapter option.

Create a client secret on your server

This example uses the current general voice-agent model and the marin voice. Keep OPENAI_API_KEY server-only.
Add authentication and rate limiting appropriate for your app before exposing a token endpoint.

Create the adapter

getClientSecret must mint a fresh secret for every start. Do not put a standard OpenAI API key in browser code or a VITE_*/NEXT_PUBLIC_* variable.

State and volume mapping

  • WebRTC negotiation -> connecting
  • input_audio_buffer.speech_started -> listening
  • input_audio_buffer.speech_stopped and response.created -> thinking
  • output audio activity -> speaking
  • completed/interrupted output -> listening
  • Realtime or WebRTC failure -> error
  • explicit stop or closed connection -> idle
The adapter meters the local microphone into inputVolume while listening and the remote WebRTC track into outputVolume while speaking. It also attaches and plays the model audio automatically.

Output calibration

OpenAI Realtime ships with tuned output defaults for noise floor, gain, curve, attack, and release. Most applications should use those defaults. If your audio environment needs different shaping, pass a partial outputVolumeCalibration object:
Pass a getter instead of an object to update calibration while a session is active. The optional onOutputVolumeSample callback reports raw RMS, shaped, and smoothed values for diagnostics.

Runtime overrides

callsUrl, mediaStreamConstraints, getUserMedia, createPeerConnection, fetch, createAudioElement, and createAudioContext are available for custom browser wrappers and tests. Most applications only need getClientSecret.

ChatGPT Live

This adapter targets the public OpenAI Realtime API. It does not wrap consumer ChatGPT voice features that do not expose a corresponding developer API.

Troubleshooting

The browser receives 401 or cannot connect. Verify that the server endpoint creates a fresh short-lived client secret for every start and that the browser sends that value to the adapter. Never substitute a standard API key in client code. The orb connects but no remote audio plays. Check browser autoplay and audio-output behavior, then inspect the WebRTC connection and remote track. The adapter creates and plays the media element unless a custom runtime override replaces that behavior. Listening does not react to speech. Confirm microphone permission and the requested media constraints. Test the raw input diagnostics before changing visual calibration; a silent or wrong input device cannot be fixed with gain. Speaking motion is too quiet or too aggressive. Start with the tuned defaults, inspect samples with onOutputVolumeSample, and change one calibration field at a time. Validate normal speech and silence rather than tuning to a single loud clip.

Production checklist

  • A server-only standard API key creates short-lived client secrets.
  • The token endpoint is authenticated and rate-limited for the application’s threat model.
  • Microphone denial, WebRTC failure, and token failure have distinct recovery messages.
  • A fresh client secret is created after stop, error, or expiration.
  • Start, interruption, stop, and restart are tested on supported browsers.
  • A visible text status and labeled controls accompany the animation.
  • Custom runtime overrides release media tracks, audio contexts, and peer connections.
Last modified on July 22, 2026