Skip to main content
createOpenAIRealtimeAdapter owns the browser voice path: microphone capture, WebRTC negotiation, remote audio playback, input/output metering, interruption-aware state, and cleanup. Your server still owns the standard OpenAI API key and creates a short-lived Realtime client secret. The adapter targets the GA Realtime API. New browser integrations should use /v1/realtime/client_secrets and /v1/realtime/calls, not the older beta session flow.

The simple setup

Your browser provides one function: getClientSecret. orb-ui owns everything after that callback, including microphone permission, WebRTC negotiation, remote playback, conversation state, audio metering, and cleanup. Most applications do not need any other adapter option.

Create a client secret on your server

This example uses the current general voice-agent model and the marin voice. Keep OPENAI_API_KEY server-only.
Add authentication and rate limiting appropriate for your app before exposing a token endpoint.

Create the adapter

getClientSecret must mint a fresh secret for every start. Do not put a standard OpenAI API key in browser code or a VITE_*/NEXT_PUBLIC_* variable.

State and volume mapping

  • WebRTC negotiation -> connecting
  • input_audio_buffer.speech_started -> listening
  • input_audio_buffer.speech_stopped and response.created -> thinking when no output is playing
  • output_audio_buffer.started -> speaking
  • completed/interrupted output -> listening
  • Realtime or WebRTC failure -> error
  • explicit stop or closed connection -> idle
The adapter meters the local microphone into inputVolume and the remote WebRTC track into outputVolume, keeping both envelopes warm during active conversation states. It also attaches and plays the model audio automatically. WebRTC playback start, stop, and clear events take priority over volume-based state inference. This prevents a listening flash before the first audio packet or a return to speaking while the output envelope fades after playback ends. Detected user speech keeps listening priority during interruption. Audio-level inference remains available when playback events are absent.

Directional calibration

OpenAI Realtime ships with separate input and output defaults that map normal speech to the same portable 0–1 distribution as the other adapters. Most applications should use those defaults. If your audio path needs different amplitude anchors or envelope timing, pass a partial directional override:
Pass a getter instead of an object to update calibration while a session is active. The optional onInputVolumeSample and onOutputVolumeSample callbacks report raw, mapped, and normalized values for diagnostics. See volume calibration for the exact contract and guided profile generator.

Runtime overrides

callsUrl, mediaStreamConstraints, getUserMedia, createPeerConnection, fetch, createAudioElement, and createAudioContext are available for custom browser wrappers and tests. Most applications only need getClientSecret.

ChatGPT Live

This adapter targets the public OpenAI Realtime API. It does not wrap consumer ChatGPT voice features that do not expose a corresponding developer API.

Troubleshooting

The browser receives 401 or cannot connect. Verify that the server endpoint creates a fresh short-lived client secret for every start and that the browser sends that value to the adapter. Never substitute a standard API key in client code. The orb connects but no remote audio plays. Check browser autoplay and audio-output behavior, then inspect the WebRTC connection and remote track. The adapter creates and plays the media element unless a custom runtime override replaces that behavior. Listening does not react to speech. Confirm microphone permission and the requested media constraints. Test the raw input diagnostics before changing calibration; a silent or wrong input device cannot be fixed with amplitude mapping. Speaking motion is too quiet or too aggressive. Start with the shipped defaults and use the guided calibration runner to capture silence, quiet, normal, and energetic speech. Avoid tuning the provider profile to compensate for one theme’s visual range.

Production checklist

  • A server-only standard API key creates short-lived client secrets.
  • The token endpoint is authenticated and rate-limited for the application’s threat model.
  • Microphone denial, WebRTC failure, and token failure have distinct recovery messages.
  • A fresh client secret is created after stop, error, or expiration.
  • Start, interruption, stop, and restart are tested on supported browsers.
  • A visible text status and labeled controls accompany the animation.
  • Custom runtime overrides release media tracks, audio contexts, and peer connections.
Last modified on September 8, 2026