> ## Documentation Index
> Fetch the complete documentation index at: https://orb-ui.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenAI Realtime Voice UI for React

> Connect OpenAI's Realtime API to an audio-reactive React voice UI with native WebRTC, short-lived client secrets, and orb-ui.

`createOpenAIRealtimeAdapter` owns the browser voice path: microphone capture, WebRTC negotiation,
remote audio playback, input/output metering, interruption-aware state, and cleanup. Your server
still owns the standard OpenAI API key and creates a short-lived Realtime client secret.

The adapter targets the GA Realtime API. New browser integrations should use
`/v1/realtime/client_secrets` and `/v1/realtime/calls`, not the older beta session flow.

## The simple setup

Your browser provides one function: `getClientSecret`. orb-ui owns everything after that callback,
including microphone permission, WebRTC negotiation, remote playback, conversation state, audio
metering, and cleanup. Most applications do not need any other adapter option.

## Create a client secret on your server

This example uses the current general voice-agent model and the `marin` voice. Keep
`OPENAI_API_KEY` server-only.

```ts theme={null}
export async function POST() {
  const response = await fetch('https://api.openai.com/v1/realtime/client_secrets', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      session: {
        type: 'realtime',
        model: 'gpt-realtime-2.1',
        instructions: 'You are a concise, friendly voice assistant.',
        audio: { output: { voice: 'marin' } },
      },
    }),
  })

  return Response.json(await response.json(), { status: response.status })
}
```

Add authentication and rate limiting appropriate for your app before exposing a token endpoint.

## Create the adapter

```tsx theme={null}
import { Orb } from 'orb-ui'
import { createOpenAIRealtimeAdapter } from 'orb-ui/adapters'

const adapter = createOpenAIRealtimeAdapter({
  getClientSecret: async () => {
    const response = await fetch('/api/openai-realtime-token', { method: 'POST' })
    const data = await response.json()

    if (!response.ok) throw new Error(data.error ?? 'Could not create a Realtime session')
    return data.value
  },
})

export function OpenAIRealtimeVoiceUI() {
  return <Orb adapter={adapter} theme="circle" aria-label="Start OpenAI voice assistant" />
}
```

`getClientSecret` must mint a fresh secret for every start. Do not put a standard OpenAI API
key in browser code or a `VITE_*`/`NEXT_PUBLIC_*` variable.

## State and volume mapping

* WebRTC negotiation -> `connecting`
* `input_audio_buffer.speech_started` -> `listening`
* `input_audio_buffer.speech_stopped` and `response.created` -> `thinking`
* output audio activity -> `speaking`
* completed/interrupted output -> `listening`
* Realtime or WebRTC failure -> `error`
* explicit stop or closed connection -> `idle`

The adapter meters the local microphone into `inputVolume` while listening and the remote WebRTC
track into `outputVolume` while speaking. It also attaches and plays the model audio automatically.

## Output calibration

OpenAI Realtime ships with tuned output defaults for noise floor, gain, curve, attack, and release.
Most applications should use those defaults. If your audio environment needs different shaping,
pass a partial `outputVolumeCalibration` object:

```tsx theme={null}
const adapter = createOpenAIRealtimeAdapter({
  getClientSecret,
  outputVolumeCalibration: {
    release: 0.14,
  },
})
```

Pass a getter instead of an object to update calibration while a session is active. The optional
`onOutputVolumeSample` callback reports raw RMS, shaped, and smoothed values for diagnostics.

## Runtime overrides

`callsUrl`, `mediaStreamConstraints`, `getUserMedia`, `createPeerConnection`, `fetch`,
`createAudioElement`, and `createAudioContext` are available for custom browser wrappers and tests.
Most applications only need `getClientSecret`.

## ChatGPT Live

This adapter targets the public OpenAI Realtime API. It does not wrap consumer ChatGPT voice
features that do not expose a corresponding developer API.

## Troubleshooting

**The browser receives `401` or cannot connect.** Verify that the server endpoint creates a fresh
short-lived client secret for every start and that the browser sends that value to the adapter.
Never substitute a standard API key in client code.

**The orb connects but no remote audio plays.** Check browser autoplay and audio-output behavior,
then inspect the WebRTC connection and remote track. The adapter creates and plays the media element
unless a custom runtime override replaces that behavior.

**Listening does not react to speech.** Confirm microphone permission and the requested media
constraints. Test the raw input diagnostics before changing visual calibration; a silent or wrong
input device cannot be fixed with gain.

**Speaking motion is too quiet or too aggressive.** Start with the tuned defaults, inspect samples
with `onOutputVolumeSample`, and change one calibration field at a time. Validate normal speech and
silence rather than tuning to a single loud clip.

## Production checklist

* A server-only standard API key creates short-lived client secrets.
* The token endpoint is authenticated and rate-limited for the application's threat model.
* Microphone denial, WebRTC failure, and token failure have distinct recovery messages.
* A fresh client secret is created after stop, error, or expiration.
* Start, interruption, stop, and restart are tested on supported browsers.
* A visible text status and labeled controls accompany the animation.
* Custom runtime overrides release media tracks, audio contexts, and peer connections.

## Related

* [Custom integrations](/docs/adapters/custom)
* [React voice agent UI lifecycle guide](/docs/guides/voice-agent-ui)
* [OpenAI Realtime WebRTC guide](https://developers.openai.com/api/docs/guides/realtime-webrtc)