> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orgo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Audio WebSocket

> Stream a computer's speaker over WebSocket as low-latency PCM frames.

Stream real-time audio from a computer's virtual speaker over WebSocket. Audio is captured from the computer's speaker output (PulseAudio on Linux, WASAPI loopback on Windows) and delivered as raw PCM in 20 ms frames. That is low-latency enough for live monitoring.

## Connection URL

```text theme={null}
wss://www.orgo.ai/desktops/{instance_id}/ws/audio?token={password}
```

`{instance_id}` is the computer's `instance_id`, returned by [Create computer](/api-reference/computers/create). It is also the last path segment of the computer's `connection_url`.

### Authentication

Pass the computer's password as the `token` query parameter. Retrieve it from [Get VNC password](/api-reference/computers/vnc-password) before connecting. That endpoint takes the computer's UUID, not its `instance_id`. You need both identifiers: the UUID to fetch the password, the `instance_id` to build this URL.

Send the password in `?token=`, from a server or a browser. An `Authorization` header does not work on this socket, whether it carries an API key or the password: orgo-web accepts it, but it is not forwarded to the computer, which authenticates only `?token=`. The connection opens and then closes with `1000`. An API key in `?token=` fails the same way, and from a browser (any connection with an `Origin` header) it is refused with `4001`. Never put an account key in browser code.

A connection with no usable credential is closed with code `4001`. See [Close codes](#close-codes).

### Query parameters

<ParamField query="token" type="string" required>
  Computer password, from [Get VNC password](/api-reference/computers/vnc-password). Optional only when the request carries a signed-in orgo.ai session cookie, in which case orgo-web supplies the password for you.
</ParamField>

<ParamField query="sample_rate" type="number" default="24000">
  Capture sample rate in Hz. On Linux any positive integer is accepted. On Windows only 8000 to 48000 is accepted. A missing, unparseable, or out-of-range value falls back to 24000.
</ParamField>

<ParamField query="channels" type="number" default="1">
  Number of channels. Only `1` (mono) and `2` (stereo) are accepted; any other value is ignored and the default of 1 is used.
</ParamField>

## Audio format

| Property | Value |
| - | - |
| Encoding | `s16le` (signed 16-bit little-endian PCM) |
| Sample rate | 24,000 Hz by default |
| Channels | 1 (mono) by default |
| Frame duration | 20 ms |
| Frame size | 960 bytes at the defaults |
| Bitrate | \~48 KB/s at the defaults |

Raw PCM: no codec, no container. Frame size follows the settings you request: `sample_rate × channels × 2 × 0.02` bytes.

## Protocol

### Connection flow

1. Client connects with `?token=`.
2. Server validates the credential.
3. Server sends a JSON text frame echoing the configuration actually in use:
   ```json theme={null}
   { "type": "started", "sample_rate": 24000, "channels": 1 }
   ```
4. Server starts capture and sends **binary frames** of raw PCM. If capture fails to start, an `error` text frame arrives instead and the connection closes normally.
5. Client decodes and plays via the Web Audio API or any PCM-capable player.

### Client → server messages

<AccordionGroup>
  <Accordion title="stop" icon="stop">
    Stop capture and close the connection.

    ```json theme={null}
    {
      "type": "stop"
    }
    ```
  </Accordion>

  <Accordion title="ping" icon="heart-pulse">
    Ask for a `pong`.

    ```json theme={null}
    {
      "type": "ping"
    }
    ```
  </Accordion>
</AccordionGroup>

<Note>
  A frame that parses as JSON but carries an unrecognised `type` is discarded silently. On Linux, a frame that is not valid JSON gets an `error` reply with the message `invalid JSON`, and the connection stays open. On Windows it is discarded silently.
</Note>

### Server → client messages

<AccordionGroup>
  <Accordion title="started" icon="play">
    Sent immediately after the connection opens, before capture begins. Echoes the configuration in use, which is your requested values after validation.

    ```json theme={null}
    {
      "type": "started",
      "sample_rate": 24000,
      "channels": 1
    }
    ```

    <ResponseField name="type" type="string">
      Always `"started"`.
    </ResponseField>

    <ResponseField name="sample_rate" type="number">
      Sample rate in Hz.
    </ResponseField>

    <ResponseField name="channels" type="number">
      Number of channels.
    </ResponseField>
  </Accordion>

  <Accordion title="Binary frames" icon="waveform-lines">
    Raw PCM audio data. Each binary frame contains signed 16-bit little-endian samples.

    At the default 24 kHz mono, each 20 ms frame is 960 bytes (480 samples × 2 bytes per sample).
  </Accordion>

  <Accordion title="pong" icon="heart-pulse">
    Response to a `ping`.

    ```json theme={null}
    {
      "type": "pong"
    }
    ```
  </Accordion>

  <Accordion title="error" icon="circle-exclamation">
    Capture could not be started, or (on Linux) a client frame was not valid JSON.

    ```json theme={null}
    {
      "type": "error",
      "message": "failed to start audio capture: starting parec: executable file not found in $PATH"
    }
    ```

    <ResponseField name="type" type="string">
      Always `"error"`.
    </ResponseField>

    <ResponseField name="message" type="string">
      Human-readable error description. `invalid JSON` for an unparseable client frame, on Linux only. When capture could not start, `failed to start audio capture: …` on Linux and `audio capture unavailable: …` on Windows.
    </ResponseField>

    A capture failure is followed immediately by a normal (`1000`) close, not by an error close code. Watch for this frame in your message handler. A client that only inspects close codes sees a clean disconnect and never learns why. An `invalid JSON` reply does not close the connection.
  </Accordion>
</AccordionGroup>

## Examples

<CodeGroup>
  ```javascript JavaScript (Web Audio API) theme={null}
  // COMPUTER_ID is the computer's UUID. INSTANCE_ID is its instance_id, which
  // fills the /desktops/{instance_id} slot. Create computer returns both.
  const computerId = process.env.COMPUTER_ID;
  const instanceId = process.env.INSTANCE_ID;
  const apiKey = process.env.ORGO_API_KEY;

  // Step 1: Get the computer password. This endpoint takes the UUID.
  const res = await fetch(
    `https://www.orgo.ai/api/computers/${computerId}/vnc-password`,
    { headers: { 'Authorization': `Bearer ${apiKey}` } }
  );
  const { password } = await res.json();

  // Step 2: Connect to the audio stream
  const ws = new WebSocket(
    `wss://www.orgo.ai/desktops/${instanceId}/ws/audio?token=${password}`
  );
  ws.binaryType = 'arraybuffer';

  // Step 3: Set up Web Audio API playback
  const ctx = new AudioContext({ sampleRate: 24000 });
  let nextPlayTime = 0;

  ws.onmessage = (event) => {
    // Text frames are JSON control messages
    if (typeof event.data === 'string') {
      const msg = JSON.parse(event.data);
      if (msg.type === 'started') {
        console.log(`Audio stream: ${msg.sample_rate}Hz, ${msg.channels}ch`);
      }
      if (msg.type === 'error') {
        console.error('Audio error:', msg.message);
      }
      return;
    }

    // Binary frames are raw PCM audio
    const pcm = new Int16Array(event.data);
    const floats = new Float32Array(pcm.length);
    for (let i = 0; i < pcm.length; i++) {
      floats[i] = pcm[i] / 32768;
    }

    const buffer = ctx.createBuffer(1, floats.length, 24000);
    buffer.getChannelData(0).set(floats);

    const source = ctx.createBufferSource();
    source.buffer = buffer;
    source.connect(ctx.destination);

    // Schedule with drift correction
    const now = ctx.currentTime;
    if (nextPlayTime < now || nextPlayTime > now + 0.15) {
      nextPlayTime = now + 0.02;
    }
    source.start(nextPlayTime);
    nextPlayTime += buffer.duration;
  };
  ```

  ```python Python theme={null}
  import asyncio
  import websockets
  import json
  import os
  import struct
  import requests

  async def stream_audio(computer_id: str, instance_id: str, api_key: str):
      # Step 1: Get the computer password. This endpoint takes the UUID.
      res = requests.get(
          f"https://www.orgo.ai/api/computers/{computer_id}/vnc-password",
          headers={"Authorization": f"Bearer {api_key}"}
      )
      password = res.json()["password"]

      # Step 2: Connect to the audio stream
      url = f"wss://www.orgo.ai/desktops/{instance_id}/ws/audio?token={password}"

      async with websockets.connect(url) as ws:
          async for message in ws:
              if isinstance(message, str):
                  data = json.loads(message)
                  if data["type"] == "started":
                      print(f"Audio: {data['sample_rate']}Hz, {data['channels']}ch")
                  elif data["type"] == "error":
                      print(f"Error: {data['message']}")
                      break
              else:
                  # Binary frame: raw s16le PCM
                  samples = struct.unpack(f"<{len(message)//2}h", message)
                  # Process samples (save to file, play back, analyze, etc.)
                  print(f"Received {len(samples)} audio samples")

  # COMPUTER_ID is the UUID. INSTANCE_ID fills the /desktops/{instance_id} slot.
  asyncio.run(stream_audio(
      os.environ["COMPUTER_ID"], os.environ["INSTANCE_ID"], os.environ["ORGO_API_KEY"]
  ))
  ```

  ```typescript TypeScript theme={null}
  interface AudioConfig {
    type: 'started';
    sample_rate: number;
    channels: number;
  }

  class AudioStream {
    private ws!: WebSocket;
    private ctx!: AudioContext;
    private nextPlayTime = 0;

    static async connect(computerId: string, instanceId: string, apiKey: string) {
      const stream = new AudioStream();

      // Get the computer password. This endpoint takes the UUID.
      const res = await fetch(
        `https://www.orgo.ai/api/computers/${computerId}/vnc-password`,
        { headers: { 'Authorization': `Bearer ${apiKey}` } }
      );
      const { password } = await res.json();

      // Connect
      stream.ctx = new AudioContext({ sampleRate: 24000 });
      stream.ws = new WebSocket(
        `wss://www.orgo.ai/desktops/${instanceId}/ws/audio?token=${password}`
      );
      stream.ws.binaryType = 'arraybuffer';

      stream.ws.onmessage = (event) => {
        if (typeof event.data === 'string') {
          const msg = JSON.parse(event.data);
          if (msg.type === 'started') {
            console.log(`Audio: ${msg.sample_rate}Hz, ${msg.channels}ch`);
          }
          if (msg.type === 'error') {
            console.error('Audio error:', msg.message);
          }
          return;
        }
        stream.playPCM(event.data);
      };

      return stream;
    }

    private playPCM(data: ArrayBuffer) {
      const pcm = new Int16Array(data);
      const floats = new Float32Array(pcm.length);
      for (let i = 0; i < pcm.length; i++) {
        floats[i] = pcm[i] / 32768;
      }

      const buffer = this.ctx.createBuffer(1, floats.length, 24000);
      buffer.getChannelData(0).set(floats);

      const source = this.ctx.createBufferSource();
      source.buffer = buffer;
      source.connect(this.ctx.destination);

      const now = this.ctx.currentTime;
      if (this.nextPlayTime < now || this.nextPlayTime > now + 0.15) {
        this.nextPlayTime = now + 0.02;
      }
      source.start(this.nextPlayTime);
      this.nextPlayTime += buffer.duration;
    }

    stop() {
      this.ws.send(JSON.stringify({ type: 'stop' }));
      this.ws.close();
      this.ctx.close();
    }
  }

  // COMPUTER_ID is the UUID. INSTANCE_ID fills the /desktops/{instance_id} slot.
  const audio = await AudioStream.connect(
    process.env.COMPUTER_ID!,
    process.env.INSTANCE_ID!,
    process.env.ORGO_API_KEY!
  );

  // Later: stop streaming
  audio.stop();
  ```
</CodeGroup>

## Playback tips

<CardGroup cols={2}>
  <Card title="Browser autoplay" icon="browser">
    Browsers require a user gesture before `AudioContext` can play. Create the context inside a click handler, or call `ctx.resume()` after a user interaction.
  </Card>

  <Card title="Drift correction" icon="clock">
    Schedule buffers slightly ahead of real time and reset when drift exceeds \~150 ms. This prevents gaps and keeps latency low.
  </Card>

  <Card title="Heartbeat" icon="heart-pulse">
    Send a `ping` every 30 seconds to detect a dead connection early. Each message you send, `ping` included, also counts as activity for the idle timeout.
  </Card>

  <Card title="Sample rate" icon="waveform-lines">
    The default 24 kHz mono suits voice and system sounds. Pass `sample_rate=48000` for higher fidelity, and size your playback buffers from the `started` frame rather than assuming the default.
  </Card>
</CardGroup>

<Note>
  Audio streaming requires the computer to be running; a stopped computer is closed with code `4003`. Audio is captured from the computer's speaker output, so any sound the computer produces is streamed: browser media, system alerts, application audio.
</Note>

## Idle disconnect

An idle audio socket is closed with code `4008`. The audio the server streams does not count as activity. Messages you send on this socket do, including `ping`. It also borrows the activity of the other connections you hold to the same computer: a VNC or terminal session you are actively using keeps it open. In production the window is 30 minutes. Reconnect to resume.

## Close codes

| Code | Meaning |
| - | - |
| `1000` | Normal closure. You sent `stop` or disconnected, capture failed to start after an `error` frame, or the computer rejected the credential (see [Authentication](#authentication)). |
| `1001` | orgo-web is draining for a restart or deploy. Reconnect. |
| `1006` | Dropped with no close frame. The server terminates a socket that stops answering its 30-second protocol ping. |
| `4001` | Authentication required. No session cookie and no valid `token`, or a cookie-authenticated browser connecting from a disallowed origin. |
| `4003` | The computer exists but is not running or not ready, or its workspace is not active. Also sent to an open connection when your workspace access changes or a trial ends. Start the computer, then reconnect. |
| `4004` | No computer with that id. |
| `4006` | The API key is valid but scoped to a different workspace. |
| `4007` | MFA is required for this session. Re-challenge, then reconnect. |
| `4008` | Idle timeout. See [Idle disconnect](#idle-disconnect). |
| `4500` | Internal proxy error. |
| `4502` | orgo-web has no route to the computer's host. Retry; if it persists, contact support. |
| `4503` | orgo-web could not look up the computer. Retry. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.