> ## Documentation Index
> Fetch the complete documentation index at: https://developer.sanas.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Stream Message Layer

> The shared application protocol for the WebSocket Stream API and the WebRTC API

The Stream message layer defines the application protocol shared by the WebSocket Stream API and the WebRTC API. It covers session setup, pipeline configuration, turn finalization, transcription and translation events, lifecycle events, errors, and usage modes.

Transport mapping:

| API                                                              | JSON messages                               | Audio                                    |
| ---------------------------------------------------------------- | ------------------------------------------- | ---------------------------------------- |
| [WebSocket Stream API](/Docs/Language-Translation/WebSocket-API) | WebSocket text frames                       | Raw PCM bytes in WebSocket binary frames |
| [WebRTC API](/Docs/Language-Translation/WebRTC-API)              | Stringified JSON on the WebRTC data channel | WebRTC audio media tracks                |

Do not send audio in JSON messages. The `init`, `configure`, `flush`, and server event shapes are the same across transports.

## Protocol Overview

### Connection Lifecycle

```mermaid theme={null}
sequenceDiagram
    autonumber
    participant Client
    participant Server

    Client->>Server: Open authenticated Stream transport
    Client->>Server: Send init with session parameters
    Client->>Server: Send configure with languages and voice
    Server-->>Client: Send configured
    Client->>Server: Stream input audio over transport
    Server-->>Client: Send transcription events
    Server-->>Client: Send translation events
    Server-->>Client: Stream output audio over transport
    Server-->>Client: Send output boundary and lifecycle events
    alt Manual turn finalization
        Client->>Server: Send flush
    else Automatic turn finalization
        Server-->>Client: Detect end of speech with VAD
    end
    Server-->>Client: Send remaining results and flushed
    opt Start another turn
        Client->>Server: Send new configure
    end
    opt End session
        Client->>Server: Close transport
    end
```

### Key Concepts

* **Init**: Sent once per connection. Sets session-level parameters (sample rates, conversation ID).
* **Configure**: Sent after init and again at any point to change languages, voice, or glossary. Each configure starts a new pipeline configuration. The server responds with `configured` when ready.
* **Flush**: Signals the server to finalize the current input and produce all remaining output. The server responds with `flushed` when complete. Use this for manual turn-taking or to force processing of buffered audio.
* **Utterance**: A continuous segment of speech. The server assigns an incrementing `utterance_idx` to each detected utterance. When VAD (Voice Activity Detection) is active, the server automatically splits speech into utterances.

***

## Client Messages (Client -> Server)

Client control messages are JSON objects with a `"type"` field. Send them as WebSocket text frames on the WebSocket transport, or as stringified JSON messages on the WebRTC data channel. Audio transport is described in [Input Audio](#input-audio).

### `init`

**Required as the first message on every connection.** Sets connection-level parameters that remain fixed for the session lifetime.

```json theme={null}
{
  "type": "init",
  "conversation_id": "optional-id",
  "session_name": "my-session",
  "input_sample_rate": 16000,
  "output_sample_rate": 16000,
  "realtime_playback": true
}
```

| Field                | Type    | Required | Default             | Description                                                                                                                                                                                                                                                  |
| -------------------- | ------- | -------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `type`               | string  | Yes      | --                  | Must be `"init"`                                                                                                                                                                                                                                             |
| `conversation_id`    | string  | No       | auto-generated UUID | Identifier to group related sessions (e.g., a multi-turn conversation). If empty or omitted, the server auto-generates a UUID.                                                                                                                               |
| `session_name`       | string  | No       | `""`                | Human-readable label for the session                                                                                                                                                                                                                         |
| `input_sample_rate`  | integer | No       | `16000`             | Sample rate of input audio in Hz. Allowed: `16000`                                                                                                                                                                                                           |
| `output_sample_rate` | integer | No       | `16000`             | Desired sample rate of output audio in Hz. Allowed: `16000`                                                                                                                                                                                                  |
| `realtime_playback`  | boolean | No       | `false`             | Set to `true` if you intend to play output audio in real-time as it streams. This enables server-side pacing optimizations for simultaneous mode. Set to `false` for consecutive mode where the full translation audio is played after the speaker finishes. |

### `configure`

Initializes or reconfigures the translation pipeline. Must be sent after `init`. Can be sent again mid-session to change languages, voice, glossary, or features without reconnecting.

```json theme={null}
{
  "type": "configure",
  "language_routes": [
    {"lang_in": "en-US", "lang_out": "es-ES"}
  ],
  "features": [],
  "voice_id": null,
  "glossary": [
    {"terms": {"en-US": "Sanas", "es-ES": "Sanas"}}
  ],
  "request_id": "optional-correlation-id"
}
```

| Field             | Type           | Required | Default | Description                                                                                  |
| ----------------- | -------------- | -------- | ------- | -------------------------------------------------------------------------------------------- |
| `type`            | string         | Yes      | --      | Must be `"configure"`                                                                        |
| `language_routes` | array          | No       | `[]`    | Array of language route objects (see below)                                                  |
| `features`        | array          | No       | `[]`    | Pipeline features to enable (see [Features](#features))                                      |
| `voice_id`        | string or null | No       | `null`  | Custom voice ID for speech synthesis. If null, uses a default voice for the target language. |
| `glossary`        | array          | No       | `[]`    | Array of glossary entry objects (see below)                                                  |
| `request_id`      | string or null | No       | `null`  | Client-specified ID echoed back in the `configured` response for correlation                 |

The server responds with a [`configured`](#configured) message when the pipeline is ready to accept audio.

#### Language Routes

Language routes define the translation directions the pipeline supports. Each route specifies a source and target language.

```json theme={null}
{"lang_in": "en-US", "lang_out": "es-ES"}
```

| Field      | Type   | Description                                                          |
| ---------- | ------ | -------------------------------------------------------------------- |
| `lang_in`  | string | Source language code (BCP-47, e.g., `"en-US"`, `"es-ES"`, `"fr-FR"`) |
| `lang_out` | string | Target language code                                                 |

**Single route** -- translates one direction only:

```json theme={null}
"language_routes": [
  {"lang_in": "en-US", "lang_out": "es-ES"}
]
```

**Bidirectional routes** -- the server detects which language is being spoken and translates to the other. Use this when both parties may speak either language (replaces the old `can_lang_swap` behavior):

```json theme={null}
"language_routes": [
  {"lang_in": "en-US", "lang_out": "es-ES"},
  {"lang_in": "es-ES", "lang_out": "en-US"}
]
```

When multiple routes are provided, the server performs language identification on the incoming audio and selects the matching route. You will receive a [`language_route`](#language_route) event indicating which route was chosen for each utterance.

**Auto-detect mode (any language to one target)** -- use `"*"` as `lang_in` to accept any supported language and translate it to a fixed target. The server identifies the spoken language automatically and translates to the specified output language:

```json theme={null}
"language_routes": [
  {"lang_in": "*", "lang_out": "en-US"}
]
```

This translates any incoming language to English. The server will send a [`language_route`](#language_route) event with the detected language filled in:

```json theme={null}
{"type": "language_route", "utterance_idx": 0, "lang_in": "ja-JP", "lang_out": "en-US", "is_final": true}
```

In auto-detect modes, the server may send tentative `language_route` messages (`is_final: false`) while language identification is still refining, followed by a single `is_final: true` message once the route is locked in. Clients that only care about the final answer can ignore tentative messages and act on `is_final: true`.

You can combine auto-detect with an explicit route. For example, to translate English to Spanish and auto-detect everything else to English:

```json theme={null}
"language_routes": [
  {"lang_in": "en-US", "lang_out": "es-ES"},
  {"lang_in": "*", "lang_out": "en-US"}
]
```

NOTE: When multiple language routes are specified, they will be matched in list order. The first route to match wins. In the following language route configuration, the server will **always** choose the first route because `*` matches any detected language. In this case, the consequence is: English speech will remain as English.

```json theme={null}
"language_routes": [
  {"lang_in": "*", "lang_out": "en-US"}
  {"lang_in": "en-US", "lang_out": "es-ES"},
]
```

**Wildcard catch-all route** -- add a `"*"` -> `"*"` route alongside your normal routes to handle speech in unexpected languages gracefully. When the detected language does not match any explicit route, the wildcard route is selected and the pipeline enters a **transcription-only mode** (no translation or TTS output). This is useful for prompting the user to switch languages:

```json theme={null}
"language_routes": [
  {"lang_in": "en-US", "lang_out": "es-ES"},
  {"lang_in": "es-ES", "lang_out": "en-US"},
  {"lang_in": "*", "lang_out": "*"}
]
```

When the wildcard catch-all route is triggered, you will receive a `language_route` event with the detected language echoed into **both** `lang_in` and `lang_out`. For the configuration above, if the user speaks French you will see:

```json theme={null}
{"type": "language_route", "utterance_idx": 1, "lang_in": "fr-FR", "lang_out": "fr-FR", "is_final": true}
```

Note that the server then emits `transcription` / `transcription_ended` for the utterance but no `translation`, `translation_ended`, `output_audio`, `output_speech_started`, or `output_speech_ended` (transcription-only mode).

You can use this `language_route` to show a prompt to the user, for example: *"It seems like you are speaking French. Would you like to switch your language?"* If the user confirms, send a new `configure` with the appropriate routes.

#### Glossary Entries

Glossary entries either (a) keep a phrase verbatim across languages or (b) lock in a specific translation between two languages. Each entry is a map of language codes to the desired term in that language.

**Keep a phrase verbatim across languages — use `"*"`.** For brand names, product names, and other proper nouns that should never be translated, use the `"*"` wildcard:

```json theme={null}
{"glossary": [
  {"terms": {"*": "Sanas"}},
  {"terms": {"*": "ACME"}}
]}
```

This tells the pipeline that "Sanas" should remain "Sanas" in every language, and "ACME" should remain "ACME" in every language.

**Lock in a specific translation between 2+ languages — use language codes.** To force a phrase in one language to a specific phrase in another, list each phrase by its language code:

```json theme={null}
{
  "terms": {
    "es": "servicio al cliente",
    "fr": "service client",
    "en": "customer service"
  }
}
```

Here, "servicio al cliente" spoken in Spanish becomes "service client" when translating to French, and "service client" spoken in French becomes "servicio al cliente" when translating to Spanish. A glossary entry only takes effect on a language route when **both** the input and output languages are specified. If either side isn't covered, the entry is ignored for that route.

Note that using a short code like `es` will match all variants of Spanish (`es-419` and `es-ES`). You are also free to specify a long code (`es-419`) if you need more fine-grained tuning.

**Combining both.** You can mix `"*"` with explicit language codes in the same entry. The `"*"` then acts as the term used whenever a route's input or output language isn't explicitly listed. For example, `{"terms": {"*": "customer service", "es": "servicio al cliente", "fr": "service client"}}` resolves Spanish → French to "service client" via the explicit codes, while Spanish → Russian falls back to "customer service" via the wildcard.

#### Features

The `features` array controls which pipeline mode is used. By default, you can leave the `features` array empty or omitted. Note that feature names are case-sensitive:

| Feature                       | Description                                                                                                                                                                                                                                                                                                         |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `"consecutive"`               | By default, **simultaneous mode** is used, producing streaming translation and audio output in real-time as the speaker talks. This feature enables consecutive translation mode. The pipeline waits for the speaker to finish (via VAD or manual flush), then produces the full translation and synthesized audio. |
| `"detect_languages_once"`     | By default, when `language_routes` actually require detection (any `"*"` `lang_in`, or multiple routes), every utterance redetects the language. This feature locks the session to only detect language a single time.                                                                                              |
| `"hide_output_text_boundary"` | Suppresses [`output_text_boundary`](#output_text_boundary) emission. By default the server aligns TTS word timestamps back to the transcription/translation and emits boundary messages, suitable for word-level highlighting. Set this feature to skip all alignment work.                                         |
| `"skip_logging"`              | If not already, opts this session out of server-side recording. Once set on any `configure` in the session, it sticks for the lifetime of the stream.                                                                                                                                                               |

### Input Audio

Streams audio data to the server after receiving `configured`.

Audio transport depends on the API you are using:

| Transport            | Input audio behavior                                                                          |
| -------------------- | --------------------------------------------------------------------------------------------- |
| WebSocket Stream API | Send raw PCM bytes as Binary WebSocket frames. Do not wrap audio in JSON.                     |
| WebRTC API           | Send microphone audio on the WebRTC audio media track. Do not send audio on the data channel. |

For WebSocket binary audio, use this format:

* Encoding: 16-bit signed integer PCM (little-endian)
* Channels: Mono (1 channel)
* Sample rate: Must match `input_sample_rate` from `init`

### `flush`

Signals the server to finalize processing of all buffered audio and produce remaining results. The server will respond with a [`flushed`](#flushed) message after all output has been sent.

```json theme={null}
{
  "type": "flush",
  "request_id": "optional-correlation-id"
}
```

| Field        | Type           | Required | Default | Description                                                               |
| ------------ | -------------- | -------- | ------- | ------------------------------------------------------------------------- |
| `type`       | string         | Yes      | --      | Must be `"flush"`                                                         |
| `request_id` | string or null | No       | `null`  | Client-specified ID echoed back in the `flushed` response for correlation |

***

## Server Messages (Server -> Client)

Server control and text messages are JSON objects with a `"type"` field. Receive them as WebSocket text frames on the WebSocket transport, or as stringified JSON messages on the WebRTC data channel. Audio transport is described in [Output Audio](#output-audio).

**Note on optional fields:** Fields documented as "or null" (such as `request_id`, `probability`, `utterance_idx` on errors) are **omitted from the JSON** when not present, rather than sent as `"field": null`. Your client should treat a missing key the same as a null value.

### `configured`

Confirms the pipeline is ready to accept audio after a `configure` message.

```json theme={null}
{
  "type": "configured",
  "request_id": "r1"
}
```

| Field        | Type           | Description                                                                     |
| ------------ | -------------- | ------------------------------------------------------------------------------- |
| `type`       | string         | `"configured"`                                                                  |
| `request_id` | string or null | Echoes the `request_id` from the corresponding `configure` message, if provided |

### `input_speech_started`

Indicates the server's VAD has detected the start of speech in the input audio.

```json theme={null}
{
  "type": "input_speech_started",
  "utterance_idx": 0,
  "input_time": 0.5
}
```

| Field           | Type    | Description                                                                                     |
| --------------- | ------- | ----------------------------------------------------------------------------------------------- |
| `utterance_idx` | integer | Index of the utterance that started                                                             |
| `input_time`    | float   | Timestamp in seconds (relative to the start of the input audio stream) when speech was detected |

### `language_route`

Indicates which language route was selected for the current utterance. Sent when the system identifies the spoken language, particularly useful when multiple language routes are configured.

```json theme={null}
{
  "type": "language_route",
  "utterance_idx": 0,
  "lang_in": "en-US",
  "lang_out": "es-ES",
  "is_final": true
}
```

| Field           | Type    | Description                                               |
| --------------- | ------- | --------------------------------------------------------- |
| `utterance_idx` | integer | Index of the utterance this route applies to              |
| `lang_in`       | string  | Detected/confirmed input language                         |
| `lang_out`      | string  | Target output language                                    |
| `is_final`      | boolean | `true` when this is the locked-in route for the utterance |

### `transcription`

Real-time transcription of the input audio. Complete words have been finalized and will not change. Partial words are in-progress and may be updated in subsequent messages.

```json theme={null}
{
  "type": "transcription",
  "utterance_idx": 0,
  "complete": [
    {"word": "Hello ", "start": 0.0, "end": 0.5, "probability": 0.98}
  ],
  "partial": [
    {"word": "world", "start": 0.5, "end": 0.8, "probability": 0.85}
  ]
}
```

| Field           | Type    | Description                                                                                       |
| --------------- | ------- | ------------------------------------------------------------------------------------------------- |
| `utterance_idx` | integer | Index of the utterance this transcription belongs to                                              |
| `complete`      | Word\[] | Finalized words -- will not change and are sent only once. Store these on the client for display. |
| `partial`       | Word\[] | In-progress words -- may be updated or replaced in subsequent messages                            |

**Word object:**

| Field         | Type          | Description                                                                    |
| ------------- | ------------- | ------------------------------------------------------------------------------ |
| `word`        | string        | The transcribed word (may include trailing whitespace)                         |
| `start`       | float         | Start time in seconds relative to the beginning of the utterance's input audio |
| `end`         | float         | End time in seconds                                                            |
| `probability` | float or null | Confidence score (0.0 to 1.0), or null if unavailable                          |

### `transcription_ended`

Indicates that all transcription for a given utterance is complete. No further `transcription` messages will be sent for this utterance.

```json theme={null}
{
  "type": "transcription_ended",
  "utterance_idx": 0
}
```

### `translation`

Translated text derived from the transcription. Like transcription, includes complete (finalized) and partial (in-progress) words.

```json theme={null}
{
  "type": "translation",
  "utterance_idx": 0,
  "complete": [
    {"word": "Hola ", "start": 0.0, "end": 0.5, "probability": 0.95}
  ],
  "partial": [
    {"word": "mundo", "start": 0.5, "end": 0.8, "probability": 0.85}
  ]
}
```

| Field           | Type    | Description                                        |
| --------------- | ------- | -------------------------------------------------- |
| `utterance_idx` | integer | Index of the utterance this translation belongs to |
| `complete`      | Word\[] | Finalized translated words                         |
| `partial`       | Word\[] | In-progress translated words                       |

### `translation_ended`

Indicates that all translation for a given utterance is complete.

```json theme={null}
{
  "type": "translation_ended",
  "utterance_idx": 0
}
```

### `input_speech_ended`

Indicates the server's VAD has detected the end of speech in the input audio. After this, the server will not accept further input audio for this utterance and will finalize the translation.

```json theme={null}
{
  "type": "input_speech_ended",
  "utterance_idx": 0,
  "input_time": 3.2
}
```

| Field           | Type    | Description                                          |
| --------------- | ------- | ---------------------------------------------------- |
| `utterance_idx` | integer | Index of the utterance that ended                    |
| `input_time`    | float   | Timestamp in seconds when end-of-speech was detected |

### `output_speech_started`

Indicates the server has begun sending synthesized output audio for an utterance.

```json theme={null}
{
  "type": "output_speech_started",
  "utterance_idx": 0,
  "output_time": 0.0
}
```

| Field           | Type    | Description                                    |
| --------------- | ------- | ---------------------------------------------- |
| `utterance_idx` | integer | Index of the utterance                         |
| `output_time`   | float   | Timestamp in the output audio stream (seconds) |

### Output Audio

Synthesized audio from the translated text.

Audio transport depends on the API you are using:

| Transport            | Output audio behavior                                                                            |
| -------------------- | ------------------------------------------------------------------------------------------------ |
| WebSocket Stream API | Receive raw PCM bytes as Binary WebSocket frames. Audio is not wrapped in JSON.                  |
| WebRTC API           | Receive translated audio on the WebRTC audio media track. Audio is not sent on the data channel. |

For WebSocket binary audio, the format is:

* Encoding: 16-bit signed integer PCM (little-endian)
* Channels: Mono (1 channel)
* Sample rate: Matches `output_sample_rate` specified in `init`

### `output_text_boundary`

Timing information that maps a position in the output audio stream back to positions in the transcription and translation text. Use this to synchronize text highlighting with audio playback.

Sent by default. Clients that don't need word-level highlighting can opt out by setting the [`"hide_output_text_boundary"`](#features) feature in `configure`, which suppresses these messages.

```json theme={null}
{
  "type": "output_text_boundary",
  "utterance_idx": 0,
  "output_time": 1.5,
  "transcription": {"word_idx": 3, "char_idx": 15},
  "translation": {"word_idx": 2, "char_idx": 12}
}
```

| Field           | Type                 | Description                                                                                                                                                            |
| --------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `utterance_idx` | integer              | Index of the utterance                                                                                                                                                 |
| `output_time`   | float                | Timestamp in seconds within the output audio stream                                                                                                                    |
| `transcription` | TextPosition or null | Position in the transcription text that aligns with the output audio at `output_time`                                                                                  |
| `translation`   | TextPosition or null | Position in the translation text that aligns with the output audio at `output_time`. All translation before this position has already been spoken in the output audio. |

**TextPosition object:**

| Field      | Type    | Description                                        |
| ---------- | ------- | -------------------------------------------------- |
| `word_idx` | integer | Index of the word within the utterance's word list |
| `char_idx` | integer | Character offset within the word                   |

### `output_speech_ended`

Indicates all synthesized audio for an utterance has been sent.

```json theme={null}
{
  "type": "output_speech_ended",
  "utterance_idx": 0,
  "output_time": 4.2
}
```

| Field           | Type    | Description                         |
| --------------- | ------- | ----------------------------------- |
| `utterance_idx` | integer | Index of the utterance              |
| `output_time`   | float   | Final timestamp of the output audio |

### `flushed`

Confirms that a `flush` has been fully processed and all pending output has been sent.

```json theme={null}
{
  "type": "flushed",
  "request_id": "f1"
}
```

| Field        | Type           | Description                                                   |
| ------------ | -------------- | ------------------------------------------------------------- |
| `request_id` | string or null | Echoes the `request_id` from the `flush` message, if provided |

### `error`

Indicates an error occurred during processing.

```json theme={null}
{
  "type": "error",
  "message": "Invalid config message",
  "code": "CORE_ERROR",
  "utterance_idx": 3,
  "request_id": "r1"
}
```

| Field           | Type            | Description                                                                                                     |
| --------------- | --------------- | --------------------------------------------------------------------------------------------------------------- |
| `message`       | string          | Human-readable error description                                                                                |
| `code`          | string          | Error code identifier (e.g., `"STREAM_ERROR"` from the proxy, or an application-specific code from the backend) |
| `utterance_idx` | integer or null | Utterance associated with the error, if applicable                                                              |
| `request_id`    | string or null  | Associated request ID, if applicable                                                                            |

Non-fatal errors (e.g., malformed JSON, unknown message type) are sent as error messages with code `"STREAM_ERROR"` but the connection remains open. You can continue sending messages after receiving an error. Errors originating from the backend translation pipeline will have their own error codes.

***

## Examples

### Simultaneous Mode (default)

In simultaneous mode, the server produces translation and audio output in real-time as the speaker talks, with minimal delay.

**Setup:**

```json theme={null}
{"type": "init", "realtime_playback": true}

{"type": "configure",
 "language_routes": [{"lang_in": "en-US", "lang_out": "es-ES"}]}
```

**Flow:**

1. Send `init` with `"realtime_playback": true`, then `configure`
2. Wait for `configured`
3. Stream input audio over the selected transport continuously
4. Receive streaming `transcription`, `translation`, and output audio in real time
5. The server uses VAD to segment utterances automatically; each utterance gets its own `utterance_idx`
6. Use `flush` when you want to force-finalize any pending output

### Consecutive Mode with VAD (Automatic End-of-Speech)

In this mode, the speaker talks, the system detects when they stop, and then produces the full translation. This is the simplest approach for turn-based conversations.

**Setup:**

```json theme={null}
{"type": "init"}

{"type": "configure",
 "language_routes": [{"lang_in": "en-US", "lang_out": "es-ES"}],
 "features": ["consecutive"]}
```

**Flow:**

1. Send `init`, then `configure` with `"features": ["consecutive"]`
2. Wait for `configured`
3. Stream input audio over the selected transport as the speaker talks
4. The server's VAD detects end-of-speech and sends `input_speech_ended`
5. Receive `transcription`, `translation`, and output audio
6. Receive `output_speech_ended` when all audio for the utterance has been sent
7. For a multi-turn conversation, send a new `configure` to start the next turn (you can swap languages), or just keep streaming audio for the next utterance

**Example message flow:**

```
Client: {"type": "init", "input_sample_rate": 16000, "output_sample_rate": 24000}
Client: {"type": "configure", "language_routes": [{"lang_in": "en-US", "lang_out": "es-ES"}], "features": ["consecutive"], "request_id": "r1"}
Server: {"type": "configured", "request_id": "r1"}
Client: [input audio over transport]  (repeated)
Server: {"type": "input_speech_started", "utterance_idx": 0, "input_time": 0.3}
Server: {"type": "transcription", "utterance_idx": 0, "complete": [...], "partial": [...]}
Server: {"type": "input_speech_ended", "utterance_idx": 0, "input_time": 2.8}
Server: {"type": "language_route", "utterance_idx": 0, "lang_in": "en-US", "lang_out": "es-ES", "is_final": true}
Server: {"type": "transcription", "utterance_idx": 0, "complete": [...], "partial": []}
Server: {"type": "transcription_ended", "utterance_idx": 0}
Server: {"type": "translation", "utterance_idx": 0, "complete": [...], "partial": []}
Server: {"type": "translation_ended", "utterance_idx": 0}
Server: {"type": "output_speech_started", "utterance_idx": 0, "output_time": 0.0}
Server: [output audio over transport]  (repeated)
Server: {"type": "output_text_boundary", ...}
Server: {"type": "output_speech_ended", "utterance_idx": 0, "output_time": 3.5}
```

### Consecutive Mode with Manual Flush

Use manual flush when you want explicit control over when the system processes the audio (e.g., push-to-talk with a button release, or after a fixed recording duration).

**Setup:**

```json theme={null}
{"type": "init"}

{"type": "configure",
 "language_routes": [{"lang_in": "en-US", "lang_out": "es-ES"}],
 "features": ["consecutive"]}
```

**Flow:**

1. Send `init`, then `configure` with `"features": ["consecutive"]`
2. Wait for `configured`
3. Stream input audio over the selected transport while the user is speaking
4. When the user finishes (e.g., releases push-to-talk button), send `flush`
5. Receive transcription, translation, and audio output
6. Receive `flushed` when all output is complete
7. To start the next turn, simply resume sending audio (or send a new `configure` to change languages)

**Example message flow:**

```
Client: {"type": "init", "input_sample_rate": 16000, "output_sample_rate": 24000}
Client: {"type": "configure", "language_routes": [{"lang_in": "en-US", "lang_out": "es-ES"}], "features": ["consecutive"], "request_id": "r1"}
Server: {"type": "configured", "request_id": "r1"}
Client: [input audio over transport]  (repeated)
Client: {"type": "flush", "request_id": "f1"}
Server: {"type": "transcription", "utterance_idx": 0, "complete": [...], "partial": []}
Server: {"type": "translation", "utterance_idx": 0, "complete": [...], "partial": []}
Server: [output audio over transport]  (repeated)
Server: {"type": "flushed", "request_id": "f1"}
```

**Multiple turns on the same connection:**

After receiving `flushed`, you can immediately stream more audio and send another `flush` for the next turn. You do not need to re-send `init` or `configure` (unless you want to change settings). This avoids reconnection overhead.

```
(... first turn completed with flushed ...)
Client: [input audio over transport]  (next turn)
Client: {"type": "flush", "request_id": "f2"}
Server: {"type": "transcription", ...}
Server: {"type": "translation", ...}
Server: [output audio over transport]
Server: {"type": "flushed", "request_id": "f2"}
```

### Auto-Detect Mode (Any Language to One Target)

Use a wildcard `"*"` as `lang_in` to automatically detect the spoken language and translate it to a fixed target. This is ideal for scenarios like a help desk where callers may speak any language and all input should be translated to English:

**Setup:**

```json theme={null}
{"type": "init"}

{"type": "configure",
 "language_routes": [{"lang_in": "*", "lang_out": "en-US"}],
 "features": ["consecutive"],
 "request_id": "r1"}
```

**Flow:**

1. Send `init`, then `configure` with a wildcard `lang_in`
2. Wait for `configured`
3. Stream input audio over the selected transport -- the speaker can talk in any supported language
4. The server identifies the language and sends a `language_route` event with the detected language. You may receive intermediate `is_final: false` messages while detection is still refining, followed by the final `is_final: true` message once the route is locked:

   ```json theme={null}
   {"type": "language_route", "utterance_idx": 0, "lang_in": "ko-KR", "lang_out": "en-US", "is_final": true}
   ```
5. Receive transcription (in the original language), translation (in English), and English output audio
6. Display the detected language to the user so they know it was recognized correctly

You can also combine auto-detect with explicit routes:

```json theme={null}
{"type": "configure",
 "language_routes": [
   {"lang_in": "en-US", "lang_out": "es-ES"},
   {"lang_in": "*", "lang_out": "en-US"}
 ],
 "features": ["consecutive"]}
```

In this example, English input is translated to Spanish. Any other language is translated to English.

### Bidirectional Language Detection

For two-way conversations where either party may speak either language:

```json theme={null}
{"type": "configure",
 "language_routes": [
   {"lang_in": "en-US", "lang_out": "es-ES"},
   {"lang_in": "es-ES", "lang_out": "en-US"}
 ],
 "features": ["consecutive"]}
```

The server automatically detects which language is being spoken and routes to the appropriate translation direction. You will receive a `language_route` message for each utterance indicating which route was chosen.

### Mid-Session Reconfiguration

You can change languages, voice, glossary, or features at any time by sending a new `configure` message. This is useful for:

* Swapping speaker/language direction between turns
* Changing the output voice
* Updating the glossary

The server will tear down the current pipeline and build a new one. Wait for `configured` before sending new audio.

```
Client: {"type": "configure", "language_routes": [{"lang_in": "es-ES", "lang_out": "en-US"}], "features": ["consecutive"], "request_id": "r2"}
Server: {"type": "configured", "request_id": "r2"}
Client: [input audio over transport]
```

***

## Supported Languages

Language codes use BCP-47 format (e.g., `"en-US"`, `"es-ES"`, `"fr-FR"`). To retrieve the current list of supported languages, use the Gateway languages API:

```
POST https://api.sanaslt.com/v2/languages/list
```

***

## Error Handling

### Connection Errors

Authentication is handled by the transport before the message layer starts. If authentication fails, the server returns an HTTP error before accepting the WebSocket connection:

| HTTP Status | Description                                                    |
| ----------- | -------------------------------------------------------------- |
| 401         | Missing, empty, invalid, or expired authentication credentials |

### Runtime Errors

During a session, errors are delivered as `error` messages over the active Stream transport. Non-fatal errors (bad JSON, unknown message type) do not close the connection -- you can continue sending messages.

Fatal errors (backend connection failure, gRPC stream errors) may be followed by the connection closing.

### Inactivity Timeout

If no messages or media are sent or received for 60 seconds (configurable), the server closes the transport session.

***

## Best Practices

* **Audio chunking**: Send audio in small chunks (10-20 ms of audio per chunk) for the lowest latency. Larger chunks add latency equal to the chunk duration.
* **Real-time pacing**: When streaming from a live microphone, audio is naturally paced. When streaming from a file, pace sends to match real-time (e.g., sleep 20 ms between 20 ms chunks) for optimal results. Set `realtime_playback: true` in init when playing audio back in real-time.
* **Handle partial results**: Display partial transcriptions and translations for a responsive UX, but only persist complete words. Complete words will not change and are sent only once.
* **Buffer output audio**: Accumulate a small buffer of output audio before starting playback to prevent stuttering. Most client-side audio playback libraries handle this automatically.
* **Use `request_id`**: Include `request_id` in `configure` and `flush` messages to correlate responses, especially when sending multiple operations.
* **Reconnection logic**: Implement exponential backoff for connection failures.
* **Graceful shutdown**: Send `flush` before closing the connection to ensure all pending output is received.
* **Reuse sessions**: For multi-turn conversations, reuse the same transport session and send new `configure` messages between turns rather than reconnecting.

***

## Migration Guide: v2 Consecutive API to v3 Stream API

This guide helps you migrate from the v2 `/v2/consecutive` WebSocket endpoint to the v3 `/v3/stream` WebSocket endpoint. The message shape changes also apply to clients using the WebRTC data channel for Stream v3 messages.

### Endpoint Change

|              | v2                                               | v3                                               |
| ------------ | ------------------------------------------------ | ------------------------------------------------ |
| URL          | `wss://api.sanaslt.com/v2/consecutive`           | `wss://api.sanaslt.com/v3/stream`                |
| Auth         | Same query params (`token`, `api_key`)           | Same query params (`token`, `api_key`)           |
| Auth failure | WebSocket accepted, then closed with code `1008` | HTTP `401` returned **before** WebSocket upgrade |

**Important:** In v2, authentication failures happened after the WebSocket was established (close code 1008). In v3, authentication is validated before the upgrade, so failures result in a standard HTTP 401 response. Update your client error handling accordingly -- you will receive an HTTP error, not a WebSocket close frame.

### Protocol Changes

The biggest change is that v3 separates session initialization into two explicit steps (`init` + `configure`) instead of a single `config` message.

#### Initialization: `config` -> `init` + `configure`

**v2: Single config message**

```json theme={null}
{
  "type": "config",
  "lang_in": "en-US",
  "lang_out": "es-ES",
  "input_sample_rate": 16000,
  "output_sample_rate": 24000,
  "glossary": ["Sanas", "ACME"],
  "can_lang_swap": false
}
```

**v3: Two messages (init + configure)**

```json theme={null}
{
  "type": "init",
  "input_sample_rate": 16000,
  "output_sample_rate": 24000
}
```

```json theme={null}
{
  "type": "configure",
  "language_routes": [
    {"lang_in": "en-US", "lang_out": "es-ES"}
  ],
  "features": ["consecutive"],
  "glossary": [
    {"terms": {"*": "Sanas"}},
    {"terms": {"*": "ACME"}}
  ],
  "request_id": "r1"
}
```

Key differences:

* Sample rates moved to `init` (sent once per connection)
* `lang_in`/`lang_out` replaced by the `language_routes` array
* `can_lang_swap: true` replaced by adding a second route with swapped languages
* `glossary` changed from a flat string array to an array of `{"terms": {...}}` objects. Use `"*"` as the language key to apply a term across all languages (recommended default), and override specific languages as needed.
* You must include `"features": ["consecutive"]` to get consecutive mode behavior
* `voice_id` remains available on `configure` (same as v2 `config`)

#### Ready Signal: `ready` -> `configured`

**v2:**

```json theme={null}
{"type": "ready", "session_id": "abc123"}
```

**v3:**

```json theme={null}
{"type": "configured", "request_id": "r1"}
```

The `session_id` field is no longer provided in the ready response. If you need a session identifier, use the `conversation_id` from `init`.

#### Audio Messages: JSON `audio` -> Binary WebSocket frames

**v2 (input):**

```json theme={null}
{"type": "audio", "data": "<base64>"}
```

**v3 (input):** Raw PCM bytes sent as a WebSocket Binary frame (no JSON wrapper, no base64 encoding).

**v2 (output):**

```json theme={null}
{"type": "audio", "data": "<base64>"}
```

**v3 (output):** Raw PCM bytes received as a WebSocket Binary frame (no JSON wrapper, no base64 encoding).

In v2, both input and output audio used JSON text frames with base64-encoded `data`. In v3, audio is sent and received as raw binary WebSocket frames, eliminating the \~33% base64 overhead. All other messages remain JSON text frames.

#### Transcription and Translation: added `utterance_idx`

v2 `transcription` and `translation` messages contained only `complete` and `partial` word arrays. v3 adds an `utterance_idx` field to each message so you can associate results with a specific utterance when multiple utterances are in flight.

**v2:**

```json theme={null}
{"type": "transcription", "complete": [...], "partial": [...]}
```

**v3:**

```json theme={null}
{"type": "transcription", "utterance_idx": 0, "complete": [...], "partial": [...]}
```

#### End-of-Speech: `stop`/`flush` / `speech_stop` -> `flush` / `flushed` + lifecycle events

**v2 (client sends):**

```json theme={null}
{"type": "stop"}
```

or:

```json theme={null}
{"type": "flush"}
```

Both `stop` and `flush` were accepted in v2 and had identical behavior. In v3, only `flush` exists, and it now accepts an optional `request_id` for correlation.

**v3 (client sends):**

```json theme={null}
{"type": "flush", "request_id": "f1"}
```

**v2 (server confirms end-of-speech via VAD):**

```json theme={null}
{"type": "speech_stop", "utterance_idx": 0}
```

**v3 (server lifecycle events):**

```json theme={null}
{"type": "input_speech_ended", "utterance_idx": 0, "input_time": 2.8}
```

```json theme={null}
{"type": "output_speech_ended", "utterance_idx": 0, "output_time": 3.5}
```

In v3, the server provides more granular lifecycle events. The `speech_stop` message is replaced by `input_speech_ended` (when the server detects the speaker stopped) and `output_speech_ended` (when all output audio has been sent). The `flushed` message confirms a manual flush is complete.

Note: in v2, a `flushed` message could arrive without a client-initiated `stop`/`flush` (triggered automatically by VAD via `output_speech_ended`). In v3, `output_speech_ended` and `flushed` are distinct events -- `flushed` only arrives in response to a client `flush`.

#### Language Detection: `languages` -> `language_route`

**v2:**

```json theme={null}
{"type": "languages", "lang_in": "en-US", "lang_out": "es-ES"}
```

**v3:**

```json theme={null}
{"type": "language_route", "utterance_idx": 0, "lang_in": "en-US", "lang_out": "es-ES", "is_final": true}
```

The v3 version includes `utterance_idx` to associate the language detection with a specific utterance, and `is_final` to distinguish the locked-in route from tentative routes emitted while language identification is still refining.

#### `can_lang_swap` -> Multiple Language Routes

**v2:**

```json theme={null}
{"type": "config", "lang_in": "en-US", "lang_out": "es-ES", "can_lang_swap": true}
```

**v3:**

```json theme={null}
{"type": "configure", "language_routes": [
  {"lang_in": "en-US", "lang_out": "es-ES"},
  {"lang_in": "es-ES", "lang_out": "en-US"}
]}
```

Instead of a boolean flag, v3 uses explicit bidirectional routes. You can also add a wildcard `"*"` -> `"*"` catch-all route to detect unexpected languages and prompt the user to switch (see [Language Routes](#language-routes) for details):

```json theme={null}
{"type": "configure", "language_routes": [
  {"lang_in": "en-US", "lang_out": "es-ES"},
  {"lang_in": "es-ES", "lang_out": "en-US"},
  {"lang_in": "*", "lang_out": "*"}
]}
```

#### Glossary Format

**v2:**

```json theme={null}
{"glossary": ["Sanas", "ACME"]}
```

Each string was treated as a term that should be preserved identically across all languages.

**v3:**

```json theme={null}
{"glossary": [
  {"terms": {"*": "Sanas"}},
  {"terms": {"*": "ACME"}},
  {"terms": {"*": "customer service", "es-ES": "servicio al cliente"}}
]}
```

The v3 format uses structured objects with per-language mappings. Always include a `"*"` wildcard key as the default so the term is handled correctly for any language. You can override specific languages alongside the wildcard. For brand names and proper nouns that should never change, a single `"*"` entry is all you need.

#### Speech Delimiter: `speech_delimiter` -> `output_text_boundary`

**v2:**

```json theme={null}
{
  "type": "speech_delimiter",
  "time": 1.5,
  "transcription": {"utterance_idx": 0, "word_idx": 3, "char_idx": 15},
  "translation": {"utterance_idx": 0, "word_idx": 2, "char_idx": 12}
}
```

**v3:**

```json theme={null}
{
  "type": "output_text_boundary",
  "utterance_idx": 0,
  "output_time": 1.5,
  "transcription": {"word_idx": 3, "char_idx": 15},
  "translation": {"word_idx": 2, "char_idx": 12}
}
```

The `utterance_idx` moved to the top level, `time` was renamed to `output_time`, and the inner position objects no longer repeat `utterance_idx`. v3 emits boundaries by default (matching v2's unconditional `speech_delimiter`), and clients that don't need them can opt out by adding `"hide_output_text_boundary"` to the `configure.features` list to skip server-side alignment work.

#### Renamed Events

| v2 Event          | v3 Event            | Notes                                                       |
| ----------------- | ------------------- | ----------------------------------------------------------- |
| `translation_end` | `translation_ended` | Name changed for consistency. Now includes `utterance_idx`. |

#### New Events in v3

The following server events are new in v3 and were not exposed on the v2 WebSocket:

| Event                   | Description                                        |
| ----------------------- | -------------------------------------------------- |
| `input_speech_started`  | VAD detected the start of speech                   |
| `transcription_ended`   | All transcription for an utterance is complete     |
| `output_speech_started` | Server began sending output audio for an utterance |
| `output_speech_ended`   | All output audio for an utterance has been sent    |

### Quick Migration Checklist

1. Change WebSocket URL from `/v2/consecutive` to `/v3/stream`
2. Replace the single `config` message with `init` + `configure`
3. Add `"features": ["consecutive"]` to the `configure` message
4. Replace JSON `"type": "audio"` input messages with raw PCM bytes in WebSocket Binary frames
5. Handle output audio as WebSocket Binary frames containing raw PCM bytes (instead of JSON `"type": "audio"` text frames)
6. Replace `"type": "stop"` (or `"type": "flush"`) with `"type": "flush"` (now supports `request_id`)
7. Handle `"type": "configured"` instead of `"type": "ready"`
8. Handle `"type": "language_route"` instead of `"type": "languages"`
9. Handle `"type": "input_speech_ended"` instead of `"type": "speech_stop"`
10. Handle `"type": "output_text_boundary"` instead of `"type": "speech_delimiter"` (emitted by default like v2; add `"hide_output_text_boundary"` to `configure.features` to opt out)
11. Update glossary format from flat strings to `{"terms": {"*": "term"}}` objects (always include the `"*"` wildcard key)
12. Replace `can_lang_swap: true` with a second language route
13. Handle `utterance_idx` on `transcription` and `translation` messages (new in v3)
14. Handle `"type": "translation_ended"` (renamed from v2 `"type": "translation_end"`)
15. (Optional) Handle new lifecycle events for richer UX