Stream speech as it is generated

The Text to Speech stream sends audio while it is still being generated, so a listener hears the first words long before the whole reply is ready. One socket stays open for a whole conversation and takes one text after another. This page shows how to connect, play the audio as it arrives, feed a reply sentence by sentence, and stop speaking when the listener interrupts. Every message and field is on Stream text to speech.

When to stream

Stream when someone is waiting to hear the reply: a voice agent, a phone call, a live assistant. For audio you store or serve later, the synchronous endpoint gives you a file and a job.

How a session works

you                                        server
 |-- upgrade: Authorization --------------->|
 |-- {"type":"synthesize", ...} ----------->|
 |<-------------------------------- start   |
 |<---------------- binary PCM frames ...   |
 |<-------------------------------- done    |
 |-- {"type":"synthesize", ...} ----------->|   the next text, same socket
 |<-------------------------------- start   |
 |<---------------- binary PCM frames ...   |
 |-- {"type":"cancel"} -------------------->|   the listener interrupts
 |<---------------------------- cancelled   |

What you need

• An API key whose services include Text to Speech, or that has no service restriction. See Get an API key.
• A voiceId from List available voices.
• A server. Browsers cannot set the Authorization header on a WebSocket, so a web page cannot connect. Stream on your server and pass the audio on to your player; never put the key in a page.
• Credits on the workspace. See Credits.
• For the Python client below: Python 3.9 or later and pip install "websockets>=14".

Step 1: Connect

Open a WebSocket to wss://api.echonova.ai/v1/text-to-speech/stream with Authorization: Bearer YOUR_API_KEY on the upgrade request. The stream takes no query parameters. When something is wrong before the upgrade, the server answers with plain HTTP and the usual error envelope, and nothing is charged; see Errors.

Open the socket before you need it, for example when a call starts, and keep it for the whole conversation. Reusing it saves the connection time on every reply and counts as one of your concurrent streams.

Step 2: Send a synthesize message

{ "type": "synthesize", "requestId": "reply-1", "text": "أهلاً بك، كيف يمكنني مساعدتك اليوم؟", "voiceId": "66e5ef3a9c1d2b4f8a7e6c01" }

The fields mean what they mean on Generate speech synchronously; see Text to Speech voices and options.

Only one text is spoken at a time on a socket. A synthesize sent while another is still running cancels the running one first, exactly like cancel followed by synthesize.

Step 3: Play the audio as it arrives

The server answers with start, then binary frames, then done:

{ "type": "start", "requestId": "reply-1", "sampleRate": 24000, "encoding": "pcm_s16le", "channels": 1 }

Each binary frame is 16-bit signed little-endian mono PCM at the sampleRate from start. That is 24 kHz today, and synthesize takes no rate of its own, but read it from start rather than hard-coding it. Play the frames in the order they arrive, or append them to a file. Audio is generated a few seconds of speech at a time, so frames come in bursts: give your player a small buffer, a few hundred milliseconds, so it does not run dry between bursts.

{ "type": "done", "requestId": "reply-1", "durationSeconds": 2.9, "characters": 36, "creditsConsumed": 36 }

done means every frame of that text has been sent. Binary frames carry no requestId; the frames between a start and its done or cancelled belong to that request.

With the sounddevice package, for example, playback is a stream you write the frames to:

import sounddevice  # pip install sounddevice

player = sounddevice.RawOutputStream(samplerate=24000, channels=1, dtype="int16")
player.start()
# for every binary frame received:
player.write(frame)

Step 4: Speak a reply sentence by sentence

When the text comes from a language model, do not wait for the whole reply. Cut it into sentences as it is generated and send each sentence as its own synthesize, so the listener hears the first sentence while the model is still writing the rest.

Send the next sentence only after the done of the one before. A synthesize sent earlier cancels the sentence still being spoken. Keep a queue on your side:

1. When a sentence is complete (it ends with ., ?, !, ؟ or a line break), add it to the queue.
2. When the socket is idle, send the first sentence in the queue.
3. On done, send the next one.

Recommended

• One synthesize per sentence or short paragraph, each under 500 characters, sent one after the other on one socket.

Less effective

• One synthesize for every few words. Each piece is spoken on its own, so the voice loses the rhythm of the sentence.
• A new socket for every sentence. Each one adds a connection and counts against the limit of concurrent streams.

Step 5: Stop speaking when the listener interrupts

When the listener starts talking over the reply, send:

{ "type": "cancel" }

The server stops the synthesis and answers with {"type":"cancelled","requestId":"reply-1"}. No frame for that request follows cancelled.

Cancelling takes effect at the next chunk boundary: a few frames already generated can still arrive between your cancel and cancelled. Stop your player and clear its buffer at once, on your side, and drop every frame until cancelled arrives. Then send the next synthesize, or send it right away: a new synthesize cancels the running one too.

Complete Python client

This client sends one text, collects the PCM frames and writes them to speech.wav. The same client in JavaScript, TypeScript, Go and C# is on Stream text to speech.

import asyncio
import json
import wave

import websockets  # pip install "websockets>=14"

URL = "wss://api.echonova.ai/v1/text-to-speech/stream"
API_KEY = "YOUR_API_KEY"


async def main():
    pcm = bytearray()
    rate = 24000
    async with websockets.connect(
        URL, additional_headers={"Authorization": f"Bearer {API_KEY}"}
    ) as ws:
        await ws.send(json.dumps({
            "type": "synthesize",
            "requestId": "reply-1",
            "text": "أهلاً بك، كيف يمكنني مساعدتك اليوم؟",
            "voiceId": "66e5ef3a9c1d2b4f8a7e6c01",
        }))
        while True:
            message = await ws.recv()
            if isinstance(message, bytes):  # a binary frame of PCM16
                pcm.extend(message)
                continue
            event = json.loads(message)
            if event["type"] == "start":
                rate = event["sampleRate"]
            elif event["type"] == "done":
                print("seconds:", event["durationSeconds"], "credits:", event["creditsConsumed"])
                break
            elif event["type"] in ("error", "cancelled"):
                raise SystemExit(f"{event['type']}: {event.get('message', '')}")

    with wave.open("speech.wav", "wb") as wav:
        wav.setnchannels(1)
        wav.setsampwidth(2)
        wav.setframerate(rate)
        wav.writeframes(bytes(pcm))


asyncio.run(main())

Errors on an open socket

An error message carries code, message and, when it concerns one text, its requestId:

Every close code is listed on Errors.

Billing

Each synthesize costs 1 credit per character of text, spaces included, the same rate as the other Text to Speech endpoints. The credits are charged when the synthesis starts, before the first frame, and done repeats the amount in creditsConsumed. A cancelled synthesis, by cancel or by a newer synthesize, is still charged. If the synthesis fails before any audio is sent, the charge is refunded. A key's Credit Limit applies to every synthesize. See Credits.

Limits

See Rate limits for how the stream limits are counted.

Troubleshooting

My second sentence cut off the first

A synthesize sent before the previous one's done cancels it. Queue the sentences and send each one on the previous done, as in Step 4.

Frames keep arriving after I sent cancel

The synthesis stops at the next chunk boundary, so a few frames can follow your cancel. Drop every frame until cancelled arrives.

The audio plays too fast, too slow or as noise

Play it at the sampleRate from start, as 16-bit signed little-endian mono samples. A player set to another rate, to stereo or to 8-bit misreads the frames.

The socket closes after two minutes of silence

The idle timeout is 120 seconds without any frame. Send {"type":"ping"} every minute or so while the conversation is quiet; the server answers with pong.

FAQ

Can I get MP3 or Opus instead of PCM?

No. The stream sends raw 16-bit mono PCM. Encode it yourself if you need another format, or use Generate speech synchronously for a WAV file.

Can one socket use several voices?

Yes. voiceId belongs to each synthesize, so every text can use another voice, one text at a time.

Can I connect from a browser?

No. A browser cannot set the Authorization header on a WebSocket, and the key must never reach a web page. Stream on your server and pass the audio on.

Related

• Stream text to speech
• Text to Speech quickstart
• Text to Speech voices and options
• List available voices
• Synchronous and asynchronous calls
• Rate limits
• Credits