Transcribe live audio

The Speech to Text stream takes audio while it is being spoken, from a microphone, a phone call or a live broadcast, and sends back the transcript of each piece of speech as soon as that piece ends. This page shows how to prepare the audio, connect, send it, decide where each piece ends, and read the results. Every message and field is on Stream speech to text.

When to stream

Stream when the audio does not exist yet as a file and you need the words while the conversation is still going: a voice agent that must answer a caller, or captions for a live event. For a recording that is already on disk, upload it: it costs half as much.

How a session works

you                                        server
 |-- upgrade: auth, ?sample_rate, ?language >|
 |<------------------------- session.started |
 |-- binary PCM frames (100 ms each) ------->|
 |-- {"type":"end_segment"} ---------------->|
 |<------------------------------ transcript |
 |-- more frames, more end_segment ... ----->|
 |-- {"type":"stop"} ----------------------->|
 |<----------------- last transcripts, then  |
 |<--------------------------- session.ended |
 |<------------------------------ close 1000 |

What you need

• An API key whose services include Speech to Text, or that has no service restriction. See Get an API key.
• A server. Browsers cannot set the Authorization header on a WebSocket, so a web page cannot connect. If the audio comes from a browser, send it to your own server and open the stream from there; never put the key in the page.
• Credits on the workspace. See Credits.
• For the Python client below: Python 3.9 or later and pip install "websockets>=14".

Step 1: Prepare the audio

The stream accepts one format: 16-bit signed little-endian PCM, one channel (pcm_s16le), with no file header. Choose a sample rate from 8,000 to 48,000 Hz and tell the server with sample_rate when you connect. 16,000 Hz is enough for speech and keeps frames small; 8,000 Hz suits telephone audio that was never wider.

Convert stereo to mono and compressed audio to PCM before you send it. To turn a file into a WAV file the samples can read:

ffmpeg -i input.mp3 -ac 1 -ar 16000 -c:a pcm_s16le audio.wav

A WAV file of this kind is a short header followed by exactly these samples, which is why the samples read the rate from the header and send the rest. A live source such as a microphone gives you the samples directly: capture at the rate you connect with, as 16-bit integers, one channel.

Step 2: Connect

Open a WebSocket to wss://api.echonova.ai/v1/speech-to-text/stream with your key on the upgrade request:

When the key, the parameters and the balance are in order, the server accepts the upgrade and sends session.started. Wait for it before you send audio:

{ "type": "session.started", "sessionId": "c8a1f2e4-5b6d-4e7f-9a0b-1c2d3e4f5a6b", "language": "ar", "sampleRate": 16000, "maxSessionSeconds": 1500 }

Keep sessionId in your logs: it identifies the session if you contact support. When something is wrong before the upgrade, the server answers with plain HTTP and the usual error envelope instead, and nothing is charged. The statuses are under Errors.

Step 3: Send audio frames

Send the PCM as binary WebSocket frames, in order. About 100 ms of audio per frame works well:

A frame may not be larger than 64 KiB. A larger one gets a frame_too_large error and closes the session with 1008; one over 1 MiB closes it with 1009 and no error message. Keep every frame an even number of bytes, so no sample is split across two frames.

Send live audio as it is captured. When you stream a file, as the samples do, pause about 100 ms between frames so the server sees it the way it would see a live source.

Step 4: End each segment

The server transcribes audio in segments. A segment ends when you send {"type":"end_segment"}, or after 15 seconds of audio if you send nothing, and its transcript follows shortly after. Because there are no partial results, where the segments end decides how soon you see text and how well it reads.

Recommended

• Detect pauses on your side, with a voice activity detector or a simple energy threshold, and send end_segment when the speaker has been silent for 300 to 700 ms. Each transcript is then one whole sentence or turn.
• In a voice agent, send end_segment as soon as your turn detection decides the caller has finished, and answer from that transcript.

Less effective

• Never sending end_segment. Speech is then cut every 15 seconds, often in the middle of a word, and the first words wait up to 15 seconds.
• Sending end_segment on a fixed timer, such as every second. Short segments lose the context that makes a transcript accurate, and each segment is a separate charge rounded to a whole credit.

Step 5: Read the transcripts

Each segment comes back as one transcript message:

{ "type": "transcript", "segmentId": "6720f3a1c4b5d6e7f8a9b0c1", "text": "مرحبا، كيف يمكنني مساعدتك اليوم؟", "language": "ar", "durationSeconds": 2.4, "creditsConsumed": 8, "isFinal": true }

A transcript can arrive a moment after you sent end_segment and while you keep sending audio for the next segment. Keep reading and sending at the same time, as the samples do with one task for each.

Step 6: Stop

Send {"type":"stop"} when the audio ends. The server transcribes what is left, sends the last transcript messages, then session.ended, then closes the socket with code 1000:

{ "type": "session.ended", "reason": "stopped", "totalSeconds": 42.6, "creditsConsumed": 142 }

totalSeconds and creditsConsumed are the totals of the whole session. Closing the socket yourself also ends the session, but audio that was not yet transcribed is then dropped.

Complete Python client

This client streams audio.wav in 100 ms frames, ends the segment, prints each transcript and stops. The same client in JavaScript, TypeScript, Go and C# is on Stream speech to text.

import asyncio
import json
import wave

import websockets  # pip install "websockets>=14"

URL = "wss://api.echonova.ai/v1/speech-to-text/stream"
API_KEY = "YOUR_API_KEY"


async def main():
    # A 16-bit mono PCM WAV file; its header gives the sample rate.
    with wave.open("audio.wav", "rb") as wav:
        if wav.getsampwidth() != 2 or wav.getnchannels() != 1:
            raise SystemExit("audio.wav must be 16-bit mono PCM")
        rate = wav.getframerate()
        pcm = wav.readframes(wav.getnframes())

    async with websockets.connect(
        f"{URL}?sample_rate={rate}&language=ar",
        additional_headers={"Authorization": f"Bearer {API_KEY}"},
    ) as ws:
        print(json.loads(await ws.recv()))  # session.started

        async def send_audio():
            step = rate // 10 * 2  # 100 ms of 16-bit samples, in bytes
            for i in range(0, len(pcm), step):
                await ws.send(pcm[i : i + step])  # bytes: a binary frame
                await asyncio.sleep(0.1)  # real time, like a live source
            await ws.send(json.dumps({"type": "end_segment"}))
            await ws.send(json.dumps({"type": "stop"}))

        sender = asyncio.create_task(send_audio())
        async for raw in ws:  # ends when the server closes after session.ended
            message = json.loads(raw)
            if message["type"] == "transcript":
                print(message["text"])
            elif message["type"] == "error":
                print("error:", message["code"], message["message"])
            elif message["type"] == "session.ended":
                print("ended:", message["reason"], "credits:", message["creditsConsumed"])
        await sender


asyncio.run(main())

Reconnect after a failure

A session can end before you send stop. session.ended gives the reason and the close code says what to do:

Audio sent after the last transcript you received is not transcribed when a session ends with an error, and it is not charged. If those words matter, keep the last few seconds of audio on your side and send them again on the new session.

Billing

The stream is charged at the Live Speech to Text rate, 200 credits a minute of audio, per second. Each segment is charged when its transcript is sent and rounded to a whole credit; the 2.4-second segment above costs 8 credits. A segment with no speech in it comes back with empty text and is still charged for its length, so do not stream long silences you do not need transcribed. A key's Credit Limit applies to every segment. See Credits.

Limits

See Rate limits for how the stream limits are counted.

Troubleshooting

The upgrade fails with 400 sample_rate must be an integer from 8000 to 48000

sample_rate is missing or out of range. Connect with ?sample_rate= set to the rate of your audio.

The upgrade fails with 400 language must be one of ar, en

language is missing from the query, or holds something other than ar or en. Connect with ?language=ar for Arabic and mixed speech, or ?language=en for speech that is all English.

The upgrade fails with 429 Too many concurrent streams

The workspace already has 10 streams open, counting Text to Speech streams. Close one, or check that your code closes a socket when a call or session ends.

No transcript arrives

Nothing is transcribed until a segment ends. Send end_segment when the speaker pauses, or stop at the end.

Every transcript is empty or garbled

The audio does not match what you declared. Check that it is 16-bit, little-endian and mono, that it has no file header, and that sample_rate is the rate it was recorded at. A wrong rate makes speech sound too fast or too slow to the model.

The socket closes with 1008

A frame was larger than 64 KiB, a text frame was not valid JSON, or its type is not one the stream knows. Read the error message sent just before the close for the details.

FAQ

Can I get words while someone is still speaking?

No. Each transcript covers a whole segment and is sent once, when the segment ends. Send end_segment at every pause to get text sooner.

Can I connect from a browser?

No. A browser cannot set the Authorization header on a WebSocket, and the key must never reach a web page. Capture audio in the browser, send it to your server, and open the stream from there.

Can I send MP3, Opus or stereo audio?

No. Decode and convert it to 16-bit mono PCM first, as in Step 1.

Is a session also a job?

No. A stream creates no job, so it is not listed by GET /v1/speech-to-text/jobs, and it sends no webhook. The transcripts are only in the messages you receive, so store them as they arrive.

Related

• Stream speech to text
• Speech to Text quickstart
• Synchronous and asynchronous calls
• Rate limits
• Errors
• Credits