sunny-green•5mo ago

Docs for WebSocket transport with python?

I'm trying to make calls using the WebSockets transport to send some generated audio to a Vapi agent. I'm using the python WebSockets library as well and am able to successfully send audio over and get back a response but any new audio sent over the same connection doesn't seem to be acknowledged. I eventually get a silence-timed-out error. Any guidance would be appreciated.

12 Replies

Vapi Support Bot•5mo ago

Vapi's WebSocket transport expects a continuous stream of 16-bit PCM audio data (little-endian) to avoid triggering a silence-timed-out error. In Python, ensure you are sending audio chunks at regular intervals without long pauses. Here's a minimal outline:

import asyncio
import websockets

async def send_audio(uri, audio_chunks):
    async with websockets.connect(uri) as ws:
        for chunk in audio_chunks:
            await ws.send(chunk)  # chunk should be bytes in PCM s16le format
            await asyncio.sleep(0.1)  # adjust interval as needed for real-time streaming

# Make sure audio_chunks is a generator yielding small, regular PCM chunks

import asyncio
import websockets

async def send_audio(uri, audio_chunks):
    async with websockets.connect(uri) as ws:
        for chunk in audio_chunks:
            await ws.send(chunk)  # chunk should be bytes in PCM s16le format
            await asyncio.sleep(0.1)  # adjust interval as needed for real-time streaming

# Make sure audio_chunks is a generator yielding small, regular PCM chunks

Keep the stream active by sending audio data continuously, and close the connection with a control message when done. For more, see the WebSocket transport docs. Source: - WebSocket Transport – Vapi Docs

⤵️ Mention @Vapi Support Bot for follow-ups 🐥

adverse-sapphire•5mo ago

Make sure continuous audio flow without long gaps, adjust the silence timeout setting if needed, and manage the call using the control URL to avoid silence-timed-out errors..

sunny-greenOP•5mo ago

ah gotcha, was able to get it to work. Happy to contribute to any docs with a python sample so it helps others out. One thing I think that is very much needed is docs on the types of messages that are sent over the WebSocket. Which of these can we rely on? For example I need to know when the Vapi agent has finished speaking, and for that I think I can look for the speech-update message with a status of done. But I also need to get the full transcript of what the agent just said. For that should I look for the voice-input message, or transcript message with status final , or the conversation-update message. All of this I feel like I had to figure out on my own by just connecting to the WebSocket and see what messages were being sent and observing how they behaved. cc @Kings_big💫

adverse-sapphire•5mo ago

Got it To detect when the Vapi agent finishes speaking, use the speech-update message with "status": "done", and to get the full transcript, listen for the transcript message with "status": "final"

sunny-greenOP•5mo ago

what control message should be sent to avoid silence-timed-out errors? Any dummy assistant control message would work? https://docs.vapi.ai/calls/call-features

Live Call Control | Vapi

adverse-sapphire•5mo ago

Send periodic control messages like conversation-update or speech-update to keep the call active.

Kyle Brunker•5mo ago

Hey Mathusan, checking if this is resolved/solved for you?

sunny-greenOP•5mo ago

yep solved for me, thanks yall!

Vapi Support Bot•5mo ago

To avoid silence-timed-out errors, send a periodic control message like this:

sendControlMessage({ type: "conversation-update" });

sendControlMessage({ type: "conversation-update" });

Any valid control message (such as conversation-update or speech-update) will keep the call active. Adjust the interval as needed to prevent timeouts, since the docs do not specify a required frequency.
Reference: Vapi WebSocket Transport Docs Sources: - Vapi WebSocket Transport Docs - Call Features

deep-jade•5mo ago

Hi @mathusan, @Kings_big💫 , can you share the python snippet that worked for you for using Vapi websockets api with python? I am initiating sending of audio stream once I receive speech-update, status:closed event. This is my current snippet, but it seems in the vapi dashboard that the audio I am sending and the assistant that is speaking are overlapping significantly, leading to error: call.in-progress.error-assistant-did-not-receive-customer-audio

deep-jade•5mo ago

rough.py

Kyle Brunker•5mo ago

This ticket has been marked as solved, and as a result, you will no longer receive any further responses. Kindly create a new support ticket to escalate your inquiry. 

Docs for WebSocket transport with python?

Did you find this page helpful?