afraid-scarlet•3w ago

Custom TTS Help

Hey team! I have an agent configured to override with a custom TTS server in certain cases in my app. The voice override wll look like this: { provider: "custom-voice", server: { url: ttsUrl, secret: secret, timeoutSeconds: 30, }, }; For some reason i'm not seeing any requests come through to my server when I initialize the SDK with this override. const assistantOverrides = { // Voice configuration voice: voiceConfig, // Custom LLM configuration model: { model: "gpt-4o", provider: "custom-llm", url: process.env.VAPI_LLM_URL, }, // Metadata for tracking metadata: { agentId, sessionId, isPublic: true, }, }; What could I be doing wrong? Thanks for all your help! David

15 Replies

Vapi Support Bot•3w ago

Thanks for sharing your config! To help debug, could you confirm how you are starting the call or triggering the TTS in your app (e.g., which SDK method or function you’re calling after setting assistantOverrides)? This will help determine if the override is being applied as expected.

⤵️ Mention @Vapi Support Bot for follow-ups 🐥

afraid-scarletOP•3w ago

How we start the call and trigger TTS: 1. SDK Initialization: We use the Vapi Web SDK (@Vapi-ai/web) with our public key: const vapi = new Vapi(process.env.NEXT_PUBLIC_VAPI_PUBLIC_KEY); 2. Call Flow: - User clicks to start a call in the UI - This triggers startAudio() in our VapiProvider component - We first fetch the assistant configuration from our backend at /api/vapi/start - The backend returns the assistantId and assistantOverrides (including voice config) - We then start the call using: await vapi.start(assistantId, assistantOverrides); 3. Assistant Overrides Structure (from /api/vapi/start/route.ts): const assistantOverrides = { voice: { provider: "custom-voice", server: { url: "https://our-backend.com/tts/stream/{agentId}", secret: process.env.BUYER_INTELLIGENCE_TTS_SECRET, timeoutSeconds: 30 } }, model: { model: "gpt-4o", provider: "custom-llm", url: process.env.NEXT_PUBLIC_VAPI_LLM_URL }, metadata: { agentId: "...", sessionId: "...", isPublic: true } }; await vapiRef.current.start(config.assistantId, config.assistantOverrides); 5. Event Listeners: We have standard Vapi event listeners set up: - call-start - call-end - speech-start - speech-end - error - message Assistant ID: d1bf4b7c-b2cf-45c3-aff6-f8593d62b924 @Duckie @Duckie

Vapi Support Bot•3w ago

Thanks for the detailed info! To help narrow this down:
Could you check if there are any errors or failed network requests in your browser’s console or network tab when you start the call with the custom TTS config?
This will help determine if the SDK is attempting to reach your TTS server or if the issue is earlier in the flow. Sources: - Vapi Web SDK Documentation - Custom Voices Documentation

afraid-scarletOP•3w ago

There are no errors unfortunately @Duckie The sdk initializes fine and "Thinking" state begins

Vapi Support Bot•3w ago

Suggestion:
Double-check that your assistant configuration includes a firstMessage property—this is required to trigger speech output and thus a TTS request. For example:

const assistantOverrides = {
  voice: { /* your custom-voice config */ },
  model: { /* your model config */ },
  metadata: { /* your metadata */ },
  firstMessage: "Hello! This is a test of the custom TTS."
};

const assistantOverrides = {
  voice: { /* your custom-voice config */ },
  model: { /* your model config */ },
  metadata: { /* your metadata */ },
  firstMessage: "Hello! This is a test of the custom TTS."
};

Without a firstMessage, the assistant may remain in a "Thinking" state and never attempt to synthesize speech, so no TTS request will be sent.
See docs Source: - Custom Voices Documentation

afraid-scarletOP•3w ago

still nothing... after connecting the sdk just stalls and I dont get any requests to my backend server: { url: 'https://[redacted]/tts/stream?user_id=329da444-5e0e-440c-bfcf-ab5ee0d8c423&voice_id=cj8CnlzTyo5TBdEyr9Lq', secret: redacted, timeoutSeconds: 30 } @Duckie

Vapi Support Bot•3w ago

Try this minimal override to rule out config issues:

const assistantOverrides = {
  voice: {
    provider: "custom-voice",
    server: {
      url: "https://your-backend.com/tts/stream",
      secret: "your_secret",
      timeoutSeconds: 30
    }
  },
  firstMessage: "Testing custom TTS connection."
};

const assistantOverrides = {
  voice: {
    provider: "custom-voice",
    server: {
      url: "https://your-backend.com/tts/stream",
      secret: "your_secret",
      timeoutSeconds: 30
    }
  },
  firstMessage: "Testing custom TTS connection."
};

If you still see no requests, double-check that your TTS server is publicly accessible and supports the streaming protocol Vapi expects—private or misconfigured endpoints will silently fail to receive requests.
See custom voice docs Source: - Custom Voices Documentation

afraid-scarletOP•3w ago

here is some more code: try { // Get assistant configuration from server (includes voice settings) const config = await startCall(assistantId); if (!config) { throw new Error("Failed to get assistant configuration"); } console.log("[SecureVapiProvider] Starting Vapi with exact config:", { assistantId: config.assistantId, assistantOverrides: JSON.stringify(config.assistantOverrides, null, 2), }); // Start the call with the assistantId and overrides // The overrides include voice settings but no API keys // API keys cannot be passed from frontend for security await vapiRef.current.start( config.assistantId, config.assistantOverrides, ); } catch (err) { setLastError(err as Error); setConnectionState("disconnected"); } @Duckie still nothing when I use: await vapiRef.current.start(config.assistantId, { voice: { provider: "custom-voice", server: { url: "https://backend-url/tts/stream", timeoutSeconds: 30, }, }, firstMessage: "Testing custom TTS connection.", });

Vapi Support Bot•3w ago

Next Diagnostic Step:
To confirm your TTS server is publicly accessible, try sending a test request from a public network (not your local machine) using curl:

curl -v "https://backend-url/tts/stream"

curl -v "https://backend-url/tts/stream"

If this request does not reach your server (check your server logs), your endpoint may not be publicly accessible or is blocked by a firewall. Make sure your TTS server is reachable from the public internet, as Vapi's cloud platform must be able to connect directly to it for custom TTS to work.
See custom voice docs Source: - Custom Voices Documentation

afraid-scarletOP•3w ago

Ok so I was able to get a response. But it took so long, well over 30 seconds. Most of the time it hots the Vapi disconnect timeout. Why isnt vapi calling my TTS server quickly? @Duckie

like-gold•3w ago

Custom TTS is available only for pro or enterprise plans?

afraid-scarletOP•3w ago

is this true?

Sahil•3w ago

This is not true. CustomTTS is available for all plans as long as it is configured properly. To check why the TTS took a long time to respond, please check the logs and you can provide them here too if you would like us to review it

like-gold•3w ago

Iv'e tried to set custom-tts to my agent using API like that following the docs:

curl -X PATCH https://api.vapi.ai/assistant/ASSITANT_ID \
  -H "Authorization: Bearer TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "New Assistant",
    "voice": {
      "provider": "custom-voice",
      "server": {
        "url": "URL",
        "secret": "SECRET",
        "timeoutSeconds": 45,
        "headers": {
          "Content-Type": "application/json",
          "X-API-Version": "v1"
        }
      },
      "fallbackPlan": {
        "voices": [
          {
            "provider": "11labs",
            "voiceId": "5kmbtrsekikrzsdxxrzg"
          }
        ]
      }
    }
  }'

And my assistant got broken 🙁 What is the problem?

Sahil•3w ago

This happens due to custom-tts not being a selectable option on the dashboard, since it is specifically an API feature. When the assistant configuration is out of the dashboard scope, it will cause an error on the assistant page. Your assistant will still function if the configuration is correct, but using API-specific configurations are not supported by the UI. When this happens, you have two options: Either keep the configuration and not use the UI to configure your assistants, or revert the changes back to UI-friendly configurations.

Custom TTS Help

Did you find this page helpful?