Multilingual Voice Agent Not Responding in Caller’s Language (French/Italian/Spanish)

sour-pink · 2025-06-19T05:24:54.903Z

Hi team, I’ve created a voice assistant in VAPI for a dental clinic, and I'm using a system prompt where Ellie (the agent) should automatically detect the caller’s language (French, Italian, Spanish) and respond in that same language. Here’s the relevant section from the system prompt: "Ellie automatically adapts to the caller's language. If the caller speaks French, respond entirely in French. If the caller speaks Italian, respond entirely in Italian. If the caller speaks Spanish, respond entirely in Spanish. If the caller speaks English or another language, continue speaking in English." However, even when I speak in French or Spanish during the call, the assistant keeps responding in English. It doesn’t seem like the language detection is working properly. Could you please guide me on how to ensure language auto-detection and dynamic response in the caller's spoken language works correctly in VAPI? Is there a specific setting, flag, or syntax I’m missing in the system prompt or the VAPI configuration? Thanks in advance!

V

Vapi Support BotAPP•6/19/25, 5:26 AM

To enable language auto-detection in Vapi, make sure your assistant uses a transcriber that supports multilingual recognition (like Deepgram or Google). In your assistant configuration, set the transcriber as follows:

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2"
}

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2"
}

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2"
}

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2"
}

This allows the transcriber to detect and transcribe multiple languages. Note: The system prompt alone is not enough—ensure your transcriber and TTS voices support the target languages for full multilingual functionality.
See more in the Vapi multilingual documentation.

Source:

Vapi Multilingual Documentation

⤵️ Mention @Vapi Support Bot for follow-ups 🐥

S

sour-pinkOP•6/19/25, 5:33 AM

@Duckie
it's already enalbed but still not working properly.

V

Vapi Support BotAPP•6/19/25, 5:35 AM

To enable true multilingual detection, set your transcriber language to
"multi"
"multi"
"multi"
"multi" instead of
"English"
"English"
"English"
"English" in your assistant configuration. For example:

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2",
  "language": "multi"
}

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2",
  "language": "multi"
}

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2",
  "language": "multi"
}

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2",
  "language": "multi"
}

This allows the assistant to detect and transcribe French, Italian, and Spanish automatically. After updating, test again to see if the assistant responds in the caller's language.
See Vapi multilingual documentation for more details.

Source:

Vapi Multilingual Documentation

V

Vapi Support BotAPP•6/19/25, 5:35 AM

To enable true multilingual detection, set your transcriber language to
"multi"
"multi"
"multi"
"multi" instead of
"English (En)"
"English (En)"
"English (En)"
"English (En)".
Update your assistant's configuration as follows:

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2",
  "language": "multi"
}

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2",
  "language": "multi"
}

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2",
  "language": "multi"
}

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2",
  "language": "multi"
}

This should allow the assistant to detect and transcribe French, Italian, and Spanish. Also, ensure your TTS voice supports these languages for proper responses.
See Vapi multilingual documentation for more details.

Source:

Vapi Multilingual Documentation

S

sour-pinkOP•6/19/25, 5:48 AM

@Duckie
still not working over the call assistant says "Sorry. I can only assist in English."

V

Vapi Support BotAPP•6/19/25, 5:50 AM

To enable multilingual detection, set your transcriber language to
"multi"
"multi"
"multi"
"multi" in your assistant configuration:

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2",
  "language": "multi"
}

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2",
  "language": "multi"
}

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2",
  "language": "multi"
}

"transcriber": {
  "provider": "deepgram",
  "model": "nova-2",
  "language": "multi"
}

Also, make sure your TTS voice supports French, Italian, and Spanish—otherwise, the assistant may still respond in English even if the transcriber detects the language.
See Vapi multilingual documentation for more details.

Source:

Vapi Multilingual Documentation

V

Vapi Support BotAPP•6/19/25, 5:50 AM

Thanks for sharing your configuration steps. You’ve set the transcriber language to

"multi"

"multi"

"multi"

"multi", which is correct for multilingual detection. However, the assistant still responds only in English.

Clarifying question:
Can you confirm which TTS (voice) provider and voice ID you have set in your assistant’s configuration? The TTS voice must explicitly support French, Italian, and Spanish for dynamic language responses to work (see documentation). If the voice only supports English, the assistant will default to English even if the transcriber detects another language.

Example voice config for Spanish (Azure):

"voice": {
  "provider": "azure",
  "voiceId": "es-ES-ElviraNeural"
}

"voice": {
  "provider": "azure",
  "voiceId": "es-ES-ElviraNeural"
}

"voice": {
  "provider": "azure",
  "voiceId": "es-ES-ElviraNeural"
}

"voice": {
  "provider": "azure",
  "voiceId": "es-ES-ElviraNeural"
}

Could you check and share your current voice settings?

Source:

Vapi Multilingual Documentation

S

sour-pinkOP•6/19/25, 6:00 AM

@Duckie
i used azure, Elvira voice id but it only speaks in spanish

V

Vapi Support BotAPP•6/19/25, 6:02 AM

Vapi does not currently support a single TTS voice that dynamically switches between French, Italian, and Spanish.
To achieve dynamic language responses, you need to programmatically set the TTS voice based on the detected language. For example:

// Pseudocode for dynamic voice selection
if (detectedLanguage === "fr") {
  voice = { "provider": "azure", "voiceId": "fr-FR-DeniseNeural" };
} else if (detectedLanguage === "it") {
  voice = { "provider": "azure", "voiceId": "it-IT-ElsaNeural" };
} else if (detectedLanguage === "es") {
  voice = { "provider": "azure", "voiceId": "es-ES-ElviraNeural" };
} else {
  voice = { "provider": "azure", "voiceId": "en-US-JennyNeural" };
}

// Pseudocode for dynamic voice selection
if (detectedLanguage === "fr") {
  voice = { "provider": "azure", "voiceId": "fr-FR-DeniseNeural" };
} else if (detectedLanguage === "it") {
  voice = { "provider": "azure", "voiceId": "it-IT-ElsaNeural" };
} else if (detectedLanguage === "es") {
  voice = { "provider": "azure", "voiceId": "es-ES-ElviraNeural" };
} else {
  voice = { "provider": "azure", "voiceId": "en-US-JennyNeural" };
}

// Pseudocode for dynamic voice selection
if (detectedLanguage === "fr") {
  voice = { "provider": "azure", "voiceId": "fr-FR-DeniseNeural" };
} else if (detectedLanguage === "it") {
  voice = { "provider": "azure", "voiceId": "it-IT-ElsaNeural" };
} else if (detectedLanguage === "es") {
  voice = { "provider": "azure", "voiceId": "es-ES-ElviraNeural" };
} else {
  voice = { "provider": "azure", "voiceId": "en-US-JennyNeural" };
}

// Pseudocode for dynamic voice selection
if (detectedLanguage === "fr") {
  voice = { "provider": "azure", "voiceId": "fr-FR-DeniseNeural" };
} else if (detectedLanguage === "it") {
  voice = { "provider": "azure", "voiceId": "it-IT-ElsaNeural" };
} else if (detectedLanguage === "es") {
  voice = { "provider": "azure", "voiceId": "es-ES-ElviraNeural" };
} else {
  voice = { "provider": "azure", "voiceId": "en-US-JennyNeural" };
}

Update your assistant's configuration at runtime based on the detected language to ensure the correct TTS voice is used.
See Vapi multilingual documentation for more details.

Source:

Vapi Multilingual Documentation

V

Vapi Support BotAPP•6/19/25, 6:02 AM

Your current Azure voice (
es-ES-ElviraNeural
es-ES-ElviraNeural
es-ES-ElviraNeural
es-ES-ElviraNeural) only supports Spanish, so dynamic multilingual replies won't work.
To enable automatic language responses, switch to a TTS provider or voice that supports all your target languages (French, Italian, Spanish) in one configuration—such as ElevenLabs Turbo 2.5, which is mentioned in the UI tip for multilingual support.

Example voice config:

"voice": {
  "provider": "elevenlabs",
  "voiceId": "eleven_multilingual_voice_id"
}

"voice": {
  "provider": "elevenlabs",
  "voiceId": "eleven_multilingual_voice_id"
}

"voice": {
  "provider": "elevenlabs",
  "voiceId": "eleven_multilingual_voice_id"
}

"voice": {
  "provider": "elevenlabs",
  "voiceId": "eleven_multilingual_voice_id"
}

Update your assistant's voice to a multilingual-compatible option, then test again.
See Vapi multilingual documentation for more details.

Source:

Vapi Multilingual Documentation

S

sour-pinkOP•6/19/25, 6:39 AM

@Duckie
This too is not working,
can we transfer the call to another assistant not to the phone number in the VAPI ?

V

Vapi Support BotAPP•6/19/25, 6:40 AM

Yes, you can transfer a call to another assistant in Vapi without using a phone number.
Handle the

transfer-destination-request

transfer-destination-request

transfer-destination-request

transfer-destination-request webhook and respond with a payload like this:

{
  "destination": {
    "type": "assistant",
    "assistantId": "your-target-assistant-id",
    "message": "Transferring you to another assistant."
  }
}

{
  "destination": {
    "type": "assistant",
    "assistantId": "your-target-assistant-id",
    "message": "Transferring you to another assistant."
  }
}

{
  "destination": {
    "type": "assistant",
    "assistantId": "your-target-assistant-id",
    "message": "Transferring you to another assistant."
  }
}

{
  "destination": {
    "type": "assistant",
    "assistantId": "your-target-assistant-id",
    "message": "Transferring you to another assistant."
  }
}

This will route the call directly to the specified assistant within Vapi.

Source:

Dynamic Call Transfers Documentation

V

Vapi Support BotAPP•6/19/25, 6:40 AM

Based on the documentation, Vapi supports transferring calls to phone numbers or SIP URIs, but there is no explicit method to transfer a call directly to another Vapi assistant by referencing its assistant ID.

To clarify your use case:
Are you looking to transfer the call to another Vapi assistant instance (by assistant ID), or do you want to transfer to a SIP endpoint or phone number where another assistant is configured to answer?
This will help determine the best approach for your scenario.

See Vapi dynamic call transfer documentation for more details.

Sources:

S

sour-pinkOP•6/19/25, 7:22 AM

@Duckie
i created the transfer call tool but when i try to call the first assistant in which the transfer tool is included it throws error "Invalid Assistant
Management Not Found."

V

Vapi Support BotAPP•6/19/25, 7:23 AM

Thanks for sharing the error details.

To help resolve the "Invalid Destination" issue, could you please confirm:
Are you using the exact assistant ID (not just the name) of the target assistant in your transfer tool configuration?
You can find the assistant ID in the Vapi dashboard under the assistant's settings. Using the correct ID (not the display name) is required for the transfer to work if this method is supported.

Let me know, and I can guide you on the next steps!

Sources:

V

Vapi Support BotAPP•6/19/25, 7:24 AM

To resolve the "Invalid Destination. Assistant ... Not Found" error, ensure you are using the correct assistant ID (not just the name) in your transfer tool configuration.

Replace the destination with your actual assistant ID from the Vapi dashboard:

{
  "destination": {
    "type": "assistant",
    "assistantId": "your-target-assistant-id",
    "message": "Transferring you to another assistant."
  }
}

{
  "destination": {
    "type": "assistant",
    "assistantId": "your-target-assistant-id",
    "message": "Transferring you to another assistant."
  }
}

{
  "destination": {
    "type": "assistant",
    "assistantId": "your-target-assistant-id",
    "message": "Transferring you to another assistant."
  }
}

{
  "destination": {
    "type": "assistant",
    "assistantId": "your-target-assistant-id",
    "message": "Transferring you to another assistant."
  }
}

Double-check the assistant ID in your dashboard and update your transfer tool accordingly.
See Vapi dynamic call transfer docs for details.

Source:

Dynamic Call Transfers Documentation

S

sour-pinkOP•6/19/25, 7:54 AM

@Duckie
{
"type": "transferCall",
"destinations": {
"type": "assistant",
"assistantId": "6b2e5119-d4fd-411d-87ca-4f09c89b5a62",
"message": "Please wait while I connect you to the other assistant."
},
"function": {
"name": "transferToAnotherAssistant"
}
}
i used this to create the function but it throws 400 error
"message": [
"destinations must be an array",
"each value in destinations.property assistantId should not exist"
],
"error": "Bad Request",
"statusCode": 400

V

Vapi Support BotAPP•6/19/25, 7:56 AM

The
destinations
destinations
destinations
destinations field must be an array of phone numbers or SIP URIs—direct transfer to another assistant by
assistantId
assistantId
is not supported.

To fix the error, use a format like this:

"destinations": [
  {
    "type": "number",
    "number": "+1234567890",
    "message": "Transferring your call."
  }
]

"destinations": [
  {
    "type": "number",
    "number": "+1234567890",
    "message": "Transferring your call."
  }
]

"destinations": [
  {
    "type": "number",
    "number": "+1234567890",
    "message": "Transferring your call."
  }
]

"destinations": [
  {
    "type": "number",
    "number": "+1234567890",
    "message": "Transferring your call."
  }
]

If you want to route to another assistant, configure that assistant to answer at a specific phone number or SIP URI, and transfer the call there.
See Vapi dynamic call transfer docs for details.

Sources:

V

Vapi Support BotAPP•6/19/25, 7:56 AM

The "destinations" field must be an array, and only supports types like "number" or "sip"—not "assistantId".
Here is the correct format for a transfer to a phone number:

{
  "type": "transferCall",
  "destinations": [
    {
      "type": "number",
      "number": "+1234567890",
      "message": "Transferring your call."
    }
  ],
  "function": {
    "name": "transferCall"
  }
}

{
  "type": "transferCall",
  "destinations": [
    {
      "type": "number",
      "number": "+1234567890",
      "message": "Transferring your call."
    }
  ],
  "function": {
    "name": "transferCall"
  }
}

{
  "type": "transferCall",
  "destinations": [
    {
      "type": "number",
      "number": "+1234567890",
      "message": "Transferring your call."
    }
  ],
  "function": {
    "name": "transferCall"
  }
}

{
  "type": "transferCall",
  "destinations": [
    {
      "type": "number",
      "number": "+1234567890",
      "message": "Transferring your call."
    }
  ],
  "function": {
    "name": "transferCall"
  }
}

Direct transfer to another assistant using "assistantId" is not supported in the transferCall tool according to current documentation.
See Vapi transferCall documentation for details.

Sources:

K

KyleAPP•6/19/25, 2:51 PM

Ensure you’re using a multilingual transcriber, configure language-specific voices, and optionally use code or webhooks to switch voices dynamically based on the detected language, your current prompt alone isn’t enough without these settings..

https://docs.vapi.ai/customization/multilingual

Multilingual Voice Agent Not Responding in Caller’s Language (French/Italian/Spanish)

Similar Threads

Similar Threads

Similar Threads