judicial-coral•5mo ago

VAPI Custom LLM responses are truncated

I have configured a custom llm on VAPI. The issue is that all the messages received are truncated. Have you ever seen this issue? Can you help me? I asked the LLM to give me the definition of the word dinosaur.

data: {"result":{"messages":[{"role":"assistant","content":"Un"}],"metadata":""}}

data: {"result":{"messages":[{"role":"assistant","content":" din"}],"metadata":""}}

data: {"result":{"messages":[{"role":"assistant","content":"osa"}],"metadata":""}}

data: {"result":{"messages":[{"role":"assistant","content":"ure"}],"metadata":""}}

data: [DONE]

data: {"result":{"messages":[{"role":"assistant","content":"Un"}],"metadata":""}}

data: {"result":{"messages":[{"role":"assistant","content":" din"}],"metadata":""}}

data: {"result":{"messages":[{"role":"assistant","content":"osa"}],"metadata":""}}

data: {"result":{"messages":[{"role":"assistant","content":"ure"}],"metadata":""}}

data: [DONE]

11 Replies

Vapi•5mo ago

Thanks for sharing the details and the example output. From what you've described, it looks like the responses from your custom LLM are being received in small, segmented chunks rather than as a complete message. Based on the information I have, I don't have documentation or details about this specific issue with VAPI custom LLM integrations, so I can't confirm if this is a known problem or provide a specific solution. If you have any more details about your setup or configuration, feel free to share them!

⤵️ Mention @Vapi Support Bot for follow-ups 🐥

Vapi•5mo ago

⤵️ Mention @Vapi Support Bot for follow-ups 🐥

adverse-sapphire•5mo ago

Sorry, what's truncated? Can you elaborate more on what the issue is?

sunny-green•5mo ago

Check token limits, ensure proper aggregation of streamed chunks, and verify your server’s formatting of the response before sending it to Vapi.

judicial-coralOP•5mo ago

Thank you for your reply. Vapi's reply in the chat was truncated. Below is VAPI's reply and the reply I should receive.

judicial-coralOP•5mo ago

The reply from VAPI

judicial-coralOP•5mo ago

The reply sent by my custom llm to VAPI

judicial-coralOP•5mo ago

As you can see, the response is sent by my LLM client. But at the VAPI level, the message is truncated. To tell you everything, the custom LLM that I use is Anything LLM

sunny-green•5mo ago

Modify your LLM client to send full sentences or paragraphs per chunk, not tiny fragments.

judicial-coralOP•4mo ago

Ok I will try that thank you My custom LLM is Anything LLM and I cannot modify it. I tried to chat with open web ui and everything is ok. It seems Vapi cut the stream during the streaming

sunny-green•4mo ago

checking if this is resolved/solved for you?

VAPI Custom LLM responses are truncated

Did you find this page helpful?