First Custom LLM request is extremely slow

Why First Custom LLM Requests Are Often Slow

1. Cold Start Effect: According to the information I found, Custom LLMs are more complex and resource-intensive than standard implementations, which can lead to slower initial requests source.
2. Network Latency: The Custom LLM request/response cycle involves network communication, making it susceptible to network latency. High network latency can contribute to delays in voice applications source.
3. Data Transfer Overhead: Custom LLMs use Server-Sent Events (SSE) for response streaming, and the size and format of request/response payloads can impact speed source.
4. Processing Time Factors: Several factors can affect processing time, including:
   
   • Call volume (high volumes can slow down response times)
   • VAPI configuration settings
   • Token limits
   • Tool calling complexity
     source

1. Check VAPI Configuration: Review settings like silenceTimeoutSecondssilenceTimeoutSeconds, responseDelaySecondsresponseDelaySeconds, and llmRequestDelaySecondsllmRequestDelaySeconds source.

1. Verify LLM Key Configuration: Ensure your LLM key is correctly configured and hasn't exceeded usage limits source.
2. Check Token Limits: Ensure requests are within token limits source.
3. Verify Streaming Response Format: Make sure your LLM sends a streaming response in SSE format source.
4. Check Network Latency: Verify network latency from your system to the API endpoints source.
5. Review Custom LLM Configuration: Double-check provider, URL, model, and tools configuration source.

• Discord Thread on Custom LLM Complexity
• Discord Thread on Network Latency
• Vapi Documentation on Custom LLM Tool Calling Integration
• Discord Thread on Token Limits
• Discord Thread on Custom LLM Configuration