foreign-sapphire•8mo ago
🐛 Tool calls are not working with Open AI Realtime model
Context: imagine you create a tool to get the weather.
While it works well with all models, when switching to the models gpt-4o-realtime-preview
the webhook is calling but the answer is not parsed well.
The default vapi response:
{results: [{toolCallId: "call_12345678", result: "Temperature is 63F in Miami, FL"}]}
is not working.
I tried also with openai format with no luck:
{results: [{tool_call_id: "call_12345678", content: "Temperature is 63F in Miami, FL", role: "tool", name: "retrieve_weather"}]}
Even worst it's looping.
Any idea when/if it can be fixed?
Example: 6824e749-392c-44aa-b883-d6fba35be69a
9 Replies
Relevant threads
These are relevant threads you might take a look into before continuing further:
- <#1334547175253868725>
@cquilboss
I'm sorry to hear you're experiencing issues with the webhook response parsing when using the gpt-4o-realtime-preview model. It seems like there might be a discrepancy in how the model is handling the webhook response format.
To address this, I would recommend checking the following:
1. Ensure that the response format from your webhook strictly adheres to the expected schema for the VAPI. Small deviations in key names or data structure can cause parsing issues.
2. Verify that the model you're using is compatible with the webhook response format. It's possible that gpt-4o-realtime-preview has specific requirements or limitations.
3. If the issue persists, it might be a bug or a limitation with the gpt-4o-realtime-preview model's handling of webhook responses. In this case, reaching out to the support team with the example ID you provided (6824e749-392c-44aa-b883-d6fba35be69a) would be the best course of action.
Since I'm here to assist you positively, let's try to troubleshoot this together. Could you provide more details about the response format you are using and any error messages you might be receiving? This information will help me guide you more effectively.
If the issue is urgent or you're not able to share more details, please ask support for more info on this. They will have the most up-to-date information and can provide assistance tailored to your specific situation.
You can continue this conversation by mentioning me in the message.
foreign-sapphireOP•8mo ago
ok keep me posted when you have an idea why
any news?
correct-apricot•8mo ago
@cquilboss can you share few other call Ids so I can try to find patterns, for what could have been wrong.
correct-apricot•8mo ago
@cquilboss FYI: logs in the screenshot stating the error reason.
foreign-sapphireOP•8mo ago
Sure: 904be3e5-0cb0-4a59-9b5b-8049b0317c5d (from another account)
I made it work with the assistant but not with the API
same that works with the assistant: bce9c62d-70ed-47b4-bc0b-f802de2e5118
another crazy one that after is looping: fc5ccf3e-3007-4134-aa65-0d0df3ba15b9
correct-apricot•8mo ago
@cquilboss Realtime model is still in beta mode, and because of this you can expect few hiccups. I suggest using GPT-4o for production and realtime for development/testing for somemore time. It's not only us but the model is still in beta.
quickest-silver•4mo ago
I'm having the same issue - the tool call is being called correctly (via API) and is returning the right weather but it's just being ignored by realtime and it makes up some random fkn weather.
This ticket has been marked as solved, and as a result, you will no longer receive any further responses. Kindly create a new support ticket to escalate your inquiry.