OpenAI Realtime Voice is not working
None of my realtime assistants are generating any voice messages or audio. Text works fine. Example assistant: https://dashboard.vapi.ai/assistants/81e32055-6acd-4a88-82ac-45bfbf7bd093
KPlease provide a call id as well for our investigation
BYes this has actually been failing for me for like a week. I've switched to a different model stack in the meantime but would love for this to be fixed
Call Id: 8d240713-c4b2-4043-9d31-e4719a36f43b
Call Id: 8d240713-c4b2-4043-9d31-e4719a36f43b
BAre you able to get them to work when testing on your side Kyle?
KNo, GPT-4o mini realtime is currently and beta and we don't recommend using it especially for production environments. We hope that when it comes out of beta that it will perform better.
BHey Kyle, I'm sorry but I have to call BS on that. I have a setup where I run websockets through my server to connect open ai realtime API and twilio and the open ai realtime API is working on that. Something is broken on the VAPI side. I'll submit a support ticket as well with all of the information that I have.
Bok email send to support@vapi.ai -- I'll copy here as well for visibility
BHey Vapi Team,
I've noticed that the Open AI Realtime models have been broken on the VAPI platform for over a week now. I think the issue is within VAPI because I have a setup where I connect OpenAI websockets and twilio websockets in my own server and the Realtime API is working there.
Here's some call IDs from today. One is from a VAPI phone number and the other is one that I imported from twilio
52ee9bcf-7953-4bad-a807-7af56d8ff00a
c6dc8904-567e-47ac-9b5d-2a1ee03e7fd5
It's breaking across all prompts that I've tried.
I've noticed that the Open AI Realtime models have been broken on the VAPI platform for over a week now. I think the issue is within VAPI because I have a setup where I connect OpenAI websockets and twilio websockets in my own server and the Realtime API is working there.
Here's some call IDs from today. One is from a VAPI phone number and the other is one that I imported from twilio
52ee9bcf-7953-4bad-a807-7af56d8ff00a
c6dc8904-567e-47ac-9b5d-2a1ee03e7fd5
It's breaking across all prompts that I've tried.
BAlso, for my previous company, I've worked with both Bland and VAPI and have had a similar problem with Bland where the agent will just fail to start speaking. On Bland, the issue was they deprecated the voice that I was using. I don't think that's the case here since I've tried a few different open ai voices and none are working. It would be super helpful to have some sort of alarm or failure to notify that this is happening consistently so I know to change the model stack (maybe an "Agent Started Speaking" metric that I can alarm on? Maybe I have to do that myself by checking whether the agent got anything into the transcript).
Also, I'm having the same problem when trying to switch it to the sesame voice. Here is a call ID for that
52ee9bcf-7953-4bad-a807-7af56d8ff00a
The note that I have that could be helpful in the investigation:
About a month ago I was trying to use the realtime API with a VAPI generated number. Whenever I would call that number, the voice sounded really slow and deep. I think there was something going on with the audio conversion. It reminded me of when I implemented audio sample rate/format conversion manually for the gemini live API and screwed something up (I think it was converting Google's 16bit PCM 24kHz audio to the ULaw audio that twilio requires). Problem was fixed when I switched to a twilio number. Point being, it sounds like you guys are doing some sort of audio format conversion to match the model outputs with whatever phone provider. It's possible that a bug got introduced at that level and that's messing with the inputs to the models.
Also, I'm having the same problem when trying to switch it to the sesame voice. Here is a call ID for that
52ee9bcf-7953-4bad-a807-7af56d8ff00a
The note that I have that could be helpful in the investigation:
About a month ago I was trying to use the realtime API with a VAPI generated number. Whenever I would call that number, the voice sounded really slow and deep. I think there was something going on with the audio conversion. It reminded me of when I implemented audio sample rate/format conversion manually for the gemini live API and screwed something up (I think it was converting Google's 16bit PCM 24kHz audio to the ULaw audio that twilio requires). Problem was fixed when I switched to a twilio number. Point being, it sounds like you guys are doing some sort of audio format conversion to match the model outputs with whatever phone provider. It's possible that a bug got introduced at that level and that's messing with the inputs to the models.
KThank you for providing more context and sending the email to our support channel. I see your email thread and I will close this ticket so we can continue the conversation via email.
WBut my issue is not resolved
WCall ID b0da7f3c-cfdc-4ef6-bcf5-1fde97cfef9b
K