national-gold•4h ago
Evaluation Error: Expected 2 assistant messages but got 1
Hi team,
I'm working on creatign an evaluation suite for our assistants and frequently running into issues with the turn based evals, "Expected 2 assistant messages but got 1" or varying.
Question: Considering agents are non-deterministic, how can I implement an eval that doesn't fail if the agent says 1 extra line or call tools at different times?
Eg. Sometimes the agent will call a tool to verify patient immediately upon receiving all patient details, sometimes the agent may say "let me check your details" then call the tool.
Are evals only built for deterministic testing? Should we be using test suites?
1 Reply
Hi there,
Thank you for your message. Our team is currently out of the office. We operate Monday through Friday, from 9:00 AM to 8:00 PM Pacific Standard Time (PST).
We’ll get back to you as soon as possible during our normal business hours.
If your message is urgent, please mark it accordingly or include “URGENT” in the subject line, and we’ll do our best to respond promptly.
Warm regards,
Vapi
Customer Support Team
Vapi
Customer Support Team