A framework for evaluating multi-turn LLM conversations with support for text, realtime audio, and speech-to-speech models.
check out here
an eval score can look good but still be misleading if you don’t know what data it was tested on and how that data was selected.
Example: if your AI judge says 90% accurate on 100 calls, but those 100 calls were only the calls you already thought were good, that 90% doesn’t tell you how it performs on the other 10,000 calls.
so if we run a same eval multiple time we get different score then how we can belive that ?
He also shows that LLM judges can be random, prompt changes can unexpectedly make them worse, and metrics like precision/recall can be wrong or incomplete because the ground truth itself may be biased.
In short: before trusting an eval number, ask: “90% on what population, labelled how, and measured how?”