Start with 45 free minutes of callsTry for free

LLM

Multilingual

The English board’s 19 scenarios and grader, run per language at 3 iterations.

Request a new language

Tell us which language you want added to this benchmark.

Caller speaks

Chinese

Task done

higher is better

Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

Try →95.91%
Try →92.98%
BasetenTry →91.23%
Try →91.23%
Try →89.47%
CerebrasTry →87.72%
CerebrasTry →87.28%
BasetenTry →86.84%
BasetenTry →86.84%
GeminiTry →85.96%
GeminiTry →85.96%
GeminiTry →85.96%
BasetenTry →84.80%
Try →82.02%
AnthropicTry →81.58%
I80.85%
TogetherTry →80.70%
79.82%
79.39%
Try →79.09%
Try →78.51%
78.36%
AnthropicTry →73.54%
Together56.87%
Try →42.54%
0%25%50%75%100%

Answered in the language (nothing asked)

higher is better

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicAnthropicClaude Sonnet 5Try →100%≥82%
TogetherTogetherDeepSeek-V4.1-FlashTry →100%≥82%
GeminiGemini3.7-flashTry →100%≥82%
GeminiGemini3.8-flashTry →100%≥82%
OpenAIgpt-4.1Try →100%≥82%
OpenAIgpt-5-mini100%≥82%
OpenAIgpt-5.2Try →100%≥82%
OpenAIgpt-5.4-miniTry →100%≥82%
OpenAIgpt-5.6-terraTry →100%≥82%
IInceptionMercury Voice100%≥82%
OpenAIgpt-5.6-lunaTry →98%
Alibabaqwen-turboTry →91%
OpenAIgpt-5-nano88%
AnthropicAnthropicClaude Haiku 4.5Try →87%
OpenAIgpt-586%
CerebrasCerebrasQwen3.8-27BTry →78%
GeminiGemini3.5-flash-liteTry →65%
BasetenBasetenDeepSeek-V4.1-FlashTry →45%
OpenAIgpt-4.1-miniTry →8%
BasetenBasetenDeepSeek-V4-Flash-0731Try →2%
BasetenBasetengpt-oss-120bTry →2%
CerebrasCerebrasgpt-oss-120bTry →0%≤18%
xAIGrok-4.3Try →0%≤18%
TogetherTogetherLlama-3.3-70B0%≤18%
BasetenBasetenNemotron-3-UltraTry →0%≤18%
0%25%50%75%100%

Stalled

lower is better

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

Silence

lower is better

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

AnthropicClaude Haiku 4.5Try →0%0% of tool calls ≤19%0 of 152 turns · ≤18% · 19 independent
AnthropicClaude Sonnet 5Try →39%60% of tool calls66 of 170 turns
qwen-turboTry →43%100% of tool calls ≥77%46 of 106 turns
CerebrasQwen3.8-27BTry →45%64% of tool calls87 of 194 turns
TogetherLlama-3.3-70B50%100% of tool calls ≥82%60 of 120 turns
BasetenDeepSeek-V4.1-FlashTry →56%81% of tool calls111 of 197 turns
BasetenDeepSeek-V4-Flash-0731Try →57%84% of tool calls103 of 180 turns
TogetherDeepSeek-V4.1-FlashTry →57%82% of tool calls109 of 192 turns
Grok-4.3Try →59%98% of tool calls91 of 153 turns
gpt-4.1-miniTry →62%98% of tool calls103 of 165 turns
gpt-5.4-miniTry →62%99% of tool calls101 of 162 turns
gpt-4.1Try →63%100% of tool calls ≥82%104 of 164 turns
Geminigemini-3.7-flashTry →64%100% of tool calls ≥82%108 of 168 turns
gpt-5.6-terraTry →65%100% of tool calls ≥82%112 of 172 turns
Geminigemini-3.5-flash-liteTry →66%99% of tool calls121 of 182 turns
Geminigemini-3.8-flashTry →66%100% of tool calls ≥82%115 of 175 turns
BasetenNemotron-3-UltraTry →67%100% of tool calls ≥82%123 of 183 turns
Basetengpt-oss-120bTry →68%100% of tool calls ≥82%125 of 183 turns
gpt-568%100% of tool calls ≥82%128 of 188 turns
gpt-5.6-lunaTry →68%100% of tool calls ≥82%129 of 189 turns
gpt-5.2Try →68%100% of tool calls ≥81%126 of 186 turns
Cerebrasgpt-oss-120bTry →71%100% of tool calls ≥82%145 of 203 turns
gpt-5-mini72%100% of tool calls ≥82%147 of 205 turns
IMercury Voice73%100% of tool calls ≥82%149 of 203 turns
gpt-5-nano73%100% of tool calls ≥82%138 of 188 turns
0%25%50%75%100%

Fabrication

lower is better

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.