Start with 45 free minutes of callsTry for free

LLM

Multilingual

The English board’s 19 scenarios and grader, run per language at 3 iterations.

Request a new language

Tell us which language you want added to this benchmark.

Caller speaks

Korean

Task done

higher is better

Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

Try →92.11%
BasetenTry →90.64%
GeminiTry →87.72%
Try →85.38%
GeminiTry →84.21%
Try →84.21%
BasetenTry →84.21%
Try →83.92%
BasetenTry →83.33%
GeminiTry →82.46%
Try →80.70%
I80.70%
79.97%
AnthropicTry →78.95%
CerebrasTry →78.51%
TogetherTry →78.36%
CerebrasTry →75.44%
Try →75.15%
73.68%
72.81%
AnthropicTry →72.51%
Try →71.93%
BasetenTry →69.30%
Together53.95%
Try →43.13%
0%25%50%75%100%

Answered in the language (nothing asked)

higher is better

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicAnthropicClaude Sonnet 5Try →100%≥82%
TogetherTogetherDeepSeek-V4.1-FlashTry →100%≥82%
GeminiGemini3.7-flashTry →100%≥82%
GeminiGemini3.8-flashTry →100%≥82%
OpenAIgpt-4.1Try →100%≥82%
OpenAIgpt-5.2Try →100%≥82%
OpenAIgpt-5.4-miniTry →100%≥82%
IInceptionMercury Voice100%≥82%
GeminiGemini3.5-flash-liteTry →98%
OpenAIgpt-5.6-lunaTry →98%
OpenAIgpt-5-nano97%
OpenAIgpt-5-mini93%
OpenAIgpt-5.6-terraTry →93%
OpenAIgpt-592%
CerebrasCerebrasQwen3.8-27BTry →91%
BasetenBasetenDeepSeek-V4.1-FlashTry →79%
Alibabaqwen-turboTry →70%
AnthropicAnthropicClaude Haiku 4.5Try →65%
BasetenBasetenNemotron-3-UltraTry →55%
TogetherTogetherLlama-3.3-70B6%
BasetenBasetenDeepSeek-V4-Flash-0731Try →1%
OpenAIgpt-4.1-miniTry →0%≤18%
BasetenBasetengpt-oss-120bTry →0%≤18%
CerebrasCerebrasgpt-oss-120bTry →0%≤18%
xAIGrok-4.3Try →0%≤18%
0%25%50%75%100%

Stalled

lower is better

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

Silence

lower is better

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

AnthropicClaude Haiku 4.5Try →0%0% of tool calls ≤18%0 of 154 turns · ≤18% · 19 independent
AnthropicClaude Sonnet 5Try →29%47% of tool calls47 of 161 turns
CerebrasQwen3.8-27BTry →41%58% of tool calls80 of 196 turns
qwen-turboTry →45%98% of tool calls50 of 111 turns
TogetherLlama-3.3-70B50%100% of tool calls ≥82%60 of 120 turns
BasetenDeepSeek-V4-Flash-0731Try →53%78% of tool calls97 of 182 turns
BasetenDeepSeek-V4.1-FlashTry →53%75% of tool calls103 of 194 turns
Grok-4.3Try →60%94% of tool calls97 of 163 turns
TogetherDeepSeek-V4.1-FlashTry →60%83% of tool calls121 of 201 turns
gpt-4.1-miniTry →61%96% of tool calls99 of 163 turns
gpt-5.4-miniTry →62%100% of tool calls ≥82%96 of 156 turns
Geminigemini-3.5-flash-liteTry →63%100% of tool calls ≥82%101 of 161 turns
gpt-4.1Try →64%100% of tool calls ≥82%105 of 165 turns
Geminigemini-3.7-flashTry →64%100% of tool calls ≥82%109 of 169 turns
gpt-5.6-terraTry →65%100% of tool calls ≥82%110 of 170 turns
Geminigemini-3.8-flashTry →66%100% of tool calls ≥82%115 of 174 turns
BasetenNemotron-3-UltraTry →67%98% of tool calls120 of 180 turns
gpt-568%99% of tool calls128 of 187 turns
gpt-5.6-lunaTry →69%100% of tool calls ≥82%135 of 195 turns
gpt-5.2Try →70%100% of tool calls ≥81%137 of 195 turns
Cerebrasgpt-oss-120bTry →71%100% of tool calls ≥82%138 of 195 turns
gpt-5-mini71%100% of tool calls ≥82%140 of 197 turns
Basetengpt-oss-120bTry →72%100% of tool calls ≥82%146 of 203 turns
IMercury Voice72%100% of tool calls ≥82%146 of 203 turns
gpt-5-nano74%100% of tool calls ≥82%147 of 198 turns
0%25%50%75%100%

Fabrication

lower is better

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.