Benchmarks usually answer "how smart is this model?" GateBench Airline asks a narrower, more useful question: can it run a multi-step tool workflow without falling over? The scenario is a fictional airline support agent — rebookings, fare rules, loyalty edge cases — with real tool schemas, adversarial customers, and a rubric scored by a three-model council with majority rules.
Method, in brief
Every model in the catalog runs the identical 240-task set through the same gateway path customers use — same retries, same timeouts. We publish three numbers per model: task success (did the workflow reach the correct end state), effective success (success after gateway-level retry and failover, which is what you actually experience), and cost per solved task, which is the number the finance team should be reading.
What surprised us
The frontier ranks roughly as you'd expect. The interesting layer is the middle: several small, cheap models solve 80%+ of tasks at a twentieth of the cost per solve, which makes them excellent council reviewers and first-draft engines. And one household-name model aces reasoning tasks while quietly fumbling tool-call formatting under pressure — invisible in chat, decisive in agents.
Placeholder honesty
All figures on the benchmarks page are labeled placeholder data until the public harness ships; the methodology is what we're committing to here. When it lands, every score links to its raw transcripts — a leaderboard you can't audit is an ad.