Agent Benchmark Contamination and Scaffolding Sensitivity: The Four Discounts in Every Agent Score

Two numbers landed this week, and they cannot both mean what vendors want them to mean. OpenAI's agents just crossed 10 million users. And on a new benchmark of real long-horizon terminal tasks, the strongest frontier model completed 15.2% of them.

Agent benchmark contamination and scaffolding sensitivity explain how both numbers coexist. Benchmark scores are not estimates of production performance. They are ceilings, quoted with discounts that production will not honor. Three papers published this month, read together, itemize those discounts with unusual precision, and they change how I think enterprises should price any agent score a vendor or an internal team puts in front of them.

I work with Fortune-level enterprise clients architecting and building agentic frameworks, and the pattern I keep seeing is the same one these papers measure: the agent that cleared the eval gate stumbles in week three of the pilot, and nobody can say which assumption in the eval was the one that lied.

Why do agent benchmark scores collapse in production?

Start with the three studies.

AgentGym2 built its evaluation around what it calls de-idealized environments: noisy inputs, underspecified requests, tools the agent has to discover rather than receive. Across 15 proprietary and open models, frontier systems including Gemini and GPT-5 struggled. The authors' diagnosis is blunt: most existing benchmarks hand the agent pre-packaged tools, pre-solved intermediate steps, and clean inputs, and therefore systematically understate real-world difficulty.

Long-Horizon-Terminal-Bench measured what happens when tasks stop being demo-length. Its 46 terminal tasks run minutes to hours, averaging around 9.9 million tokens, 231 episodes, and 85 minutes of execution each. The best frontier model passed 15.2% of tasks. The mean across 15 models was 4.3%.

PolyWorkBench tested 67 multilingual workplace tasks across five enterprise domains. The headline result is not the multilingual degradation, real as it is. It is that the same model swung 8 to 24 Pass@1 points depending on which harness ran it. Claude Opus 4.8 scored 0.921 on one scaffold and 0.712 on another. Nothing about the model changed.

Now hold those three results against the adoption curve. Ten million users. Gartner projecting that more than 40% of agentic AI projects will be canceled by the end of 2027, citing cost, unclear value, and inadequate risk controls. The gap between the adoption number and the capability numbers is exactly the gap the benchmarks are papering over.

What is structural contamination?

We usually define benchmark contamination as leaked data: the test questions were in the training set. Agent benchmarks have a subtler failure mode that I would call structural contamination. The benchmark does not leak the answers. It leaks the structure of the solution.

When a benchmark pre-wires the right tools into the agent's context, it has answered the hardest question in the task: what capabilities does this problem need? When it decomposes a task into graded steps, it has done the planning. When it feeds clean, fully specified inputs, it has done the requirements gathering. The agent is graded on execution while the benchmark quietly performs the cognition that production would demand of it.

That is contamination in every sense that matters for a deployment decision. The score transfers only to environments that perform the same favors. Production performs none of them.

How do the Four Discounts inflate an agent's score?

Read the three papers as a single audit and you get four itemized discounts baked into a typical agent benchmark score. I use this as a checklist when someone shows me an eval result.

1. The input discount. Benchmark inputs are clean and fully specified. Production inputs are noisy, partial, and sometimes contradictory. AgentGym2 shows performance dropping precisely when requests become underspecified, which is the normal condition of enterprise work.

2. The tool discount. Benchmarks hand the agent the right tools, already wired. AgentGym2's tool-discovery and tool-composition tests, where agents must explore for capabilities and combine them for unseen tasks, are where state-of-the-art systems degraded most. Production tool surfaces are large, partially documented, and change under you.

3. The horizon discount. Most evals are demo-length. Long-Horizon-Terminal-Bench is what happens when the demo keeps going: 85 minutes, 231 episodes, and a best score of 15.2%. Reliability decays with the horizon, and short-horizon evals never sample the decay region. If your production workflow runs for an hour, a ten-minute eval has told you almost nothing.

4. The harness discount. PolyWorkBench's 8 to 24 point scaffold swings mean a benchmark score is a property of the model-harness pair, never the model alone. Unless the eval ran on your harness, with your context assembly and your recovery logic, the number describes a system you do not operate. This is the same lever I wrote about in my harness comparison breakdown; scaffolding sensitivity is that finding read as an eval-validity problem instead of a cost problem.

A 0.9 on a leaderboard, discounted four times, is how you get 40% project cancellations at 10 million users. Neither number is wrong. One of them is just quoted before tax.

What should enterprises do before trusting an agent benchmark?

The fix is not to abandon benchmarks. It is to price the discounts explicitly, the way you would audit any number that feeds a capital decision.

What I'd actually do:

First, re-run any vendor or leaderboard eval on your own harness before you let it into a business case. PolyWorkBench says the harness alone can move the score by more than the gap between the top five models. If the number was produced on someone else's scaffold, treat it as marketing until reproduced.

Second, extend one representative eval task to production length. Not fifty tasks. One task, run at the real horizon with real interruptions. The horizon discount is invisible until you sample it, and one honest long-horizon run tells you more than another hundred demo-length passes.

Third, de-idealize your inputs deliberately. Take your existing eval set and remove the favors: strip a required parameter, add a stale document, include a tool that looks right but is wrong. AgentGym2 open-sourced its framework precisely so teams can do this internally. The delta between your clean score and your de-idealized score is your input and tool discount, measured instead of guessed.

Fourth, put the discounted number, not the headline number, in the deployment decision. If the benchmark says 90% and your de-idealized, own-harness, long-horizon runs say 60%, then 60% is the number the rollout plan has to survive. This is the same discipline I argued for in evaluating AI agents in production: the eval you trust is the one that shares an environment with the failure you fear.

The teams shipping agents successfully are not the ones with the best leaderboard positions. They are the ones who know their real number and built the workflow around it.

---

If this was useful, I write about moving enterprise agents from pilot to production every week. Subscribe to the newsletter at buildnextgen.ai for the next breakdown.

This post is part of the pillar guide on deploying AI agents in production, which covers the four properties a pilot never tests.