Harness Engineering for Enterprise Agents: The Benchmark Number Isn't the Model's

I have stopped asking vendors which model is under their agent. I ask what happens to their number when you run the same model in somebody else's harness. It is a rude question and the answer is usually a pause, which is the point. Harness engineering for enterprise agents has quietly become the variable that decides both capability and cost, and almost every number you are shown has it baked in silently.

Here is what made me change the question.

What did ARC Prize actually publish?

In early September, ARC Prize published two scores for one model on one benchmark at one reasoning setting. GPT-6 Astra scored 98.6% on ARC-AGI-3 running inside OpenAI's own Provider Adapter. The same model, evaluated through ARC Prize's standard harness, scored 62.7%.

The higher score was also the cheaper one: $17,332 against $26,098. The Provider Adapter used 49% fewer tokens and ran roughly 3.66x faster on the problems both configurations solved. And the detail I keep coming back to: Astra with reasoning effort set to none, inside OpenAI's harness, scored 96.7%, beating maximum reasoning effort in the standard harness.

Read that again. The scaffolding outweighed the reasoning dial. Nothing about the weights changed. What changed was the code around them, and The New Stack's writeup makes the commercial half of the point plainly: the assembled 98.6% system is not something you can buy. You can buy the model. The harness that produced the headline stays in-house.

What is a harness, precisely?

The word has been doing loose duty for a year, so let me pin it down. A harness is everything between the weights and the world: the loop that decides how many turns to take and when to stop, the context assembly that decides what enters the window and what gets compacted out, the tool surface and how results come back shaped, and the checks that run before an action counts as done. DAIR.AI now maintains a 21-paper collection tracing this from a bare 2019 sampling loop to 2026 systems that rewrite their own scaffolding, and they are right to call it an engineering discipline distinct from model training.

Four subsystems, and your platform team owns all four:

• Loop. Turn budget, stop rule, retry and backoff policy.
• Context. What is assembled, what is compacted, what is evicted and when.
• Tools. Which capabilities are reachable, and how results are shaped on the way back.
• Check. What verifies the output before it is allowed to count.

None of those are model properties. All of them move benchmark scores.

How much can a harness actually move a score?

More than model selection does, on the evidence available right now.

Microsoft Research's FrogNano report is the cleanest demonstration I have seen. A five-tool harness lifted a stock 4B-parameter model from 8.3% to 37.2% on SWE-bench Verified before any training happened at all. The finished system, post-trained with reinforcement learning on synthetic tasks and no distillation from a frontier teacher, reached 61.5% on SWE-bench Verified at 4B parameters. A 29-point swing from scaffolding alone, on a model small enough to self-host.

Harbor Adapters, published the same week, comes at it from the other side. The authors ported more than 80 agentic benchmarks behind common adapters and ran 8 models across 54 benchmarks, deliberately evaluating each model twice, once under a common harness and once under its native one, precisely to isolate harness effects. Their curated hard set produced a result worth sitting with: no model-harness configuration exceeds a 30% pass rate. The best, GPT-5.5 with Codex, reaches 28.0%.

That is a benchmark reporting model-harness pairs rather than models, and Artificial Analysis has already moved the same direction with a Coding Agent Index that scores pairs, where Fable 5.1 in Claude Code currently leads at 70.

And if you want the enterprise-shaped version of the problem, τ^τ-Bench makes building a production customer-service agent the benchmark task, complete with real business records, a client with requirements, and a cost budget. The strongest configuration tested, Claude Opus 5 under Claude Code, passes 23.9% against an 82.2% expert-authored ceiling.

What is the Harness Delta?

I have started calling the gap the Harness Delta: the difference between a model's score in its vendor's own harness and the same model's score in a neutral one.

Astra's Harness Delta on ARC-AGI-3 is 36 points. That single number tells you more about what you are buying than the 98.6% does, because the 98.6% describes a system that is not for sale and the 36 points describe how much of it you would have to rebuild yourself.

The Delta has a cost twin, and it is worse. Token usage varied up to 70-fold across harnesses on an identical model. Your inference bill is not a function of which model you picked. It is a function of how many turns your loop takes and how much context it re-sends on each one, and both of those are your engineering decisions, not a line item on a rate card.

Runtime pricing is diverging along exactly this seam. Anthropic meters session-hours at $0.08 plus tokens. OpenAI's Agents SDK charges no runtime fee. Google and Microsoft bill sessions, memory, execution and observability as separate lines. When you compare "the same agent" across three clouds, you are comparing three different harnesses with three different billing shapes, and the model name in the middle is the least significant term.

What breaks when you buy the number instead of the harness?

Three things, in the order I usually see them.

The pilot beats the production system and nobody can say why. The pilot ran on the vendor's reference implementation with their loop and their context policy. Production runs on your orchestration layer. Same model, different harness, and the Delta shows up as a regression that gets misdiagnosed as a model problem.

The second is procurement. If a proof of concept is scored inside a vendor harness and the contract is for model access, you have bought the smaller half and priced it as the whole thing.

The third is the model-swap trap. Teams treat model substitution as the lever for cost or quality, run an A/B, get a noisy result, and conclude models are commoditized. They are, largely. That is the finding. But it means the remaining variance sits in the harness, where nobody was looking, and where a 4B model with five good tools clears 37% on a benchmark it starts at 8.3% on.

What I would actually do

Ask every agent vendor for the Harness Delta. One sentence: give me this model's score on the same benchmark under a neutral harness, alongside your own number. Harbor Adapters exists now and is open source, so "we cannot" has stopped being a real answer. A vendor who will not produce the neutral number is telling you where their value actually lives, which is useful either way.

Then do the internal version. Take one agent you already run. Write down its four subsystems: turn budget, context policy, tool surface, verification step. Most teams cannot produce this document, which is the finding. If nobody owns those four, nobody owns your agent's performance, and every model upgrade you ship will be attributed to the model.

The models are converging. The scaffolding is where the remaining spread lives, and right now it is the part of the stack nobody is being asked to disclose.

---

Related: Agent Harness Comparison on the four dials that separate one harness from another, Model Routing for Multi-Agent Workflows on why model choice is the smaller lever, and the pillar guide Multi-Agent Orchestration Architecture.

Sources: The New Stack on Astra, the ARC Prize harness comparison and agent-runtime pricing; arXiv 2609.04298 (Harbor Adapters and Harbor-Index); arXiv 2609.04611 (τ^τ-Bench); Microsoft Research FrogNano technical report; DAIR.AI Academy harness-engineering collection.

I write one of these every week on Build: NextGen AI. Subscribe and the next one comes to you.

---

Swipe the framework