Agent Harness Comparison: The 10x Cost Gap Lives in the Harness, Not the Model

Most enterprises still evaluate agents by running a model bake-off. This week produced the clearest evidence yet that an agent harness comparison is the evaluation that actually matters. LangChain and NVIDIA launched the NemoClaw Deep Agents blueprint, and the headline number is hard to ignore: an open model with a tuned harness scored 0.86 on LangChain's agent eval suite at $4.48 per task. The next closest model cost $43.48. Same quality tier, roughly 10x the price.

The 10x gap did not come from a better model. It came from tuning the layer around the model. That inverts how most agent programs currently spend their evaluation budget, and two new research results say the inversion is overdue.

Why did a $4.48 open-model stack beat a $43.48 frontier stack?

NemoClaw is three layers tuned together: NVIDIA's open Nemotron 3 Ultra model, LangChain's Deep Agents Code harness, and OpenShell, a sandboxed, policy-governed runtime. The result is not "open model catches up to frontier model." The result is that a harness tuned around a specific model beat stacks where a stronger model sits inside a generic harness.

LangChain's stated thesis matches what the numbers show: agent performance improves when model, harness, evals, and runtime are tuned as one system. And the cost delta compounds. At $4.48 a task you can afford 10x the eval runs, 10x the variant comparisons, and specialized agents that would be uneconomic at $43.48. Cheap inference is not just a procurement win; it changes how much engineering feedback you can buy.

The strategic claim underneath is more pointed: your agent memory, traces, eval datasets, and harness configuration are proprietary IP. The model is the commodity input. That is the opposite of how most enterprises currently assign value, where the model contract gets executive attention and the harness is an implementation detail.

What is an agent harness, exactly?

The harness is everything that shapes the model's behavior at runtime: what the agent sees, what it is allowed to do, which failures get repaired, and what gets verified and logged. I think of it as four dials.

Sight — what context the agent gets and how it is disclosed. Scope — which tools and actions are permitted, and under what policy. Recovery — which failures are retried, repaired, or compacted away. Proof — what is verified, evaluated, and written to the trace.

Call it the Four Dials. Every harness sets all four, whether you chose the settings or inherited them from a framework default. Two papers released this month show how much these dials move outcomes.

On the Sight dial: a controlled study of progressive disclosure (arXiv:2607.17598) tested the pattern behind most skill-loading and agent-memory designs — give the agent a path and let it decide what to read. The finding: benefits are harness-dependent. When the harness already retrieves well, progressive disclosure adds nearly nothing. When the corpus outgrows the context window, one level of disclosure wins decisively and a second routing level never helps. The authors' phrase is exact: disclosure "buys context, not intelligence." The same pattern is a win or a no-op depending on the harness around it.

On Recovery and Proof: a University of Toronto and Emory study (arXiv:2607.04528) shows the harness changes what an agent believes mid-task even with the task, environment, and model held fixed. Blocked actions, compressed repair traces, and selective verification can preserve terminal success while silently altering the beliefs that drive later steps. The harness is not plumbing. It is an experimental variable, and most enterprise evals never control for it.

What should an agent harness comparison actually measure?

If the harness drives a 10x cost gap and measurable belief divergence, then the bake-off worth running holds the model constant and varies the harness. Concretely, compare candidate harnesses on the same task suite and score four things: task quality per dollar, not per token, since cheap inference buys eval volume; behavior at corpus scale, because raw-document navigation collapses well before disclosure-based patterns do; belief stability across the run, using trace audits of what the agent asserts about its own progress and constraints; and evidence quality, meaning whether the trace is complete enough to attribute a failure to Sight, Scope, Recovery, or Proof.

This is also where the field data lands. The 37% gap between benchmark and field performance reported across 6,259 deployed agents is exactly what you would expect when benchmarks are run in one harness and production runs in another. I wrote last week that enterprise agents fail as a runtime problem, not a model problem; the harness evidence sharpens that: the runtime layer is not just where agents fail, it is where quality and cost are set. It is also why evaluating agents in production has to happen inside the production harness, not a lab scaffold.

Does harness tuning lock you into one model?

The fair objection: if the harness is tuned around Nemotron, is this just lock-in at a different layer? Partly, yes — a tuned harness is coupled to its model's failure modes. But the assets you build while tuning transfer: the eval suite, the trace schema, the policy definitions, the task decompositions. Those are model-portable even when the tuned settings are not. The teams that own those assets can re-tune around a new model in weeks. The teams that outsourced the harness to a framework default start over.

I work with Fortune-level enterprise clients architecting and building agentic frameworks, and the pattern is consistent: the programs that treat harness configuration, eval data, and traces as owned IP are the ones that can actually exploit a cheaper model when it arrives. The model-swap is easy. The harness is the accumulated learning.

What I'd actually do

Stop running model bake-offs first. Run an agent harness comparison first: hold the model constant, vary the Four Dials, and score quality per dollar at production scale. Fund the eval suite before the model contract. Treat harness config, traces, and eval datasets as proprietary IP with an owner and a roadmap. Then pick the model last — it is the most reversible decision in the stack.

The 10x number will get quoted as an open-model victory. It is a harness victory. Price your engineering attention accordingly.

Primary source: LangChain and NVIDIA launch the NemoClaw Deep Agents blueprint.

---

If this is the layer you are working in, my newsletter covers one enterprise agentic-AI decision like this each week — the orchestration, decisioning, and governance calls that get agents from pilot to production. Subscribe at buildnextgen.ai.