Same model, 30 points apart

Same model. 30.4 points apart.

That is the number I care about most in Databricks' Grounded Reasoning Cup results.

Not Stanford winning.
Not the frontier-agent baselines.

The 30.4-point average gap between the highest- and lowest-scoring teams using the same model.

Because the model was the same. What changed was everything around it: parsing, retrieval, tools, skills, verification, parallelism, retry logic, operational infrastructure.

We have spent the last few years talking about AI capability largely through model names. Which model is better? Which one scores higher? Which one reasons better?

Once models start operating inside agent systems, that question becomes incomplete.

Model capability is no longer system capability. It is one of its inputs.

This wasn't a normal benchmark

The Grounded Reasoning Cup brought together 11 academic teams from the US and Canada. For roughly two months they developed agent systems using OfficeQA, Databricks' benchmark for grounded reasoning over complex enterprise-style documents. Each team was paired with OpenAI, Anthropic or Google DeepMind and restricted to models from its assigned lab's family.

Then Databricks moved the ground underneath them.

36 hours before the live competition, teams received a new benchmark built around the US Treasury's Accounts of Receipts and Expenditures — roughly 120,000 pages of historical Treasury documents. The competition ran across six increasingly difficult 15-minute rounds, 15 questions per round.

That matters because the teams were not being tested on the environment they had spent months optimizing against. They had to transfer what they had built to a new corpus, under time pressure.

And the corpus was ugly in exactly the way real enterprise data is ugly.

Historical documents.
Changing representations.
Tables.
Financial terminology.
Different structures across different periods.
Questions needing evidence scattered across several documents.

This was not:

Find this sentence in a PDF.

It was closer to:

Find the right evidence across a messy document collection, reconcile it, calculate what needs calculating, verify it, and return the answer before everyone else does.

That is a much more interesting agent problem.

Stanford won, and that is not the interesting result

There is an obvious headline hiding in that table:

Stanford's custom agent system beat frontier agents by more than 30 points.

I wouldn't use it. Databricks explicitly warns these conditions are not comparable. The baseline agents ran offline with frozen configurations across the test set. Competition teams operated in rounds — they could observe failures, change strategies and modify their systems between rounds.

So 63.3 against sub-30 is not a clean measurement of harness uplift.

Fortunately there is a much more interesting number. Databricks reports that the average gap between the highest- and lowest-scoring teams using the same model was 30.4 points.

One caveat worth stating plainly: that is a best-to-worst range, not an average uplift. It does not mean a better harness hands a model 30.4 points.

What it does tell us is arguably more important:

The same model can produce radically different capability depending on the system built around it.

Three top teams, three different systems

Stanford, UMass Amherst and Yale finished at the top — and did not converge on one architecture.

What the model decides

• how well it reasons
• how it uses a tool it is handed
• how it behaves over long context

What the system decides

• what the model sees at all
• which representation it reads
• which tools exist
• what it remembers
• when it verifies
• when it retries
• when it stops

Stanford built around reusable skills and adaptive verification. UMass optimized heavily for speed and parallelism. Yale built multiple independent reasoning paths with a meta-verifier selecting among their answers.

There is no single agent loop explaining the leaderboard.

Stanford built 100+ skills

During development Stanford repeatedly traced incorrect answers back to the exact point where its agent failed.

Did it fail to locate the right table?
Misinterpret financial terminology?
Format the answer incorrectly?
Lose context in the parsed representation?
Handle unit scaling wrongly?

Instead of adding more instructions to one giant prompt, the team converted recurring failure patterns into reusable skills. By competition day Stanford had a playbook of roughly 100+ skills.

"Skill library" is technically correct, and it undersells what they built. I would call it compiled failure memory.

The agent fails Trace the root cause Extract the pattern Write it down as a skill The next run starts with it

That is memory, whether the artifact is called a memory, a rule, a skill or a procedure.

And Stanford didn't only teach the agent what to look for. Some skills helped it decide how to look — search the parsed corpus text, use a Markdown-style document representation, or fall back to the source PDF when parsing had lost important context.

A strong agent system does not just contain more knowledge. It knows which representation and which tool suit the problem in front of it.

But memory has a scope problem

The competition also exposed the uncomfortable half of that story. Techniques the teams developed on OfficeQA did not always transfer reliably to the new corpus.

That should matter to anyone building agent memory, because storing knowledge is the easy problem. The hard one is knowing where that knowledge applies.

A skill can be useful in one corpus and harmful in another.
A decision can be right for one repository and wrong for another.
A procedure can work under one tool environment and be obsolete when the environment changes.
A lesson learned yesterday does not automatically remain true tomorrow.

This is where a lot of current agent-memory thinking feels incomplete to me. We focus on remembering: what did the agent learn, what did we decide, what happened last session, what should persist.

But persistence alone is not enough. Memory needs applicability. It needs scope. Eventually it probably needs some notion of validity decaying as the environment changes.

The Grounded Reasoning Cup demonstrated this almost accidentally. Teams spent months learning. Then Databricks changed the distribution, and some of that learning stopped transferring.

An agent that remembers everything but does not know where its memories apply can become confidently wrong because of its own experience.

That is a much harder problem than storing another vector.

UMass optimized for something else entirely

Its primary model was Claude Opus 4.8 Fast. The team preprocessed the corpus into a metadata catalog built for rapid search and filtering. For every question, three agents ran in parallel, and a final Opus call selected the best answer.

The result was speed.

By halftime UMass led Stanford by 10.25 points, and entered the final round still 3.75 ahead. Then the scoring environment changed: the final round held the hardest questions and was worth double. Stanford's slower, accuracy-oriented system caught up, and UMass held the lead until the final 56 seconds. Stanford won by 1.75 points.

There is an agent-engineering lesson inside that finish. There is no universally optimal architecture. There is an objective function.

Optimize for latency

Preprocess into a catalog Three agents in parallel Four minutes

Optimize for accuracy

Skills for known failures A second agent verifies Fewer wrong answers

In a benchmark that objective function is the scoring system. In production it is your SLA, your cost structure and your risk profile.

Stanford won partly by turning verification off

This is my favourite detail of the whole competition.

For the first three rounds Stanford ran a second Claude Code agent as a verifier: independently checking intermediate values, tracing revised values through data lineage, catching unit-scaling problems, patching calculations when it found inconsistencies.

That sounds like an obvious improvement. It was. It was also slow — Stanford earned only two speed bonuses across those rounds.

So they did something that sounds wrong if you think of an agent as a static system. They turned verification off. For the final three rounds the extra pass was gone, latency dropped, and the team earned speed bonuses on 14 questions.

Then came the hardest part of the competition, and Stanford turned the verifier back on. In the final round it helped correct an answer through one of the resubmissions.

The lesson is not that verification is good, nor that it is too expensive.

The value of verification depends on the task. On easy questions the chance of catching an important error may not justify the latency. On hard ones that trade-off reverses.

So verification does not have to be a permanent layer in the pipeline. It can be a runtime decision.

Which points at something broader. Maybe the next generation of agent systems will not only have better tools, retrieval, memory and verification. Maybe they will get better at deciding when those capabilities are worth using.

Retrieval tells the same story

The strongest systems did not dump documents into embeddings and do generic top-k chunk retrieval. Databricks specifically notes lexical retrieval such as grep, often combined with dense retrieval in hybrid approaches.

That sounds boring next to the rest of modern AI infrastructure. It shouldn't. If an agent is looking for an exact historical account name, an identifier, a date, a financial term or an unusual phrase, semantic similarity is not automatically the right abstraction. Sometimes grep is better.

This exposes a recurring mistake in agent architecture: we choose the abstraction first — vector database, RAG, knowledge graph, long context, whatever the current default is — and then force every problem through it.

The strongest systems here did something more practical. They used different mechanisms for different problems. Parsing when parsing was enough. Lexical retrieval when exact search mattered. Dense retrieval when semantic similarity helped. Source PDFs when the transformed representation had lost information. Specialized tools for calculation and comparison.

The harness was not just connecting the model to data. It was deciding how the model should interact with that data.

The boring parts were capability too

Databricks also calls out something that disappears from agent demos: operational infrastructure. Retry logic. Parallelism. Preprocessing. Submission scaffolding.

In a live competition a correct answer that does not arrive on time is not a correct answer. A reasoning chain that dies because one operation failed is not useful capability. A system that knows the answer but cannot reliably submit it has still failed.

This is where the boundary between "AI capability" and "engineering" gets fuzzy.

If retry logic determines whether the task succeeds, retry logic is capability.
If parsing determines whether the model receives the evidence it needs, parsing is capability.
If retrieval determines what the model knows at inference time, retrieval is capability.
If verification determines whether a wrong answer escapes, verification is capability.
If memory keeps yesterday's mistake from repeating, memory is capability.

These are not infrastructure surrounding the intelligence any more. Together they determine how much of the model's intelligence becomes usable.

Distribution shift may be the more important benchmark

Two months to optimize on OfficeQA. A new corpus 36 hours before the live event. And techniques that did not always transfer.

That is exactly the failure a normal leaderboard hides. A system can become extremely good at a benchmark without becoming equally good at the underlying class of problems.

So when I look at an agent benchmark now, I care about a second question. Not only how high did it score, but:

What happens when the ground moves?

Change the corpus. Change the document structure. Change the available tools. Change the task distribution. Change the latency budget.

Do the learned strategies survive? Does retrieval still work? Do stored memories still apply? Does the harness know some of its previous assumptions are no longer valid?

That is much closer to what happens when an agent leaves a benchmark and enters a real company. Production is one giant distribution shift.

"Which model?" is becoming an incomplete question

Agent leaderboards still push us to think in model names. Agent A uses model X, agent B uses model Y, and we compare scores.

But agent A may have different parsing, retrieval, tools, context construction, verification, retry policies, parallelism, memory, stopping rules — even a different number of model calls per task.

At that point, what exactly are we benchmarking? The model, or the system?

The Grounded Reasoning Cup does not prove harness engineering is "worth 30 points" — that would be an abuse of the result. What it shows is that 30.4 points of separation can exist even when the model is held constant.

That is a very different claim, and a much more interesting one.

Model capability is an input. System capability is the result.

None of this means models stopped mattering. A stronger model raises the ceiling. Better reasoning matters, better tool use matters, better long-context behaviour matters.

But choosing the model no longer finishes the capability discussion. It starts it.

After that choice you still decide what the model sees, what it can search, which representation it uses, which tools it can call, what it remembers, where that memory applies, when it verifies itself, when it retries, when several agents should run in parallel, when the system should choose speed over confidence, and what happens when the environment changes.

Those decisions do not live in the model weights. They live in the system.

So if two agent systems using the same model can still end up 30.4 points apart, asking only which model it uses is no longer enough.

The more interesting question is:

What did you build around it?

Model capability is an input.
System capability is the result.

Sources

• Databricks — Evaluating AI Agents Live at the Grounded Reasoning Cup (August 18, 2026). Primary source for the competition setup, the 30.4-point same-model gap, Stanford's 63.3% result, the 100+ skills, Stanford's adaptive verification strategy, UMass parallelism and latency, Yale's multi-arm system, the retrieval findings, and the warning that offline baselines are not directly comparable with the live competition.
• OfficeQA on GitHub — the public evaluation repository: benchmark documentation, scoring code, corpus tooling and evaluation setup.
• OfficeQA on Hugging Face — the public benchmark and corpus. Access to the benchmark files is gated to reduce contamination and artificial score inflation.