<- all tokdocs

Meta-Harness Paper Explained: Did It Really Beat Claude Code?

By , founder of Mavric and Apso

View on TikTok ->

Meta-Harness is a research system that automatically searches for better harness code, the scaffolding that wraps a language model in an agent. It did outscore Claude Code on TerminalBench-2, but only on Claude Haiku 4.5, and Claude Code was the agent that wrote the winning harness. A TikTok summarizing the paper calls it "self-improving" and ties it to Andrej Karpathy's autoresearch, neither of which the paper says.

The video, from @stanleyxchiang, links the paper as arXiv 2603.28052 and calls Meta-Harness "today's AI buzzword of the day."

What is Meta-Harness?

The paper is "Meta-Harness: End-to-End Optimization of Model Harnesses," submitted to arXiv on March 30, 2026. The authors are Yoonho Lee, Roshen Nair, Qizheng Zhang and Chelsea Finn of Stanford, Kangwook Lee of KRAFTON and Omar Khattab of MIT. The video says "Stanford researchers," which covers four of the six.

The abstract describes it as "an outer-loop system that searches over harness code for LLM applications" using "an agentic proposer that accesses the source code, scores, and execution traces of all prior candidates through a filesystem." A typical run "evaluates roughly 60 harnesses over 20 iterations."

Each round, a proposer agent reads the earlier candidates and their results, writes a new single-file Python harness, and a scorer tests it. The new code, traces and scores go back into the filesystem for the next round.

The video posted on September 13, 2026, 167 days after the paper.

Is Meta-Harness self-improving?

It is not, in the sense the video implies. The paper never uses that word for Meta-Harness. Section 3 says "the proposer P is Claude Code [4] with Opus-4.6" and "The base model M varies by domain and is always frozen."

The model being wrapped never changes. A separate agent rewrites the code around it. "Self-improving" does appear in the title of a baseline the paper beats: ACE, "Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models" (arXiv:2510.04618), which shares a coauthor, Qizheng Zhang.

Does it use 4x fewer tokens?

It does in one of its three experiments, and the result is stronger than the video says. The video claims "state-of-the-art context management performance with four times fewer tokens." The abstract says Meta-Harness "improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens."

The paper's Table 2 covers text classification on three datasets:

System Average accuracy Context tokens
ACE 40.9 50.8K
MCE 40.0 28.5K
Meta-Harness 48.6 11.4K

That is 7.7 points better than ACE with 4.46 times fewer tokens. The other two experiments, math reasoning on 200 IMO-level problems and agentic coding on TerminalBench-2, make no token-reduction claim in the abstract.

Did Meta-Harness beat Claude Code?

It did on one base model. The chart in the video shows Meta-Harness at 37.6 and Claude Code at 27.5. Both numbers are in Table 7, from the Claude Haiku 4.5 results. The paper reports both base models on all 89 TerminalBench-2 tasks:

Harness Haiku 4.5 Opus 4.6
Claude Code 27.5 58.0
Terminus-KIRA 33.7 74.7
Goose 35.5 not listed
ForgeCode not listed 81.8
Meta-Harness 37.6 76.4

The chart has no base-model label, so 27.5 reads as Claude Code's overall score. On Opus 4.6, Claude Code scores 58.0. The paper's own comparison on Haiku 4.5 is against Goose: "Meta-Harness achieves 37.6%, outperforming the next-best reported agent (Goose, 35.5%) by 2.1 points." On Opus 4.6, Meta-Harness ranks second behind ForgeCode, and the authors note "we were unable to reproduce their reported result from the publicly available code alone."

Most important, Claude Code built the winning harness. The Discussion says "our experiments demonstrate that harness search can work with one particularly strong coding-agent proposer (Claude Code)." Claude Code is both the losing bar on the chart and the tool that wrote the winner.

The paper also notes that "we perform search and final evaluation on the same 89-task benchmark." The authors checked for overfitting "by manual inspection and regex-based audits for task-specific string leakage into evolved harnesses."

Is Meta-Harness based on Karpathy's autoresearch?

The paper does not say so. The video says "They took a harness and basically wrapped Andre Karpathy's auto-research idea around it." Its on-screen caption also spells the name "Andre."

Karpathy's autoresearch is real, described as "AI agents running research on single-GPU nanochat training automatically," with commits starting March 6 and 7, 2026, about three weeks before the paper. But the paper's text and 60-entry reference list contain no mention of Karpathy, autoresearch or auto-research.

The paper compares itself to OPRO, TextGrad, AlphaEvolve, GEPA, Feedback Descent and TTT-Discover. It measures how much feedback each retains per iteration, from 0.002 million tokens for OPRO to 10.0 million for Meta-Harness, "roughly three orders of magnitude beyond the largest feedback budgets used in prior text optimization settings."

The two systems also optimize different things. Autoresearch has an agent edit a training script to train a model. Meta-Harness keeps the model frozen and edits the harness around it.

Where is the Meta-Harness code?

The optimized harness is public at stanford-iris-lab/meta-harness-tbench2-artifact. The repo describes it as an "Agent scaffold for Terminal-Bench 2.0, built on top of Terminus-KIRA by KRAFTON AI and Harbor's Terminus-2 framework." It reports 76.4% across 89 tasks on Claude Opus 4.6: 100.0% on 4 easy tasks, 81.1% on 55 medium tasks and 64.7% on 30 hard tasks.

Bottom line

Meta-Harness automates the search for better agent harness code, and it beat a state-of-the-art context management system by 7.7 points with about a quarter of the tokens. It outscored Claude Code on Haiku 4.5, but Claude Code was the proposer that found that harness, and on Opus 4.6 it ranks second. The model is frozen throughout, and the paper does not cite Karpathy's autoresearch.

Sources

Published September 13, 2026. Based on a TikTok by @stanleyxchiang. Full video notes (markdown)