Harnesses Don't Make a Moat Until You Train an LLM Judge on Your Own Traces

Robert Ta opens with the take now circulating on Hacker News, in Y Combinator circles, and in OpenAI's thought leadership: harnesses are the moat. His guest Jonathan, who builds the Clarity Harness product shown on screen, pushes the claim one level deeper. A harness is a container, and the defensible part is the annotated expert judgment inside it: roughly 150 annotated traces per agent, split into training and test sets, used to train an LLM judge that can certify the agent works without a human watching. The clip is a segment from a longer conversation about what they call the self-driving organization.

The moat claim, restated

The caption frames it with a factory analogy. Saying the harness is the moat is about as true as saying the factory is the moat. Anyone can copy a container architecture off GitHub. What competitors cannot copy is the accumulated record of expert decisions sitting inside it: which outputs were approved, which were rejected, and against which criteria. Jonathan calls this the data science side, a philosophy of subjective expertise captured through annotation.

Stacked harnesses incubate one agent at a time

Jonathan describes their approach as a stacked harness architecture. Each agent gets isolated in its own harness and incubated: it starts with no training and graduates when an LLM judge can vouch for it. The on-screen product, Clarity Harness, shows what this looks like for a "PM Story Writer" agent generating recruiting stories. The left panel lists acceptance criteria ("no candidate is ever auto-rejected," "if the agent's confidence for a slot is below threshold, it asks the recruiter a clarifying question instead of guessing"). The center panel shows the run flow as discrete steps: tool call to ingest the source packet, model call to extract story ticket learning, a decision node for new story versus update, then a final output that awaits human verdicts as decidable units.

The LLM judge is the threshold that matters

The key question Jonathan centers on: have we trained the agent and prompt well enough that, without human oversight, we can assert it is acting well. The judge decides whether the agent's stories, standups, and other artifacts can be trusted unreviewed. Frames from later in the clip show the judge in action, with verdict checklists scored against persona requirements and a review queue where humans approve, annotate, or reject individual requirements. One panel shows a rejected item struck through in red with the correction captured as training signal for the next run.

150 traces, then split like any other dataset

The incubation process is standard supervised evaluation applied to agent behavior. Collect about 150 sample traces, annotate them with expert judgment, then split them into datasets: 15 to 20 percent for training, another slice held out to test the LLM judge itself. Human corrections keep feeding back in. The Runs view in the product shows this loop operating across many agents at once (Recruiter Persona, Hiring Manager Persona, Design Agent, Privacy Agent), each with pending reviews and versioned requirement documents with full lineage.

The end state is a closed loop

The payoff Jonathan describes is the self-driving organization: closed loops where an LLM judge continuously reports that output is good and the system is working, and humans only intervene on flagged items. The threshold for each agent shifts from "does it work" to "can we stop watching it," and the judge trained on your own annotated traces is what earns that trust.

Key Takeaways

• A harness is a container. The defensible asset is the annotated expert judgment accumulated inside it.
• Isolate each agent in its own harness and incubate it from untrained to judge-verified before trusting its output.
• The LLM judge marks the real production threshold: asserting the agent acts well without human oversight.
• Their recipe: annotate about 150 traces, use 15 to 20 percent for training, and hold out a separate set to test the judge itself.
• Human corrections (approve, annotate, reject) are captured as training signal for the next run, closing the loop.

Resources

• Your AI Has Amnesia, That's Why 95% of Deployments Fail -- the full YouTube conversation this TikTok clip is taken from.