How Complex Should an LLM Judge Be? Hamel Husain's One-Failure Rule
By Matt Cullerton, founder of Mavric and Apso
An LLM judge for evals should check one specific failure, not many at once. That is the advice in a short TikTok from @hamel_husain, and it matches his own published guidance, which says "each judge checks exactly one thing." The clip itself gives no example and no measurement, and it leaves out two steps his written materials treat as essential: binary pass/fail verdicts and checking the judge against human labels.
What is an LLM judge?
An LLM judge is a model prompt that grades another model's output, usually as part of an evaluation suite. The video opens on the question "How complex should my LLM judge be for Evals?"
How complex should an LLM judge be?
The speaker's answer is that complexity is "not about like the dimensions of input you're putting in." Instead, the question is "what is the failure that you're trying to catch", and you should "scope the failure to be very, very specific."
The problem he describes is a judge that is "scoped too large" and is "trying to catch way too many things." When that judge fails an output, it reports "some vague error", and you do not know what to fix. His goal is "to make that judge immediately actionable."
The phrase "dimensions of input" is his framing. I found no source that defines it as a complexity measure.
What does Hamel Husain's written guidance say?
The write-judge-prompt skill in the hamelsmu/evals-skills repository says the same thing in writing:
- Design a binary Pass/Fail judge "for one specific failure mode".
- "Each judge checks exactly one thing."
- A judge that covers several dimensions at once is an anti-pattern because it "produces unactionable verdicts."
- Verdicts must be binary, with "No Likert scales". When severity matters, use several binary judges.
His October 29, 2024 blog post on LLM judges asks a domain expert for "Just a clear pass or fail decision" and says multi-point scales are not actionable. That post covers binary judgments but does not state the one-failure rule directly.
Pass/fail vs. rating scales
The clip mentions a "pass fail score" in passing but does not explain why. The written guidance does. A 1-to-5 score on a broad rubric hides which part of the output went wrong. A pass/fail verdict on one narrow failure tells you exactly what broke. If you need to distinguish minor from severe problems, write a separate binary judge for each level rather than one scale.
How do you know a narrow judge is right?
The clip skips validation entirely. The 2024 post recommends labeling about 100 examples per failure mode and checking the judge against those human labels. Without that step, you know your judge is narrow, but you do not know whether it agrees with a person reviewing the same outputs.
How to apply the one-failure rule
Based on the speaker's written guidance, a workable sequence is:
- Pick one failure mode you have seen in real outputs.
- Write a judge prompt that checks only that failure and returns pass or fail.
- Label about 100 examples of that failure mode by hand.
- Compare the judge's verdicts to your labels before you trust it.
- Repeat for the next failure mode with a separate judge.
What the clip does not show
The clip names no concrete failure mode, shows no prompt and reports no result. No source cited here measures narrow judges against broad ones on agreement with human labels or time to find a bug, so the benefit is a design rationale rather than a measured result.
Bottom line
Scope each LLM judge to one failure and have it return pass or fail. That advice is consistent between the clip and Hamel Husain's published skill and blog post. Before relying on any judge, validate it against about 100 hand-labeled examples, which the clip does not mention.
Sources
- write-judge-prompt SKILL.md, hamelsmu/evals-skills on GitHub: the one-failure rule, binary verdicts and the multi-dimension anti-pattern.
- LLM judge blog post, hamel.dev, 2024-10-29: pass/fail advice, the warning on multi-point scales and about 100 examples per failure mode.
- write judge prompt, Tessl registry mirror: a secondary mirror of the same skill.
- TikTok post by @hamel_husain: the clip reviewed here.
Published October 30, 2026. Based on a TikTok by @hamel_husain. Full video notes (markdown)