---
title: "How Complex Should an LLM Judge Be? Hamel Husain's One-Failure Rule"
date: "2026-10-30"
video_id: "7702576270743325966"
tiktok_url: "https://www.tiktok.com/@hamel_husain/video/7702576270743325966"
creator: "@hamel_husain"
category: "AI"
tags: ["llm as a judge", "ai evals", "llm evaluation", "hamel husain", "llm"]
excerpt: "Hamel Husain says each LLM judge should catch one specific failure. His published guidance backs that up and adds binary verdicts and human-label checks."
---
An LLM judge for evals should check one specific failure, not many at once. That is the advice in a short TikTok from @hamel_husain, and it matches his own published guidance, which says "each judge checks exactly one thing." The clip itself gives no example and no measurement, and it leaves out two steps his written materials treat as essential: binary pass/fail verdicts and checking the judge against human labels.

## What is an LLM judge?

An LLM judge is a model prompt that grades another model's output, usually as part of an evaluation suite. The video opens on the question "How complex should my LLM judge be for Evals?"

## How complex should an LLM judge be?

The speaker's answer is that complexity is "not about like the dimensions of input you're putting in." Instead, the question is "what is the failure that you're trying to catch", and you should "scope the failure to be very, very specific."

The problem he describes is a judge that is "scoped too large" and is "trying to catch way too many things." When that judge fails an output, it reports "some vague error", and you do not know what to fix. His goal is "to make that judge immediately actionable."

The phrase "dimensions of input" is his framing. I found no source that defines it as a complexity measure.

## What does Hamel Husain's written guidance say?

The [write-judge-prompt skill](https://github.com/hamelsmu/evals-skills/blob/main/skills/write-judge-prompt/SKILL.md) in the hamelsmu/evals-skills repository says the same thing in writing:

- Design a binary Pass/Fail judge "for one specific failure mode".
- "Each judge checks exactly one thing."
- A judge that covers several dimensions at once is an anti-pattern because it "produces unactionable verdicts."
- Verdicts must be binary, with "No Likert scales". When severity matters, use several binary judges.

His [October 29, 2024 blog post on LLM judges](https://hamel.dev/blog/posts/llm-judge/) asks a domain expert for "Just a clear pass or fail decision" and says multi-point scales are not actionable. That post covers binary judgments but does not state the one-failure rule directly.

## Pass/fail vs. rating scales

The clip mentions a "pass fail score" in passing but does not explain why. The written guidance does. A 1-to-5 score on a broad rubric hides which part of the output went wrong. A pass/fail verdict on one narrow failure tells you exactly what broke. If you need to distinguish minor from severe problems, write a separate binary judge for each level rather than one scale.

## How do you know a narrow judge is right?

The clip skips validation entirely. The 2024 post recommends labeling about 100 examples per failure mode and checking the judge against those human labels. Without that step, you know your judge is narrow, but you do not know whether it agrees with a person reviewing the same outputs.

## How to apply the one-failure rule

Based on the speaker's written guidance, a workable sequence is:

1. Pick one failure mode you have seen in real outputs.
2. Write a judge prompt that checks only that failure and returns pass or fail.
3. Label about 100 examples of that failure mode by hand.
4. Compare the judge's verdicts to your labels before you trust it.
5. Repeat for the next failure mode with a separate judge.

## What the clip does not show

The clip names no concrete failure mode, shows no prompt and reports no result. No source cited here measures narrow judges against broad ones on agreement with human labels or time to find a bug, so the benefit is a design rationale rather than a measured result.

## Bottom line

Scope each LLM judge to one failure and have it return pass or fail. That advice is consistent between the clip and Hamel Husain's published skill and blog post. Before relying on any judge, validate it against about 100 hand-labeled examples, which the clip does not mention.

## Sources

- [write-judge-prompt SKILL.md, hamelsmu/evals-skills on GitHub](https://github.com/hamelsmu/evals-skills/blob/main/skills/write-judge-prompt/SKILL.md): the one-failure rule, binary verdicts and the multi-dimension anti-pattern.
- [LLM judge blog post, hamel.dev, 2024-10-29](https://hamel.dev/blog/posts/llm-judge/): pass/fail advice, the warning on multi-point scales and about 100 examples per failure mode.
- [write judge prompt, Tessl registry mirror](https://tessl.io/registry/skills/github/hamelsmu/evals-skills/write-judge-prompt): a secondary mirror of the same skill.
- [TikTok post by @hamel_husain](https://www.tiktok.com/@hamel_husain/video/7702576270743325966): the clip reviewed here.


## Video description and verification notes

**The file is a 35.5 s vertical clip (ffprobe: 1080x1920, MP4/ISOM container, HEVC Main video at 30 fps and about 606 kbps, HE-AACv2 stereo audio at 44.1 kHz and about 48 kbps, 2,932,151 bytes, about 661 kbps overall). metadata.json lists format bytevc1_1080p_660710-1 and leaves fps null, so the 30 fps comes from ffprobe. It was posted at epoch 1793395808, which is 2026-10-30 21:30:08 UTC (upload_date 20261030), by handle @hamel_husain (the channel name is also hamel_husain). The audio credit is "original sound - hamel_husain". The description is 272 characters long and carries the hashtags #aievals #llm #llmjudge. Counts at archive capture (2026-10-31T12:03:27 UTC) were 351 views, 3 likes, 0 comments, 1 repost and 3 saves. I read only one frame, frame_001 (0 to 2 s), so everything I say about the visuals is limited to that frame. It shows a white upper panel with the monospaced black text "Q: How complex should my LLM judge be for Evals?". Below that is a burned-in caption chip reading "of LLM as a". Below the chip, a bald man with glasses, one AirPod and a dark striped sweater sits in an office chair with his palms pressed together. I did not read the other frames, so I cannot say whether a terminal, slide, chart or URL appears later, and I make no claim about them. The transcript has 7 Whisper segments and about 120 words, with segments running from 0.0 s to 35.04 s. The audio lasts 35.50 s, so the last segment ends inside the audio and nothing is fabricated. Whisper wrote "LMS judge" in the first line where the description and frame use "LLM judge", so I treat that as a transcription error. The spoken argument is that a judge should target one very specific failure, because a broad judge that catches many error types returns a pass/fail result that does not say what to fix.**

## The core claim matches the speaker's own written guidance

At 0.0 s the video says the judge question is "what is the failure that you're trying to catch", and at about 4 s it says to "scope the failure to be very, very specific." The speaker's published skill for writing judge prompts says the same thing in nearly the same terms. The skill in the hamelsmu/evals-skills repository tells the author to design a binary Pass/Fail judge "for one specific failure mode" and adds that "each judge checks exactly one thing." The same page lists a judge that covers several dimensions at once as an anti-pattern because it "produces unactionable verdicts." That matches the clip's complaint at 13.72 s that the judge is "scoped too large" and is "trying to catch way too many things." The clip's argument therefore agrees with the speaker's own written guidance, and the repository is the source I could read. I could not read a first-party Hamel Husain page that states the principle as a general rule. His October 2024 blog post covers binary judgments but does not state it directly.

## The "actionable" argument is supported, but the clip shows no demonstration

At 25.22 s the video says "your goal is to make that judge immediately actionable," and at about 30 s it contrasts that with a judge that reports "some vague error." The repository skill ties actionability to the same design choice: a single-failure judge produces a verdict that points at one thing to fix. The clip gives no concrete failure mode, no example prompt and no before/after comparison in the audio. The one frame I read shows only the question card, so I cannot say whether later frames carry an example. The claim is a design rationale, and the clip offers no measurement for it. No source I read gives a measured result of narrow versus broad judges, such as agreement with human labels or time to find a bug.

## "Not about the dimensions of input": a framing the clip leaves unexplained

The opening line, at 0.0 s, says judge complexity is "not about like the dimensions of input you're putting in." It reads as a reply to the on-screen question "How complex should my LLM judge be for Evals?". I found no source that defines or tests "dimensions of input" as a complexity measure, so I treat that phrase as the speaker's framing, not a sourced claim. The clip's answer to the question is that complexity is managed by narrowing the failure the judge targets, and that is consistent with the repository guidance above.

## What the clip leaves out that the speaker's own materials include

The clip says nothing about binary outputs versus rating scales, although "pass fail score" appears at about 25 s. The repository guidance says verdicts must be binary, with "No Likert scales", and suggests several binary judges when severity matters. Hamel Husain's October 29, 2024 post asks a domain expert for "Just a clear pass or fail decision" and says it is not actionable to use multi-point scales. It also recommends labeling about 100 examples per failure mode when validating a judge. The clip omits all of this, including any check of the judge against human labels. A viewer who acts only on this clip would know to narrow the scope, but not how to confirm that the narrow judge agrees with human reviewers. The search summaries I saw also said the skill suggests code-based checks, such as regex or schema validation, before using a judge for anything checkable. That detail came from third-party mirrors, so I do not rely on it.

## Key Takeaways

- **Verified:** The advice to scope a judge to one specific failure matches the speaker's own skill text, which says each judge "checks exactly one thing" and calls multi-dimension judges unactionable.
- **Verified:** The transcript ends at 35.04 s inside a 35.50 s audio stream, so there is no hallucinated tail. Whisper wrote "LMS judge" for "LLM judge".
- **Clarification:** The clip never names a concrete failure mode, shows a prompt or reports a result. I read one frame of about 18, so I can only vouch for the question card at the start.
- **Unverified:** No source I read measures narrow judges against broad ones, and I found none that defines "dimensions of input" as a complexity measure.
- **Context:** The speaker's fuller guidance adds binary verdicts, human-label validation and about 100 labeled examples per failure mode, none of which the clip mentions.

## Resources

- [write-judge-prompt SKILL.md, hamelsmu/evals-skills on GitHub (undated)](https://github.com/hamelsmu/evals-skills/blob/main/skills/write-judge-prompt/SKILL.md) establishes the "one specific failure mode" rule, the binary-verdict requirement and the multi-dimension anti-pattern I cite.
- [LLM judge blog post, hamel.dev, 2024-10-29](https://hamel.dev/blog/posts/llm-judge/) establishes the pass/fail verdict advice, the warning about multi-point scales and the 100-examples-per-failure-mode guidance I cite.
- [write judge prompt, Tessl registry mirror (undated)](https://tessl.io/registry/skills/github/hamelsmu/evals-skills/write-judge-prompt) is a secondary mirror seen in search results. I did not open it and use it only as a pointer.
