LLM Judge Scoring: Pass/Fail vs 1-5 Scales, Hamel Husain Checked
By Matt Cullerton, founder of Mavric and Apso
Hamel Husain recommends that an LLM judge give a binary pass/fail verdict instead of a 1 to 5 score, and that advice matches what he has published since 2024. But the main reason in his TikTok, that "LLMs are not good at continuous scores," is not the reason his own FAQ gives. His written argument is about human annotators and teams, and research on ranking tasks shows graded scales can beat binary ones.
In the video, posted as @hamel_husain, he gives two reasons: LLMs handle continuous scores badly, and an average like 3.2 versus 3.7 is not actionable.
Why does Hamel Husain recommend pass/fail evaluations?
This is his long-standing position. His guide "Using LLM-as-a-Judge For Evaluation: A Complete Guide," dated October 29, 2024, has a subsection titled "Don't stray from binary pass/fail judgments when starting out." It says: "If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you're doing it wrong."
The guide gives two reasons. Scales are not actionable: "People don't know what to do with a 3 or 4. It's not immediately obvious how this number is better than a 2." And they often do not track what experts care about: "Every time I've analyzed data on domain expert judgments, they tend not to correlate with these kind of metrics."
It also makes the positive case: "A binary decision forces everyone to consider what truly matters. It simplifies the evaluation to a single, crucial question." The video's closing "Good enough to ship? YES / NO" card is that question.
What does the 3.2 vs 3.7 example mean?
The video says "When you see an average score of 3.2 versus 3.7, no one really knows what the hell that means, and it's not really actionable." This restates a line from the 2024 guide: "What makes something a 3 versus a 4? Nobody knows, and different evaluators often interpret these scales differently."
This is an argument about usefulness. A gap between 3.2 and 3.7 can be real in the data and still give a team nothing to act on.
Are LLMs bad at continuous scores?
This is the video's reason one, and Husain's own FAQ does not make it. The FAQ entry, co-written with Shreya Shankar and published May 29, 2025, explains the binary recommendation this way:
"Likert scales introduce significant challenges: the difference between adjacent points (like 3 vs 4) is subjective and inconsistent across annotators, detecting statistical differences requires larger sample sizes, and annotators often default to middle values to avoid making hard decisions. Having binary options forces people to make a decision rather than hiding uncertainty in middle values."
Every failure named there belongs to a person: inconsistent annotators, annotators defaulting to the middle, people avoiding decisions. The sample-size point is about statistical power. None of it says the language model is the problem.
The FAQ also hedges: "Start with binary labels to understand what 'bad' looks like. Numeric labels are advanced and usually not necessary." The video says "99% of the time" instead. I could not find the 99% figure in any Husain post.
When do fine-grained scales beat binary?
Research on document ranking points the other way. "Beyond Yes and No" (arXiv:2310.14122, October 2023) by Honglei Zhuang and colleagues found that binary Yes/No prompts "may cause the LLM to provide noisy or biased answers for documents that are partially relevant to the query." Across eight BEIR datasets, "adding fine-grained relevance labels significantly improves the performance of LLM rankers."
"Likert or Not" (arXiv:2505.19334, May 2025) by Charles Godfrey and colleagues confirmed that result. It also found that the gap between pointwise scoring and listwise ranking "shrinks when pointwise scoring is implemented using a sufficiently large ordinal relevance label space."
These papers narrow Husain's advice without refuting it. They measure ranking, where a graded score carries ordering information that a yes/no label cannot. Husain is talking about a ship decision, where the answer is one bit. The problem is that the video states "LLMs are not good at continuous scores" with no qualification.
Does binary scoring have downsides?
A 2026 paper measures both sides. "The Format is the Filter" (arXiv:2610.17158, October 12, 2026) by Lovedeep Gondara and colleagues scored 15,406 UltraFeedback candidates under five formats with three judge models, then fine-tuned 66 models on the curated data.
One finding supports a narrow version of the video's point. A 0 to 100 scale behaves like only 7 to 12 equally used levels, two to three times a 1 to 5 scale instead of twenty. That resolution ceiling is the defensible form of "LLMs are not good at continuous scores."
Another finding cuts against binary. The paper found that a binary verdict leaves the top score tied in 77 to 78 percent of prompts, "so most selections are made at random." A five-point scale hides indecision in the middle. A binary scale hides it in a tie at the top.
On that paper's task, filtering training data, curated data beat random selection under every judge, but "no ordering among formats is detectable." The choice between binary and five-point scales did not detectably change final model quality.
What the video leaves out: critiques
The 2024 guide pairs every pass/fail verdict with a written critique. Its section "The Role of Critiques" says the critique lets you "note if something was mostly correct but had areas for improvement" and helps "Balance Simplicity with Depth." The video never mentions critiques. Someone who follows only the clip gets the verdict without the part of the method that recovers the detail a binary score throws away.
The clip shows no data or citations, and its 3.2 and 3.7 are illustrations. hamel.dev describes Husain as a machine learning engineer with more than twenty years of experience, previously at Airbnb and GitHub.
Bottom line
For a ship-or-not product decision, pass/fail with a written critique is Husain's published method, and his reasoning about unclear averages holds. His stated reason that LLMs cannot handle continuous scores does not match his own FAQ, which blames human annotators. On ranking tasks, graded scales beat binary, and binary verdicts produce many ties at the top.
Sources
- Using LLM-as-a-Judge For Evaluation: A Complete Guide, hamel.dev - the 2024 guide, binary recommendation and critiques
- Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings?, hamel.dev - the FAQ argument about annotators
- AI Evals FAQ, hamel.dev - the full FAQ by Husain and Shankar
- Hamel Husain, hamel.dev - author background
- Beyond Yes and No, arXiv:2310.14122 - fine-grained labels beat binary for LLM rankers
- Likert or Not, arXiv:2505.19334 - graded scales on ranking benchmarks
- The Format is the Filter, arXiv:2610.17158 - resolution limits and binary ties
Published October 9, 2026. Based on a TikTok by @hamel_husain. Full video notes (markdown)