<- all tokdocs
2026.10.09@hamel_husain

Hamel Husain's pass/fail advice matches his own published FAQ, but his on-screen reason one does not: the FAQ blames human annotators, and two retrieval papers find fine-grained scales beat binary

AI
Watch on TikTok

View on TikTok ->

A 42.776349-second vertical clip, 720x1280 at an aspect ratio of 0.56, delivered in a QuickTime/MOV container that ffprobe reports as mov,mp4,m4a,3gp,3g2,mj2, carrying an HEVC Main-profile video stream tagged hvc1 in yuv420p at a constant 30/1 fps for 1,283 frames and 228,695 bps, plus an HE-AAC audio stream tagged mp4a at 44,100 Hz stereo for 923 frames and 96,196 bps, with an overall container bitrate of 331,742 bps and a filesize of 1,773,840 bytes, matching the filesize field in metadata.json exactly and the format_id string bytevc1_720p_331737-1; the extracted audio.mp3 runs 42.788571 seconds at 614,105 bytes and 114,816 bps. The post timestamp is epoch 1791581823, which is 2026-10-09 21:37:03 UTC, and the upload_date field agrees at 20261009. The uploader handle and channel name are both hamel_husain with uploader_id 6921589110637724678; metadata.json carries no separate display nickname field. Audio is credited as original sound - hamel_husain with artist set to the same handle, so there is no licensed music track. The description runs 177 characters: "LLMs aren't good at continuous scores, and nobody knows what an average of 3.2 versus 3.7 actually means. A 1 to 5 scale lets you avoid making a call, a pass or fail forces one." At capture time, recorded as the metadata.json fetch epoch 1792388061, which is 2026-10-19 05:34:21 UTC and roughly nine days and eight hours after posting, the clip showed 2,841 views, 65 likes, 2 comments, 7 reposts and 39 saves. I read 10 of the 21 extracted frames: 001, 004, 007, 010, 011, 013, 016, 018, 019 and 021, which at the 2-second extraction interval correspond to roughly 0, 6, 12, 18, 20, 24, 30, 34, 36 and 40 seconds. The transcript runs 122 words across 11 segments. Frame 001 shows a pill-shaped lower-third card reading "LLM Judge" over a black upper half, with a white caption block beneath it reading "Stop scoring your LLM judge 1 to 5"; the lower half of every frame is Husain speaking to camera in a blue pin-dot button-down over a black tee, glasses, bald, standing in a bright room with a sliding glass door and garden foliage to his left. Frame 004 replaces the title card with a white heading "Likert scale" above five rounded tiles reading 1, 2, 3, 4 and 5, with the burned-in caption "export like a Likert". Frame 007 keeps the same tiles and strikes a red horizontal line through all five, caption "score because that introduces". Frame 010 shows a numbered list item, a rounded "1" tile beside the text "LLMs are bad at continuous scores", caption "number one." Frame 011 adds item two, "The output isn't clear", caption "output is not gonna". Frame 013 shows a grey "Average score" label above a large white "3.2" and a smaller "vs", caption "average score of 3.2". Frame 016 completes it to "3.2 vs 3.7" with a red question mark below, caption "honestly." Frame 018 shows the heading "Hiding behind a score" alone, caption "behind a score". Frame 019 adds a pink-outlined pill below it reading "= No decision", caption "a decision." Frame 021 shows "Good enough to ship?" above a green-outlined "YES" button and a dimmed red-outlined "NO" button, caption "Yes or no?" The spoken argument is that an LLM judge should emit a binary score rather than a Likert or one-to-five scale, for two stated reasons, that LLMs handle continuous scores badly and that an averaged score is uninterpretable and unactionable, closing with the reframe that the only question worth asking is whether the feature is good enough to ship. On the transcript-end check, the final segment "Make a decision." is timestamped 41.540 to 42.400 seconds against a real audio duration of 42.788571 seconds and a video stream duration of 42.766667 seconds, so the transcript ends inside the audio with roughly 0.39 seconds to spare and shows no sign of Whisper hallucinating past the end of the file; the only transcription defect is at 4.440 seconds, where Whisper wrote "you don't want to export like a Likert scale" and the burned-in on-screen caption in frame 004 agrees with the error, though the surrounding argument and the opening line "that you output a binary score" make clear the spoken word is "output".

The binary recommendation is Husain's own published position, documented since October 2024

At 0.00 seconds the video says "It's very important for an LLM judge that you output a binary score." That is not a new or compressed claim. It restates the central instruction of his long-form guide, which hamel.dev publishes at /blog/posts/llm-judge/ under the title "Using LLM-as-a-Judge For Evaluation: A Complete Guide", authored by Hamel Husain and dated October 29, 2024. The guide's table of contents names Step 3 as "Direct The Domain Expert to Make Pass/Fail Judgments with Critiques", with a subsection titled "Don't stray from binary pass/fail judgments when starting out".

The body of that subsection states the position bluntly: "If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you're doing it wrong." It then gives two reasons. The first is that scales are "not actionable", and the text reads "People don't know what to do with a 3 or 4. It's not immediately obvious how this number is better than a 2. You need to be able to say 'this interaction passed because…' and 'this interaction failed because…'." The second reason is correlational: "More often than not, these metrics do not matter. Every time I've analyzed data on domain expert judgments, they tend not to correlate with these kind of metrics."

The guide also supplies the positive case the video compresses into its closing frame. Under "Why are simple pass/fail metrics important?", the post lists "Clarity and Focus: A binary decision forces everyone to consider what truly matters. It simplifies the evaluation to a single, crucial question." The video's final card, "Good enough to ship? YES / NO" in frame 021 at roughly 40 seconds, is that single crucial question rendered as a UI mock. The spoken line at 37.62 seconds, "The frame here is, is this feature good enough to ship?", tracks the published text closely enough that the video should be read as a restatement rather than an extension.

One element of the written guide does not appear in the video at all. The published method pairs every binary verdict with a written critique. The guide devotes a section to it, "The Role of Critiques", and says the critique "allows you to note if something was mostly correct but had areas for improvement" and exists to "Balance Simplicity with Depth: While the pass/fail offers a clear verdict, the critique offers the depth needed to understand the reasoning behind the judgment." No frame I read mentions critiques, and the transcript never uses the word. A viewer who implements only what the clip shows gets the verdict without the mechanism the guide uses to recover the nuance the verdict throws away.

"LLMs are not good at continuous scores" is the video's reason one, and it is not the reason his FAQ gives

At 15.76 seconds the video says "LLMs are not good at continuous scores, number one," and frame 010 at roughly 18 seconds renders it as list item 1, "LLMs are bad at continuous scores". This is the one load-bearing claim in the clip that his own published argument does not make.

The dedicated FAQ entry is at hamel.dev under the slug why-do-you-recommend-binary-passfail-evaluations-instead-of-1-5-ratings-likert-scales.html, authored by Hamel Husain and Shreya Shankar and published May 29, 2025, as part of their "AI Evals FAQ". Its one-line summary is "Binary labels produce clearer judgments than rating scales, while separate checks can track gradual progress." The full argument, quoted verbatim from the page, is this:

"Engineers often believe that Likert scales (1-5 ratings) provide more information than binary evaluations, allowing them to track gradual improvements. However, this added complexity often creates more problems than it solves in practice. Binary evaluations force clearer thinking and more consistent labeling. Likert scales introduce significant challenges: the difference between adjacent points (like 3 vs 4) is subjective and inconsistent across annotators, detecting statistical differences requires larger sample sizes, and annotators often default to middle values to avoid making hard decisions. Having binary options forces people to make a decision rather than hiding uncertainty in middle values. Binary decisions are also faster to make during error analysis - you don't waste time debating whether something is a 3 or 4."

Every failure mode named there belongs to a person, not a model. "Inconsistent across annotators", "annotators often default to middle values", "forces people to make a decision", "faster to make during error analysis". The sample-size point is a statistical-power argument about detecting differences, not a claim about model capability. The published reason for preferring binary is that human labelers and the teams reading their output behave badly on a five-point scale. The video's reason one asserts instead that the language model is the thing that handles continuous scores badly. That is a different claim with a different evidence base, and Husain's own FAQ does not advance it.

The FAQ does close with a hedge the video drops: "Start with binary labels to understand what 'bad' looks like. Numeric labels are advanced and usually not necessary." That is a sequencing recommendation with an explicit escape hatch. "Usually not necessary" is a weaker and more careful statement than the spoken "99% of the time" at 4.44 seconds, and I could not source the 99% figure to any Husain post. Treat it as rhetorical emphasis rather than a measured rate.

The "3.2 versus 3.7" complaint is lifted from the 2024 guide, where it is an actionability argument

At 22.86 seconds the video says "When you see an average score of 3.2 versus 3.7, no one really knows what the hell that means, and it's not really actionable, honestly." Frames 013 and 016, at roughly 24 and 30 seconds, build the comparison on screen as "Average score / 3.2 vs 3.7" with a red question mark underneath.

This is the strongest-sourced claim in the clip. The 2024 guide's opening diagnosis names "Arbitrary Scoring Systems: Using uncalibrated scales (like 1-5) across multiple dimensions, where the difference between scores is unclear and subjective. What makes something a 3 versus a 4? Nobody knows, and different evaluators often interpret these scales differently." The spoken line swaps the specific pair of numbers and keeps the structure intact. The verdict is verified, and it is worth stating what the claim actually is: it is an argument about the interpretability of an aggregate, not an argument that the aggregate is numerically wrong.

That distinction matters for the next two sections. A mean of 3.2 against a mean of 3.7 can be a real and reproducible difference in the underlying data while still being useless to a product team that cannot act on it. Husain's complaint is about usefulness. The video's reason one asserts the stronger claim about reality.

Two retrieval papers find the opposite of "99% of the time": fine-grained scales beat binary

At 4.44 seconds the video says "99% of the time, you don't want to [output] like a Likert scale or a score of one to five or some kind of score because that introduces tremendous complexity." There is a published body of work in information retrieval that found the reverse on its own task.

"Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels", arXiv:2310.14122, submitted October 21, 2023 by Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang and Michael Bendersky, states in its abstract: "Existing prompts for pointwise LLM rankers mostly ask the model to choose from binary relevance labels like 'Yes' and 'No'. However, the lack of intermediate relevance label options may cause the LLM to provide noisy or biased answers for documents that are partially relevant to the query." The measured result across eight BEIR datasets is that "adding fine-grained relevance labels significantly improves the performance of LLM rankers."

"Likert or Not: LLM Absolute Relevance Judgments on Fine-Grained Ordinal Scales", arXiv:2505.19334, submitted May 25, 2025 by Charles Godfrey, Ping Nie, Natalia Ostapuk, David Ken, Shang Gao and Souheil Inati, confirms that result in its related-work section: "Zhuang et al. (2024a) demonstrated that pointwise scoring using fine-grained relevance scales with more than two labels significantly outperforms binary scoring. They considered scales of 2, 3 and 4 labels." Its own abstract pushes against the prevailing assumption that LLMs judge better relatively than absolutely: "we find that the gap between pointwise scoring and listwise ranking shrinks when pointwise scoring is implemented using a sufficiently large ordinal relevance label space, becoming statistically insignificant for many LLM-benchmark dataset combinations." The evaluation spans four LLMs, eight BEIR and TREC-DL datasets and two proprietary datasets whose relevance labels were collected after every evaluated model's training cutoff.

These papers do not refute Husain. They scope him. Both measure document relevance ranking, where the output is an ordering and a graded score carries ordering information a binary label cannot. Husain is talking about a product ship decision, where the output is one bit and the extra levels buy nothing an engineer can act on. The correction to the video is narrower than "he is wrong": the claim "LLMs are not good at continuous scores" is stated without qualification at 15.76 seconds and on screen in frame 010, and on at least one well-measured task, graded scales demonstrably outperform the binary alternative the video recommends.

A 2026 paper measures the resolution problem and also measures binary's own failure mode

"The Format is the Filter: Scoring Format Fixes What an LLM Judge Can Express", arXiv:2610.17158, submitted October 12, 2026 by Lovedeep Gondara, Jekaterina Novikova, Sagarika Suresh, Jack Ji and Jithin Pradeep, is the closest thing I found to a direct measurement of the video's reason one. The authors scored 15,406 UltraFeedback candidates under five scoring formats with three judge models, curated the data, fine-tuned 66 models and evaluated them with four judges, holding the candidate pool, judge prompt, decoding and selection budget fixed.

Two abstract findings matter here. The first supports the spirit of the video's reason one: "A 0--100 scale is worth only 7--12 equally used levels, two to three times a 1--5 scale rather than twenty." An LLM asked for a hundred-point score produces roughly a ten-point score wearing a hundred-point costume. That resolution ceiling is the defensible version of "LLMs are not good at continuous scores".

The second finding cuts against the video. "A binary verdict leaves the top score tied in 77--78% of prompts, so most selections are made at random." A five-point scale hides a decision in the middle values. A binary scale hides it in a mass tie at the top. The paper notes that the usual summary statistic obscures this: "Kendall's tau_b hides them: it ranks binary best while tau_a and top-1 agreement rank it last, because it discounts the ties a selector must break at random."

Downstream, the paper reports that "curated data beat random selection under every evaluation judge" but that "no ordering among formats is detectable". On the task it measures, filtering instruction-tuning data rather than deciding whether to ship a feature, the choice between binary and a five-point scale did not change final model quality detectably. That is a real limit on how far the video's framing generalizes, and a 42-second clip that cites nothing has no room to carry it.

The clip shows no sources, no data and no judge prompt

Across the 10 frames I read, spanning 0 to roughly 40 seconds, there is no terminal, no code editor, no browser, no repository name, no URL, no paper citation, no dashboard screenshot and no chart with real data. Every graphic is a motion-design card built for the clip. The 3.2 and 3.7 in frames 013 and 016 are illustrative, not measured, and nothing on screen claims otherwise.

The claim at 32.22 seconds, "I've found that when you try to hide behind a score, you're not really making a decision," is framed as personal experience, and the 2024 guide backs it the same way: "Every time I've analyzed data on domain expert judgments, they tend not to correlate with these kind of metrics." That is an assertion about unpublished client work, consistent across two years of his writing and not independently checkable. The credentials behind it are checkable. hamel.dev describes Husain as a machine learning engineer with more than twenty years of experience who previously worked at Airbnb and GitHub, and the AI Evals FAQ he co-authors with Shreya Shankar, published September 18, 2026 and last modified September 21, 2026, states that their course has taught more than 5,000 engineers and PMs.

Key Takeaways

  • Verified: The binary-score recommendation at 0.00 s is Husain's own long-standing published position. His October 29, 2024 guide says "If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you're doing it wrong."
  • Verified: The "3.2 versus 3.7" complaint at 22.86 s restates a line from the same 2024 guide: "What makes something a 3 versus a 4? Nobody knows, and different evaluators often interpret these scales differently."
  • Verified: The name is spelled Hamel Husain, and his FAQ co-author is Shreya Shankar. Both spellings are taken from the bylines on hamel.dev, not from the video.
  • Correction: The video's reason one, "LLMs are not good at continuous scores" at 15.76 s and on screen in frame 010, is not the reason his published FAQ gives. The May 29, 2025 FAQ entry attributes every failure to human annotators: adjacent points are "subjective and inconsistent across annotators", "annotators often default to middle values", and the method "forces people to make a decision". None of it is a claim about model capability.
  • Partial correction: "99% of the time, you don't want to [output] a Likert scale" at 4.44 s overstates his own written hedge. The FAQ says "Numeric labels are advanced and usually not necessary." I could not source the 99% figure to any published Husain post.
  • Correction: Stated without qualification, "LLMs are not good at continuous scores" is contradicted on at least one well-measured task. arXiv:2310.14122 found that "adding fine-grained relevance labels significantly improves the performance of LLM rankers" over binary Yes/No prompting on eight BEIR datasets, and arXiv:2505.19334 confirmed and extended that result across four LLMs and ten datasets.
  • Clarification: Binary has its own measured failure mode. arXiv:2610.17158 found that "a binary verdict leaves the top score tied in 77--78% of prompts, so most selections are made at random," and that on its task "no ordering among formats is detectable" downstream.
  • Context: The same 2026 paper supplies the defensible version of reason one: "A 0--100 scale is worth only 7--12 equally used levels, two to three times a 1--5 scale rather than twenty." The resolution ceiling is real. The blanket statement is not.
  • Clarification: The video omits the written critique that the 2024 guide pairs with every binary verdict. The guide says the critique exists to "Balance Simplicity with Depth". No frame I read and no transcript segment mentions it.
  • Context: Scope separates the video from the papers. Husain describes a ship-or-not product decision where the output is one bit. The retrieval papers measure document ranking, where a graded label carries ordering information a binary label cannot.
  • Context: At capture time on 2026-10-19 05:34:21 UTC, roughly nine days after the 2026-10-09 21:37:03 UTC post, the clip had 2,841 views, 65 likes, 2 comments, 7 reposts and 39 saves. The save-to-like ratio of 39 to 65 is unusually high for a 42-second talking-head clip and suggests reference-saving rather than casual viewing.
  • Clarification: Whisper transcribed "output" as "export" at 4.44 s, and the burned-in on-screen caption in frame 004 carries the same error. The transcript's last segment ends at 42.400 s against 42.788571 s of real audio, so there is no hallucinated tail.

Resources

Published October 9, 2026. Writeup generated from a favorited TikTok.