<- all tokdocs

AI Agent Vanity Metrics: Hamel Husain on Why Evals Matter More

By , founder of Mavric and Apso

View on TikTok ->

Hamel Husain, an AI evaluation consultant and former Airbnb and GitHub engineer, argues that counts like "200 agents overnight" or "50,000 lines of code" are vanity metrics. In a short TikTok from @hamel_husain he says real product work means asking whether the output works for the customer, which is where evals matter. The position matches what he has published since 2024, and research backs part of it. The viral tweet shown in the clip is an animated prop, though, and the measured evidence suggests output volume is real even when it does not prove value.

What are vanity metrics in AI agent work?

Husain's line: "Those are vanity metrics, right?" He describes people on social media saying "hey, I have a thing that runs overnight" that will "produce XYZ ABC," and says "It looks great. Look at the demo. It's really exciting." Then: "But when you start to get into real product work," you have to ask different questions.

He asks: "Is it working well for the customer? Is it doing what we want? Is it meeting the goal of the customer? Are people happy?" The clip turns those into a numbered on-screen list with shorter wording. The numbering was added in editing. He ends the list with "So on and so forth," so it is an open set of examples rather than a fixed framework.

Is the "200 agents overnight" tweet real?

No. The clip shows a post from "Sam Rivera / @samships" reading "Woke up to 200 agents that ran all night and wrote 50,000 lines of code." Its like counter climbs from 0 to 883 to 8.1K to 8.4K during the first ten seconds while the text stays the same, timed to Husain's words. A real screenshot would show one set of numbers. It is a motion graphic, and "200 agents," "50,000 lines" and the like counts are props. Husain himself uses placeholders ("XYZ ABC"), not those numbers, and the clip starts partway through an answer to a question we do not hear.

What does Hamel Husain say about AI evals?

The clip restates a position he has published for over two years.

  • "Your AI Product Needs Evals" (March 29, 2024). "To know whether your AI product is working, you need a way to measure success beyond vibe checks (which are useful, but not enough)." He ties failed AI products to "a failure to create robust evaluation systems," and says teams focused only on prompts are "prevented from improving their LLM products beyond a demo."
  • "A Field Guide to Rapidly Improving AI Products" (March 24, 2025). "Generic metrics are worse than useless, they actively impede progress." He calls error analysis "the single most valuable activity in AI development and consistently the highest-ROI activity," and describes a case at NurtureBoss where three failure modes caused more than 60 percent of problems.
  • "AI Evals: Everything You Need to Know" (September 18, 2026). "If your dashboard reports a composite score that jumps from 3.2 to 3.7 week over week, it's easy to feel good about the increase without knowing what improved for users."
  • "It's Hard to Eval Is a Product Smell" (June 29, 2026). "artifacts that are hard for you to verify are often hard for users too."

On the Chain of Thought podcast in August 2025 he said generic metrics give "this illusion of, I checked the box, but in reality, like, you're not monitoring anything."

His site describes over 20 years as a machine learning engineer, including Airbnb and GitHub, and early LLM work on code understanding used by OpenAI. He co-teaches the Maven course "AI Evals For Engineers & PMs" with Shreya Shankar, where the course page says 25 years.

Do AI coding tools actually make developers faster?

The evidence is mixed, and it backs the point that how fast work feels is a poor guide. A randomized trial by METR of early-2025 AI tools, by Joel Becker, Nate Rush, Elizabeth Barnes and David Rein, gave 16 experienced developers 246 real tasks in repositories they knew. With AI tools, tasks took 19 percent longer. The developers had predicted a 24 percent speedup and still believed afterward they had been 20 percent faster.

The 2025 DORA report from Google Cloud, based on nearly 5,000 technology professionals, found 90 percent using AI at work. It reports "a positive relationship between AI adoption on both software delivery throughput and product performance," and that "AI adoption does continue to have a negative relationship with software delivery stability." Its explanation: "Without robust control systems, like strong automated testing, mature version control practices, and fast feedback loops, an increase in change volume leads to instability."

Is agent output volume meaningless?

Not quite. The two studies support Husain's core point: output volume and how productive people feel do not tell you whether the work helped anyone. But DORA found throughput really rising. A narrower version of the argument holds up better: shipping more is real and measurable, and whether it served users needs its own measurement. That is where evals come in.

Bottom line

Counting agents or lines of code tells you how much ran, not whether it worked for anyone. Husain's advice, consistent across his writing, is to measure outcomes for the customer, start with error analysis, and distrust generic dashboard scores. Treat the viral tweet in the clip as an illustration, not data.

Sources

Published October 21, 2026. Based on a TikTok by @hamel_husain. Full video notes (markdown)