---
title: "evals-skills: Hamel Husain and Shreya Shankar's AI Eval Skills Repo"
date: "2026-09-14"
video_id: "7685439436099751200"
tiktok_url: "https://www.tiktok.com/@therobertta_/video/7685439436099751200"
creator: "@therobertta_"
category: "AI"
tags: ["ai evals", "evals-skills", "error analysis", "hamel husain", "claude code skills"]
excerpt: "evals-skills is a set of agent skills for building AI evals, by Hamel Husain and Shreya Shankar. Here is what error-discovery does and how to install it."
---
evals-skills is a GitHub repository of skills that guide AI coding agents to help you build evals for your own AI product. A TikTok from @therobertta_ (Robert Ta) recommends its error-discovery skill and describes it accurately. The repo has no license file, so calling it "open source" overstates it, and the video credits Hamel Husain alone when Shreya Shankar co-authored it.

## What is evals-skills?

The repository is `ai-evals-course/evals-skills`, created 2026-06-23. Its README says: "Skills that guide AI coding agents to help you build product-specific AI evals (not foundation model benchmarks). These skills guard against common mistakes we've seen helping 50+ companies and teaching thousands of students in our AI Evals course."

It contains eight skills: `evals-start`, `eval-audit`, `error-discovery`, `generate-synthetic-data`, `write-judge-prompt`, `validate-evaluator`, `evaluate-rag` and `build-review-interface`. The video mentions two.

On 2026-09-15 the GitHub API showed 598 stars, 49 forks and 65 commits. The screen recording in the video showed 449 stars.

## Who made evals-skills?

The video says "Hamel" put it out. The plugin manifest lists the author as "Shreya Shankar and Hamel Husain." Contributor data shows `hamelsmu` with 49 commits, `shreyashankar` with 15 and `strickvl` with 1. Shankar is a PhD candidate in UC Berkeley's Data Systems and Foundations group, the author of DocETL, and co-author with Husain of the O'Reilly book *Evals for AI Engineers*, listed for release on 2026-10-31. The README links to her live walkthrough of the error-discovery skill.

## Is evals-skills open source?

Not formally. The GitHub API returns `"license": null`, and there is no `LICENSE` file on `main`. A public repository can be read by anyone, but without a license, default copyright applies and no one has been granted the right to copy, modify or redistribute it. That matters for a repo meant to be installed and edited. The authors may not intend any restriction, and adding a license file would settle it.

## What does the error-discovery skill do?

Its description reads: "Run error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes." The README is firm about order: "You should only write evals after doing this step."

`SKILL.md` runs four phases:

1. **Understand the domain and data.** Inventory the data, identify its structure and dimensions of variation, and define what "bad" looks like.
2. **Design the visual encoding** using Gestalt principles: color for categories, spacing for hierarchy, opacity for importance.
3. **Build the interface.** The README says it "Builds a single-file HTML review app served by a Python stdlib server. No dependencies."
4. **Cluster the data** and pick a starting sample from cluster representatives plus random picks.

Phase 5 lives in `review-loop.md`. It has a breadth mode, which proposes new samples to widen coverage, and a depth mode, which scans the data for more cases of a failure you already found. The skill is "meant for interactive sessions only" and tells the agent "Do not go silent for long stretches."

## How do you check that an LLM judge is right?

The video asks how you know an AI product is "objectively getting better," but does not mention the skill that answers it. `validate-evaluator` calibrates an LLM judge against human labels using data splits, true positive and true negative rates, and bias correction.

## How to install evals-skills

The README uses the `npx skills` tool from `vercel-labs/skills`:

`npx skills add https://github.com/ai-evals-course/evals-skills`

Add `--skill error-discovery` to install one skill. On 2026-08-31 the `start` skill was renamed `evals-start` and the plugin gained Codex packaging, so the "start" skill shown in the video no longer exists under that name.

## Why the rubric analogy works

Ta's explanation is the best part of the clip. An English teacher grades 40 to 50 subjective essays fairly by publishing a rubric covering grammar, prose, working bibliography links, formatting, argument and concision. For LLM output, a missing bibliography or an essay on the wrong topic is a failure mode, failure modes produce the rubric, and the rubric produces the eval. That is the same order the repo prescribes.

He calls evals "the moat for any domain-specific agent" and labels it "my true belief." The auto-captions render "moat" as "mode," but the source episode's chapter at 1:14:03 is titled "What's the meta: evals are the moat."

## Who benefits from this advice?

The README links twice to the Maven course *AI Evals For Engineers & PMs* by Husain and Shankar, listed at $4,200 for the cohort running 2026-10-10 to 2026-11-21 and rated 4.7 out of 5 across 901 reviews. The repo is a free entry point to a paid course.

Ta has a related product. The source episode ends with a demo of his company's agent review harness, which queues agent runs for human review with approve, edit, reject and defer verdicts. The clip does not mention that overlap.

The "Watch the full video" link goes to an 89-minute episode titled "Opus 5 Is The First Model That Gets Worse With Use," where the evals discussion starts at 1:14:03.

## Bottom line

evals-skills is a real, actively maintained set of eval skills from Hamel Husain and Shreya Shankar, and error-discovery does what the video says. Install it with `npx skills add`, use `evals-start` instead of `start`, and look at `validate-evaluator` too. It has no license file, so check with the authors before redistributing a modified version.

## Sources

- [ai-evals-course/evals-skills](https://github.com/ai-evals-course/evals-skills): the repository and install instructions.
- [error-discovery SKILL.md](https://github.com/ai-evals-course/evals-skills/blob/main/skills/error-discovery/SKILL.md): phases 1 to 4.
- [error-discovery review-loop.md](https://github.com/ai-evals-course/evals-skills/blob/main/skills/error-discovery/review-loop.md): the breadth and depth review loop.
- [Opus 5 Is The First Model That Gets Worse With Use](https://www.youtube.com/watch?v=4AGg_6s_xf0): the source episode.
- [How to Automate AI Evals (Correctly)](https://youtu.be/tqUDjc1HzO4): the walkthrough linked from the README.
- [AI Evals For Engineers & PMs](https://maven.com/parlance-labs/evals): the paid course.
- [Hamel's evals notes](https://hamel.dev/notes/llm/evals/): free eval resources.
- [The Revenge of the Data Scientist](https://hamel.dev/blog/posts/revenge/): post linked from the README.
- [vercel-labs/skills](https://github.com/vercel-labs/skills): the `npx skills` installer.
- [writing-for-agents by Matt Pocock](https://github.com/mattpocock/skills/tree/main/skills/productivity/writing-for-agents): reference for writing your own skills.
- [Shreya Shankar](https://www.sh-reya.com/): the co-author's site.
- [heyclarity.dev](https://heyclarity.dev/): Robert Ta's review product.
- [Robert Ta on YouTube](https://www.youtube.com/@therobertta): the creator's channel.


## Video description and verification notes

**The repo Robert Ta walks through is `ai-evals-course/evals-skills`, it does what he says it does, and the GitHub API reports its license as null, so calling it an "open source repo" on camera overstates what a viewer actually gets when they clone it.** The clip runs 1 minute 55 seconds at 576x1024. The top half is an unedited screen recording of GitHub in a Mac browser. The bottom half is a video call: two webcam boxes for the first four seconds, Robert Ta on the left and a second man on the right, then a crop to Ta alone for the rest. Burned-in word-by-word captions run across the video pane the whole time. A white sticker over the first five seconds reads "The skill that finds your agent's FAILURES." There is no music, only the call audio.

## What is actually on screen, frame by frame

The browser sits on `github.com/ai-evals-course/evals-skills` for the first 62 seconds without moving. The repo header reads "Skills that guide AI coding agents to help you build product-specific AI evals." The sidebar counters read 449 stars, 36 forks, 7 watching, 1 branch, 0 tags, and the repo is labeled "Public repository." The nav row shows Code, Issues 1, Pull requests, Actions, Projects, More.

The last commit line reads `hamelsmu` / "Bump version to 0.2.0 after router rename" / `b91c188` / "last week." Four entries sit in the file tree: `.claude-plugin` ("Bump version to 0.2.0 after r..."), `skills` ("Rename error-analysis route..."), `README.md` ("Mirror start routing bullets fr..."), and `workflow.png` ("Fix annotation UX: margin no...", 2 months ago).

A GitHub security banner sits across the top of the page for the entire first minute: "Your recovery codes have not been saved in the past year. Make sure you still have them stored somewhere safe by viewing and downloading them again." Nobody dismissed it. The recording is a raw screen share, not a produced walkthrough.

At about 64 seconds the page scrolls to the README, which reads "Eval Skills / Skills that guide AI coding agents to help you build product-specific AI evals (not foundation model benchmarks). / These skills guard against common mistakes we've seen helping 50+ companies and teaching thousands of students in our AI Evals course." Under "Why skills for evals" the visible bullets describe `start` as the entry point routing to `eval-audit` or `error-discovery`.

At about 70 seconds the browser jumps to `skills/error-discovery/`, showing two files, `SKILL.md` and `review-loop.md`, with the commit line "Merge pull request #14 from ai-evals-cour..." Then it opens `SKILL.md`, header "203 lines (151 loc) · 14.6 KB," and stays there through the end. The frontmatter table is visible and legible: name `error-discovery`, description "Run error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes." Ta highlights that description line with his cursor while narrating it. The body below reads "This skill is meant for interactive sessions only," a two-part split between `SKILL.md` (phases 1 through 4) and `review-loop.md` (phase 5), and a "Progress updates" section instructing the agent to narrate each step and "Do not go silent for long stretches."

## The repo checks out, and the license does not exist

I pulled the repo through the GitHub API on 2026-09-15. `ai-evals-course/evals-skills` is real, created 2026-06-23, last pushed 2026-08-31, 65 commits on `main`. The description string matches the on-screen text word for word. Current counters are 598 stars, 49 forks, 9 watchers, 1 open issue, and one tag, `evals--v0.3.1`.

The API returns `"license": null`. A request for `LICENSE` on the `main` branch returns 404. GitHub's "Public repository" label means the code is readable by anyone, and it says nothing about reuse rights. Without a license file, default copyright applies and nobody has been granted permission to copy, modify, or redistribute the skills. For a repo whose entire purpose is for other people to install it into their agent and edit it into their own version, that is a real gap. I am not claiming the authors intend to restrict anyone. I am saying the video's phrase "open source repo" is not supported by what is in the repo, and the fix is one file.

The video credits "Hamel" for the repo. Contributor data shows `hamelsmu` with 49 commits, `shreyashankar` with 15, and `strickvl` with 1. The plugin manifest at `.claude-plugin/plugin.json` lists the author as "Shreya Shankar and Hamel Husain." Shreya Shankar is a PhD candidate in the Data Systems and Foundations group at UC Berkeley, the author of DocETL, and co-author with Husain of the O'Reilly book *Evals for AI Engineers*, listed for release on 2026-10-31. The README even points at her live walkthrough of the error-discovery skill. Crediting only Husain leaves out the co-author whose demo the README uses as its own documentation.

## What error-discovery actually does, which is more than the video says

Ta's summary is accurate as far as it goes: it runs error analysis on a dataset, builds a bespoke annotation UI, selects diverse samples, monitors annotations, and organizes failure modes. He is reading the frontmatter description off the screen, so accuracy is expected. The interesting material is in the 203 lines he scrolls past.

`SKILL.md` runs five phases. Phase 1, "Understand the domain and data," has four sub-steps: read and inventory the data, identify the content structure, identify dimensions of variation, and think about what "bad" means. Phase 2 designs the visual encoding using Gestalt principles, with the README specifying color for categories, spacing for hierarchy, and opacity for importance. Phase 3 builds the interface. The README is explicit about the architecture: "Builds a single-file HTML review app served by a Python stdlib server. No dependencies." Phase 4 clusters the data and picks a starting sample from cluster representatives plus random picks. Phase 5 hands off to `review-loop.md`.

`review-loop.md` is the part the video never reaches, and it is where the loop lives. It defines two modes, breadth and depth. Breadth proposes new samples to widen coverage. Depth scans the corpus for more instances of a failure mode you already found. It also has sections for processing annotations as they arrive, encouraging re-review, and converging. A commit from 2026-08-16 reads "fix: gate the corpus scan, not depth mode entry," which tells you the depth scan was expensive enough to need gating.

There are eight skills in the repo, not the two the video mentions: `evals-start`, `eval-audit`, `error-discovery`, `generate-synthetic-data`, `write-judge-prompt`, `validate-evaluator`, `evaluate-rag`, and `build-review-interface`. The `validate-evaluator` skill is arguably the more important one for the argument Ta is making, since it calibrates an LLM judge against human labels using data splits, TPR and TNR, and bias correction. A rubric nobody calibrated is a rubric you are trusting on vibes, which is the exact failure the video description complains about.

The video also never says how to install it. The README does: `npx skills add https://github.com/ai-evals-course/evals-skills`, using the `npx skills` tool from `vercel-labs/skills`, which has 31,677 stars and was created 2026-01-14. A single skill installs with `--skill error-discovery`.

## The screen recording was already a month stale when the clip posted

The commit shown as HEAD, "Bump version to 0.2.0 after router rename," landed 2026-08-16 and GitHub renders it as "last week" in the recording. The TikTok posted 2026-09-14. The source episode published 2026-09-07. In the gap, the repo moved.

On 2026-08-31 Husain renamed the `start` skill to `evals-start`, bumped the plugin to 0.2.1, added Codex plugin packaging and skill metadata, released 0.3.1, and rewrote the README install section with the commit message "Keep README installation agent-neutral." The repo now has `.agents/plugins` and `.codex-plugin` directories alongside `.claude-plugin`. The README bullet the video shows on screen, "start is the entry point," no longer describes a skill that exists under that name. Anyone who pauses the video and types what they see will look for the wrong thing.

The star count moved too, from 449 in the recording to 598 when I checked, a 33 percent gain in roughly three weeks. That is worth stating plainly because it is the only growth number attached to this repo that I can verify, and neither the video nor the repo makes any claim about adoption.

## The rubric analogy is the strongest part, and the transcript mangles the punchline

The teaching move in this clip is good. Ta asks how an English literature teacher grades 40 to 50 students fairly in a subjective domain, answers that she publishes a rubric covering grammar, prose, whether the bibliography links work, formatting, argument quality, and concision, and then maps that onto grading LLM output "in an unbounded space." He then makes the second half of the mapping, which is the part most explainers skip: a missing bibliography is a failure mode, an essay on the wrong topic is a failure mode, failure modes produce the rubric, and the rubric produces the eval. That ordering matches the repo's own position, which is stated in the README as "You should only write evals after doing this step."

The automatic transcript garbles three things, and one of them matters. "Camel put out this really nice evals skill" is Hamel. "The output of the ln" is the LLM. "Evals are really the mode for any domain specific agent" is "moat," which I confirmed against the source episode's own chapter list, where the segment at 1:14:03 is titled "What's the meta: evals are the moat." Reading the transcript alone loses the entire thesis.

The "Watch the full video" link in the description goes to `youtu.be/4AGg_6s_xf0`, which is titled "Opus 5 Is The First Model That Gets Worse With Use," runs 5,378 seconds, and published 2026-09-07 on Robert Ta's channel with 37 views when I checked. The evals material is one chapter starting at 1:14:03 out of nine chapters covering Anthropic's messaging, a GitHub issue about Opus 5 gating work behind questions, a red team paper, Hugging Face funding talks, and a demo of Ta's own Clarity harness. The second person on camera is Ta's co-founder Jonathan McCoy, named in the episode description. A viewer clicking that link expecting an evals talk gets 74 minutes of other material first.

## Where the claim gets soft

"Evals are the moat for any domain-specific agent or AI product or AI business against the frontier labs" is the load-bearing claim, and Ta labels it correctly on camera as "that's my true belief." I cannot verify it and neither can anyone else. It is a position, not a finding, and it is a position with two visible incentives behind it.

The repo's README links twice to the AI Evals course on Maven, once with a tracking promo code. I checked the course page: *AI Evals For Engineers & PMs*, taught by Husain and Shankar, listed at $4,200 USD for the cohort running 2026-10-10 to 2026-11-21, rated 4.7 out of 5 across 901 reviews. The repo is a free on-ramp to a paid course, which is a normal and disclosed arrangement, and it is worth knowing before you treat the README's advice as neutral.

Ta has an incentive too. The episode this clip comes from ends with a live demo of the Clarity harness from his company, and `heyclarity.dev` describes a product that captures agent runs, queues them for human review with approve, edit, reject, and defer verdicts, and promotes agents through Advisor, Copilot, and Autopilot tiers as they earn reliability. That is the same review-and-annotate loop the error-discovery skill implements. Praising the skill and selling the harness are compatible, and the clip does not mention the overlap.

The README's social proof is self-reported and inconsistent across sources. The README says "50+ companies and teaching thousands of students." Shankar's academic bio says 4,000-plus professionals from 500-plus companies. The Maven page says over 5,000 engineers and PMs have engaged with the material. The O'Reilly book blurb says 35-plus AI products. None of these are auditable from outside, and they do not agree with each other on scale. Treat them as marketing numbers.

## Key Takeaways

- The repo is real and the video identifies it correctly. `ai-evals-course/evals-skills` was created 2026-06-23, has 65 commits, and its description matches the on-screen text exactly.
- The "open source" label does not hold up. The GitHub API returns `"license": null` and there is no `LICENSE` file at `main`. Public visibility is not a license, so no reuse rights have been granted.
- The star count in the recording, 449, is verified as a past state. The repo showed 598 stars, 49 forks, and 9 watchers when I checked on 2026-09-15.
- Ta credits Hamel Husain alone. Shreya Shankar has 15 of the 65 commits, is named as co-author in `plugin.json`, and is the person the README sends you to for the live walkthrough of the skill he is praising.
- The screen recording is stale. The `start` skill was renamed `evals-start` on 2026-08-31, two weeks before the TikTok posted, along with Codex packaging and a plugin bump to 0.3.1.
- The video names two skills. The repo has eight. `validate-evaluator`, which calibrates an LLM judge against human labels with TPR and TNR, goes unmentioned despite being the direct answer to the video's own "how do you know it's objectively getting better" question.
- The error-discovery description Ta reads aloud is accurate because he is reading the SKILL.md frontmatter verbatim. The implementation details he skips, a single-file HTML app served by a Python stdlib server with no dependencies, Gestalt-based visual encoding, and the breadth/depth review loop in `review-loop.md`, are all confirmed in the source files.
- The auto transcript says "mode" where the speaker said "moat." I confirmed this against the source episode's chapter list, which labels the 1:14:03 segment "What's the meta: evals are the moat."
- The "Watch the full video" link goes to an 89-minute episode titled "Opus 5 Is The First Model That Gets Worse With Use," where the evals content is one chapter starting at 1:14:03.
- I could not verify the central claim that evals are a moat against frontier labs, and it is not the kind of claim that can be verified. Ta flags it as opinion on camera, which is the honest framing.
- The social proof numbers are self-reported and inconsistent. The README says 50-plus companies and thousands of students, Shankar's bio says 4,000-plus students from 500-plus companies, and the Maven page says 5,000-plus. The course itself lists at $4,200.

## Resources

- [ai-evals-course/evals-skills](https://github.com/ai-evals-course/evals-skills) - The repo shown on screen for the entire clip, with all eight skills and the install instructions the video omits.
- [error-discovery SKILL.md](https://github.com/ai-evals-course/evals-skills/blob/main/skills/error-discovery/SKILL.md) - The exact 203-line file Ta scrolls through, covering phases 1 through 4.
- [error-discovery review-loop.md](https://github.com/ai-evals-course/evals-skills/blob/main/skills/error-discovery/review-loop.md) - Phase 5, the breadth and depth annotation loop the video never reaches.
- [Opus 5 Is The First Model That Gets Worse With Use](https://www.youtube.com/watch?v=4AGg_6s_xf0) - The 89-minute source episode by Robert Ta with co-founder Jonathan McCoy; the evals chapter starts at 1:14:03.
- [How to Automate AI Evals (Correctly)](https://youtu.be/tqUDjc1HzO4) - The live walkthrough of the error-discovery skill that the repo's own README links to.
- [AI Evals For Engineers & PMs](https://maven.com/parlance-labs/evals) - The $4,200 Husain and Shankar course the README links to twice, including once with a promo code.
- [Hamel's evals notes](https://hamel.dev/notes/llm/evals/) - The free resource hub the README recommends for eval fundamentals.
- [The Revenge of the Data Scientist](https://hamel.dev/blog/posts/revenge/) - The post the README links behind the phrase "easily avoidable footguns."
- [vercel-labs/skills](https://github.com/vercel-labs/skills) - The `npx skills` installer the README uses, 31,677 stars as of 2026-09-15.
- [writing-for-agents by Matt Pocock](https://github.com/mattpocock/skills/tree/main/skills/productivity/writing-for-agents) - The skill-writing reference the README points to for authoring your own.
- [Shreya Shankar](https://www.sh-reya.com/) - Site for the co-author the video does not name.
- [Clarity Harness](https://heyclarity.dev/) - Robert Ta's own agent review product, demoed at the end of the source episode.
- [Robert Ta on YouTube](https://www.youtube.com/@therobertta) - The channel hosting the full episodes these TikToks are cut from.
