evals-skills: Hamel Husain and Shreya Shankar's AI Eval Skills Repo
By Matt Cullerton, founder of Mavric and Apso
evals-skills is a GitHub repository of skills that guide AI coding agents to help you build evals for your own AI product. A TikTok from @therobertta_ (Robert Ta) recommends its error-discovery skill and describes it accurately. The repo has no license file, so calling it "open source" overstates it, and the video credits Hamel Husain alone when Shreya Shankar co-authored it.
What is evals-skills?
The repository is ai-evals-course/evals-skills, created 2026-06-23. Its README says: "Skills that guide AI coding agents to help you build product-specific AI evals (not foundation model benchmarks). These skills guard against common mistakes we've seen helping 50+ companies and teaching thousands of students in our AI Evals course."
It contains eight skills: evals-start, eval-audit, error-discovery, generate-synthetic-data, write-judge-prompt, validate-evaluator, evaluate-rag and build-review-interface. The video mentions two.
On 2026-09-15 the GitHub API showed 598 stars, 49 forks and 65 commits. The screen recording in the video showed 449 stars.
Who made evals-skills?
The video says "Hamel" put it out. The plugin manifest lists the author as "Shreya Shankar and Hamel Husain." Contributor data shows hamelsmu with 49 commits, shreyashankar with 15 and strickvl with 1. Shankar is a PhD candidate in UC Berkeley's Data Systems and Foundations group, the author of DocETL, and co-author with Husain of the O'Reilly book Evals for AI Engineers, listed for release on 2026-10-31. The README links to her live walkthrough of the error-discovery skill.
Is evals-skills open source?
Not formally. The GitHub API returns "license": null, and there is no LICENSE file on main. A public repository can be read by anyone, but without a license, default copyright applies and no one has been granted the right to copy, modify or redistribute it. That matters for a repo meant to be installed and edited. The authors may not intend any restriction, and adding a license file would settle it.
What does the error-discovery skill do?
Its description reads: "Run error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes." The README is firm about order: "You should only write evals after doing this step."
SKILL.md runs four phases:
- Understand the domain and data. Inventory the data, identify its structure and dimensions of variation, and define what "bad" looks like.
- Design the visual encoding using Gestalt principles: color for categories, spacing for hierarchy, opacity for importance.
- Build the interface. The README says it "Builds a single-file HTML review app served by a Python stdlib server. No dependencies."
- Cluster the data and pick a starting sample from cluster representatives plus random picks.
Phase 5 lives in review-loop.md. It has a breadth mode, which proposes new samples to widen coverage, and a depth mode, which scans the data for more cases of a failure you already found. The skill is "meant for interactive sessions only" and tells the agent "Do not go silent for long stretches."
How do you check that an LLM judge is right?
The video asks how you know an AI product is "objectively getting better," but does not mention the skill that answers it. validate-evaluator calibrates an LLM judge against human labels using data splits, true positive and true negative rates, and bias correction.
How to install evals-skills
The README uses the npx skills tool from vercel-labs/skills:
npx skills add https://github.com/ai-evals-course/evals-skills
Add --skill error-discovery to install one skill. On 2026-08-31 the start skill was renamed evals-start and the plugin gained Codex packaging, so the "start" skill shown in the video no longer exists under that name.
Why the rubric analogy works
Ta's explanation is the best part of the clip. An English teacher grades 40 to 50 subjective essays fairly by publishing a rubric covering grammar, prose, working bibliography links, formatting, argument and concision. For LLM output, a missing bibliography or an essay on the wrong topic is a failure mode, failure modes produce the rubric, and the rubric produces the eval. That is the same order the repo prescribes.
He calls evals "the moat for any domain-specific agent" and labels it "my true belief." The auto-captions render "moat" as "mode," but the source episode's chapter at 1:14:03 is titled "What's the meta: evals are the moat."
Who benefits from this advice?
The README links twice to the Maven course AI Evals For Engineers & PMs by Husain and Shankar, listed at $4,200 for the cohort running 2026-10-10 to 2026-11-21 and rated 4.7 out of 5 across 901 reviews. The repo is a free entry point to a paid course.
Ta has a related product. The source episode ends with a demo of his company's agent review harness, which queues agent runs for human review with approve, edit, reject and defer verdicts. The clip does not mention that overlap.
The "Watch the full video" link goes to an 89-minute episode titled "Opus 5 Is The First Model That Gets Worse With Use," where the evals discussion starts at 1:14:03.
Bottom line
evals-skills is a real, actively maintained set of eval skills from Hamel Husain and Shreya Shankar, and error-discovery does what the video says. Install it with npx skills add, use evals-start instead of start, and look at validate-evaluator too. It has no license file, so check with the authors before redistributing a modified version.
Sources
- ai-evals-course/evals-skills: the repository and install instructions.
- error-discovery SKILL.md: phases 1 to 4.
- error-discovery review-loop.md: the breadth and depth review loop.
- Opus 5 Is The First Model That Gets Worse With Use: the source episode.
- How to Automate AI Evals (Correctly): the walkthrough linked from the README.
- AI Evals For Engineers & PMs: the paid course.
- Hamel's evals notes: free eval resources.
- The Revenge of the Data Scientist: post linked from the README.
- vercel-labs/skills: the
npx skillsinstaller. - writing-for-agents by Matt Pocock: reference for writing your own skills.
- Shreya Shankar: the co-author's site.
- heyclarity.dev: Robert Ta's review product.
- Robert Ta on YouTube: the creator's channel.
Published September 14, 2026. Based on a TikTok by @therobertta_. Full video notes (markdown)