<- all tokdocs
2026.09.14@therobertta_

The Eval Skills Repo in This TikTok Is Real and Well Built, but It Ships Without a License and the Speaker Credits Only Half of Its Authors

AI
Watch on TikTok

View on TikTok ->

The repo Robert Ta walks through is ai-evals-course/evals-skills, it does what he says it does, and the GitHub API reports its license as null, so calling it an "open source repo" on camera overstates what a viewer actually gets when they clone it. The clip runs 1 minute 55 seconds at 576x1024. The top half is an unedited screen recording of GitHub in a Mac browser. The bottom half is a video call: two webcam boxes for the first four seconds, Robert Ta on the left and a second man on the right, then a crop to Ta alone for the rest. Burned-in word-by-word captions run across the video pane the whole time. A white sticker over the first five seconds reads "The skill that finds your agent's FAILURES." There is no music, only the call audio.

What is actually on screen, frame by frame

The browser sits on github.com/ai-evals-course/evals-skills for the first 62 seconds without moving. The repo header reads "Skills that guide AI coding agents to help you build product-specific AI evals." The sidebar counters read 449 stars, 36 forks, 7 watching, 1 branch, 0 tags, and the repo is labeled "Public repository." The nav row shows Code, Issues 1, Pull requests, Actions, Projects, More.

The last commit line reads hamelsmu / "Bump version to 0.2.0 after router rename" / b91c188 / "last week." Four entries sit in the file tree: .claude-plugin ("Bump version to 0.2.0 after r..."), skills ("Rename error-analysis route..."), README.md ("Mirror start routing bullets fr..."), and workflow.png ("Fix annotation UX: margin no...", 2 months ago).

A GitHub security banner sits across the top of the page for the entire first minute: "Your recovery codes have not been saved in the past year. Make sure you still have them stored somewhere safe by viewing and downloading them again." Nobody dismissed it. The recording is a raw screen share, not a produced walkthrough.

At about 64 seconds the page scrolls to the README, which reads "Eval Skills / Skills that guide AI coding agents to help you build product-specific AI evals (not foundation model benchmarks). / These skills guard against common mistakes we've seen helping 50+ companies and teaching thousands of students in our AI Evals course." Under "Why skills for evals" the visible bullets describe start as the entry point routing to eval-audit or error-discovery.

At about 70 seconds the browser jumps to skills/error-discovery/, showing two files, SKILL.md and review-loop.md, with the commit line "Merge pull request #14 from ai-evals-cour..." Then it opens SKILL.md, header "203 lines (151 loc) · 14.6 KB," and stays there through the end. The frontmatter table is visible and legible: name error-discovery, description "Run error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes." Ta highlights that description line with his cursor while narrating it. The body below reads "This skill is meant for interactive sessions only," a two-part split between SKILL.md (phases 1 through 4) and review-loop.md (phase 5), and a "Progress updates" section instructing the agent to narrate each step and "Do not go silent for long stretches."

The repo checks out, and the license does not exist

I pulled the repo through the GitHub API on 2026-09-15. ai-evals-course/evals-skills is real, created 2026-06-23, last pushed 2026-08-31, 65 commits on main. The description string matches the on-screen text word for word. Current counters are 598 stars, 49 forks, 9 watchers, 1 open issue, and one tag, evals--v0.3.1.

The API returns "license": null. A request for LICENSE on the main branch returns 404. GitHub's "Public repository" label means the code is readable by anyone, and it says nothing about reuse rights. Without a license file, default copyright applies and nobody has been granted permission to copy, modify, or redistribute the skills. For a repo whose entire purpose is for other people to install it into their agent and edit it into their own version, that is a real gap. I am not claiming the authors intend to restrict anyone. I am saying the video's phrase "open source repo" is not supported by what is in the repo, and the fix is one file.

The video credits "Hamel" for the repo. Contributor data shows hamelsmu with 49 commits, shreyashankar with 15, and strickvl with 1. The plugin manifest at .claude-plugin/plugin.json lists the author as "Shreya Shankar and Hamel Husain." Shreya Shankar is a PhD candidate in the Data Systems and Foundations group at UC Berkeley, the author of DocETL, and co-author with Husain of the O'Reilly book Evals for AI Engineers, listed for release on 2026-10-31. The README even points at her live walkthrough of the error-discovery skill. Crediting only Husain leaves out the co-author whose demo the README uses as its own documentation.

What error-discovery actually does, which is more than the video says

Ta's summary is accurate as far as it goes: it runs error analysis on a dataset, builds a bespoke annotation UI, selects diverse samples, monitors annotations, and organizes failure modes. He is reading the frontmatter description off the screen, so accuracy is expected. The interesting material is in the 203 lines he scrolls past.

SKILL.md runs five phases. Phase 1, "Understand the domain and data," has four sub-steps: read and inventory the data, identify the content structure, identify dimensions of variation, and think about what "bad" means. Phase 2 designs the visual encoding using Gestalt principles, with the README specifying color for categories, spacing for hierarchy, and opacity for importance. Phase 3 builds the interface. The README is explicit about the architecture: "Builds a single-file HTML review app served by a Python stdlib server. No dependencies." Phase 4 clusters the data and picks a starting sample from cluster representatives plus random picks. Phase 5 hands off to review-loop.md.

review-loop.md is the part the video never reaches, and it is where the loop lives. It defines two modes, breadth and depth. Breadth proposes new samples to widen coverage. Depth scans the corpus for more instances of a failure mode you already found. It also has sections for processing annotations as they arrive, encouraging re-review, and converging. A commit from 2026-08-16 reads "fix: gate the corpus scan, not depth mode entry," which tells you the depth scan was expensive enough to need gating.

There are eight skills in the repo, not the two the video mentions: evals-start, eval-audit, error-discovery, generate-synthetic-data, write-judge-prompt, validate-evaluator, evaluate-rag, and build-review-interface. The validate-evaluator skill is arguably the more important one for the argument Ta is making, since it calibrates an LLM judge against human labels using data splits, TPR and TNR, and bias correction. A rubric nobody calibrated is a rubric you are trusting on vibes, which is the exact failure the video description complains about.

The video also never says how to install it. The README does: npx skills add https://github.com/ai-evals-course/evals-skills, using the npx skills tool from vercel-labs/skills, which has 31,677 stars and was created 2026-01-14. A single skill installs with --skill error-discovery.

The screen recording was already a month stale when the clip posted

The commit shown as HEAD, "Bump version to 0.2.0 after router rename," landed 2026-08-16 and GitHub renders it as "last week" in the recording. The TikTok posted 2026-09-14. The source episode published 2026-09-07. In the gap, the repo moved.

On 2026-08-31 Husain renamed the start skill to evals-start, bumped the plugin to 0.2.1, added Codex plugin packaging and skill metadata, released 0.3.1, and rewrote the README install section with the commit message "Keep README installation agent-neutral." The repo now has .agents/plugins and .codex-plugin directories alongside .claude-plugin. The README bullet the video shows on screen, "start is the entry point," no longer describes a skill that exists under that name. Anyone who pauses the video and types what they see will look for the wrong thing.

The star count moved too, from 449 in the recording to 598 when I checked, a 33 percent gain in roughly three weeks. That is worth stating plainly because it is the only growth number attached to this repo that I can verify, and neither the video nor the repo makes any claim about adoption.

The rubric analogy is the strongest part, and the transcript mangles the punchline

The teaching move in this clip is good. Ta asks how an English literature teacher grades 40 to 50 students fairly in a subjective domain, answers that she publishes a rubric covering grammar, prose, whether the bibliography links work, formatting, argument quality, and concision, and then maps that onto grading LLM output "in an unbounded space." He then makes the second half of the mapping, which is the part most explainers skip: a missing bibliography is a failure mode, an essay on the wrong topic is a failure mode, failure modes produce the rubric, and the rubric produces the eval. That ordering matches the repo's own position, which is stated in the README as "You should only write evals after doing this step."

The automatic transcript garbles three things, and one of them matters. "Camel put out this really nice evals skill" is Hamel. "The output of the ln" is the LLM. "Evals are really the mode for any domain specific agent" is "moat," which I confirmed against the source episode's own chapter list, where the segment at 1:14:03 is titled "What's the meta: evals are the moat." Reading the transcript alone loses the entire thesis.

The "Watch the full video" link in the description goes to youtu.be/4AGg_6s_xf0, which is titled "Opus 5 Is The First Model That Gets Worse With Use," runs 5,378 seconds, and published 2026-09-07 on Robert Ta's channel with 37 views when I checked. The evals material is one chapter starting at 1:14:03 out of nine chapters covering Anthropic's messaging, a GitHub issue about Opus 5 gating work behind questions, a red team paper, Hugging Face funding talks, and a demo of Ta's own Clarity harness. The second person on camera is Ta's co-founder Jonathan McCoy, named in the episode description. A viewer clicking that link expecting an evals talk gets 74 minutes of other material first.

Where the claim gets soft

"Evals are the moat for any domain-specific agent or AI product or AI business against the frontier labs" is the load-bearing claim, and Ta labels it correctly on camera as "that's my true belief." I cannot verify it and neither can anyone else. It is a position, not a finding, and it is a position with two visible incentives behind it.

The repo's README links twice to the AI Evals course on Maven, once with a tracking promo code. I checked the course page: AI Evals For Engineers & PMs, taught by Husain and Shankar, listed at $4,200 USD for the cohort running 2026-10-10 to 2026-11-21, rated 4.7 out of 5 across 901 reviews. The repo is a free on-ramp to a paid course, which is a normal and disclosed arrangement, and it is worth knowing before you treat the README's advice as neutral.

Ta has an incentive too. The episode this clip comes from ends with a live demo of the Clarity harness from his company, and heyclarity.dev describes a product that captures agent runs, queues them for human review with approve, edit, reject, and defer verdicts, and promotes agents through Advisor, Copilot, and Autopilot tiers as they earn reliability. That is the same review-and-annotate loop the error-discovery skill implements. Praising the skill and selling the harness are compatible, and the clip does not mention the overlap.

The README's social proof is self-reported and inconsistent across sources. The README says "50+ companies and teaching thousands of students." Shankar's academic bio says 4,000-plus professionals from 500-plus companies. The Maven page says over 5,000 engineers and PMs have engaged with the material. The O'Reilly book blurb says 35-plus AI products. None of these are auditable from outside, and they do not agree with each other on scale. Treat them as marketing numbers.

Key Takeaways

  • The repo is real and the video identifies it correctly. ai-evals-course/evals-skills was created 2026-06-23, has 65 commits, and its description matches the on-screen text exactly.
  • The "open source" label does not hold up. The GitHub API returns "license": null and there is no LICENSE file at main. Public visibility is not a license, so no reuse rights have been granted.
  • The star count in the recording, 449, is verified as a past state. The repo showed 598 stars, 49 forks, and 9 watchers when I checked on 2026-09-15.
  • Ta credits Hamel Husain alone. Shreya Shankar has 15 of the 65 commits, is named as co-author in plugin.json, and is the person the README sends you to for the live walkthrough of the skill he is praising.
  • The screen recording is stale. The start skill was renamed evals-start on 2026-08-31, two weeks before the TikTok posted, along with Codex packaging and a plugin bump to 0.3.1.
  • The video names two skills. The repo has eight. validate-evaluator, which calibrates an LLM judge against human labels with TPR and TNR, goes unmentioned despite being the direct answer to the video's own "how do you know it's objectively getting better" question.
  • The error-discovery description Ta reads aloud is accurate because he is reading the SKILL.md frontmatter verbatim. The implementation details he skips, a single-file HTML app served by a Python stdlib server with no dependencies, Gestalt-based visual encoding, and the breadth/depth review loop in review-loop.md, are all confirmed in the source files.
  • The auto transcript says "mode" where the speaker said "moat." I confirmed this against the source episode's chapter list, which labels the 1:14:03 segment "What's the meta: evals are the moat."
  • The "Watch the full video" link goes to an 89-minute episode titled "Opus 5 Is The First Model That Gets Worse With Use," where the evals content is one chapter starting at 1:14:03.
  • I could not verify the central claim that evals are a moat against frontier labs, and it is not the kind of claim that can be verified. Ta flags it as opinion on camera, which is the honest framing.
  • The social proof numbers are self-reported and inconsistent. The README says 50-plus companies and thousands of students, Shankar's bio says 4,000-plus students from 500-plus companies, and the Maven page says 5,000-plus. The course itself lists at $4,200.

Resources

Published September 14, 2026. Writeup generated from a favorited TikTok.