Evals Replace Red-Green-Refactor as the Core Loop for AI Development

This 56-second video, lesson 9 of 10 in Vibe Academy's "Intro to Evals" series, maps test-driven development onto AI engineering. The core claim is that the TDD discipline survives intact in AI work, but the unit of verification changes from a pass/fail test on one case to a score over a distribution, and your eval set quietly becomes your spec.

The New Loop

The video presents a four-step cycle on a static card that stays on screen the entire time, labeled "THE LOOP" with the header "Evals are the new TDD":

1. Write the eval first
2. Change the prompt, the model, the pipeline
3. Score it. Compare to last time.
4. Keep the winner. Repeat.

The narration frames this as a direct descendant of red-green-refactor. You write the eval while you are defining the feature and before you tune anything, the same way TDD asks you to write the failing test before the implementation. Then every change to the system gets scored against the previous score, and the winner survives.

What Changed From Classic TDD

The speaker is explicit that the discipline is the same and only the unit changed. Classic TDD verifies with a binary pass or fail on a single case. AI systems are stochastic, so verification becomes a score over a distribution of cases. That shift explains why the loop keys on "compare to last time" rather than "make it green." There is no green. There is only better or worse than the last run, measured across the eval set.

The on-screen kicker "RED · GREEN · REFACTOR, PROMOTED" reinforces the framing. The practice did not die when prompts replaced functions. It moved up a level of abstraction.

The Eval Set Becomes Your Spec

The part the speaker calls sneaky is that the eval set stops being a QA artifact and becomes the executable definition of good. The card states it directly: "Your eval set is your spec." Whatever cases you score against is what your system optimizes toward, whether or not that matches your written requirements. This mirrors a known TDD dynamic where the test suite ends up being the most accurate documentation of intended behavior, except here the stakes are higher because model and pipeline changes are judged only by that score.

The practical implication is that eval curation deserves the same care as spec writing. Gaps in the eval set are gaps in the definition of the product.

Presentation Format

Visually the video is one designed card for the full runtime, with the creator speaking in a cutout at the bottom and word-by-word captions below. The header shows "09 / 10," placing this as the penultimate lesson. The closer teases the final lesson as one question that should permanently replace "does it work?" for AI systems, which follows naturally from the score-over-distribution argument: a binary "does it work" has no meaning when output quality is a distribution.

Key Takeaways

• Write the eval before tuning anything, at the same time you define the feature.
• Iterate by changing one thing (prompt, model, or pipeline), scoring it, comparing to the last score, and keeping the winner.
• Verification for AI systems is a score over a distribution, replacing pass/fail on single cases.
• Your eval set functions as your spec, so curate it with the same rigor you would apply to requirements.
• The question "does it work?" gets replaced by comparative scoring against your eval set.

Resources

• Vibe Academy - the creator's free "Intro to Evals" course referenced in the video