Agent Evals From Traces: Why Playwright Tests Are Not Enough
By Matt Cullerton, founder of Mavric and Apso
Agent evals built from traces means reading the logs of what your agent actually did, turning real failures into a test dataset and measuring changes against it. A TikTok from Robert Ta (@therobertta_) argues that traces are the starting point for evaluating AI agents, and Arize's published workflow agrees. The clip's caption, "Playwright won't tell you if your agent works," goes further than anything the speakers say.
What the video argues
The clip is a 90-second excerpt from a longer YouTube interview. It describes an engineer who is "used to deterministic systems like unit tests, integration tests, end-to-end testing, maestro, playwright." With agents, the speaker says, "now it's a what's for dinner sort of non-deterministic land."
He makes the point with a rising sequence ("two plus two," "two times two," "two to the end's power") to show how quickly the possible states multiply. That is a figure of speech rather than a measured claim.
His fix is to treat traces as "the first class citizen primitive" that "can inform what are in your evals data set." That, he says, "gets you to objective measurable improvement instead of just vibing."
Why can't Playwright test an AI agent?
It can test part of one. In the clip, Playwright and Maestro appear only in a list of tools for deterministic software, and the speaker never says they are useless for agents. A browser automation test can still confirm that an agent's web interface loads, accepts input and shows a reply.
What a scripted test cannot do is grade whether an open-ended reply was good. A fixed input with a fixed expected output does not fit an answer that changes from run to run. That narrower point is the one the speaker's argument supports. The caption turns a claim about scope into a claim that the tools fail.
How to build agent evals from traces
Arize's tutorial "How to build agent evals from traces," last updated August 19, 2026, describes the same loop in more detail. It frames evals as tests for AI and traces as logs for AI. The steps include:
- Turn product requirements into conditions a trace can show.
- Review representative runs end to end and record concrete failures.
- Group failures into a taxonomy and pick an evaluator type for each.
- Validate any LLM judge against human labels before trusting it.
- Save both failing and passing examples into datasets.
- Sample live traffic and promote confirmed failures into a regression suite.
The tutorial uses three kinds of evaluators. Code checks objective conditions, LLM judges handle semantic criteria, and human review covers ambiguous or high-stakes cases. That last point differs from a pass or fail assertion in a unit test.
LangChain's material separates offline evals on a fixed dataset from online evals on production traffic, which fits the same pattern.
What the clip leaves out
The video promises "objective measurable improvement." In Arize's version, that depends on validation. The tutorial says to compare an LLM judge against human labels, look at precision and recall counts, and review disagreements before relying on the judge. Skipping that step leaves a dataset that looks objective without being checked.
The clip also says nothing about how many traces to read, how to sample them or how to define success first. Arize starts with that step. Because this is a short cut from a longer video, the full interview may cover it.
Do AI tools help you ship a production agent?
The speaker says "AI tools is really good for vibe coding," but not great at "helping you get from like an initial agent to like a production agent." He names no study or product, so this is his opinion from experience.
Arize describes coding agents and LLM judges doing part of the trace review, which shows vendors are building tools for this step. That still fits his larger point. He says you "don't really know it's good for users" until you analyze their traces, which matches Arize's advice to read traces yourself before automating.
The article on screen
Every frame shows an X article titled "Agent Traces: Patterns for Capturing Agent Execution" by Josh Rosen (@JoshARosen). Its summary box reads: "Agent tracing captures decisions, actions, and delegation across sessions, processes, and systems the agent modifies, as no single instrumentation point holds the full picture." The audio never mentions it, and a search for the title and author did not find the article. The second person in the interview is not named in the clip or its description.
Bottom line
Reading traces, building an eval dataset from real failures and promoting confirmed production failures into a regression suite is a documented practice, and Arize lays it out step by step. Arize also says to validate LLM judges against human labels, a step the clip skips. Playwright still has a role in checking an agent's interface, even though it cannot grade the agent's answers.
Sources
- How to build agent evals from traces, Arize, last updated August 19, 2026: the trace-to-eval loop, judge validation and evaluator types.
- YouTube link from the video description: the full interview, which did not load when fetched.
Published November 11, 2026. Based on a TikTok by @therobertta_. Full video notes (markdown)