---
title: "Agent Evals From Traces: Why Playwright Tests Are Not Enough"
date: "2026-11-11"
video_id: "7706977846949891361"
tiktok_url: "https://www.tiktok.com/@therobertta_/video/7706977846949891361"
creator: "@therobertta_"
category: "AI"
tags: ["agent evals", "llm traces", "arize", "playwright", "ai agents"]
excerpt: "Robert Ta argues traces should be the base of AI agent evals. How that matches Arize's trace-to-eval workflow, and the judge-validation step the clip skips."
---
Agent evals built from traces means reading the logs of what your agent actually did, turning real failures into a test dataset and measuring changes against it. A TikTok from Robert Ta (@therobertta_) argues that traces are the starting point for evaluating AI agents, and Arize's published workflow agrees. The clip's caption, "Playwright won't tell you if your agent works," goes further than anything the speakers say.

## What the video argues

The clip is a 90-second excerpt from a longer YouTube interview. It describes an engineer who is "used to deterministic systems like unit tests, integration tests, end-to-end testing, maestro, playwright." With agents, the speaker says, "now it's a what's for dinner sort of non-deterministic land."

He makes the point with a rising sequence ("two plus two," "two times two," "two to the end's power") to show how quickly the possible states multiply. That is a figure of speech rather than a measured claim.

His fix is to treat traces as "the first class citizen primitive" that "can inform what are in your evals data set." That, he says, "gets you to objective measurable improvement instead of just vibing."

## Why can't Playwright test an AI agent?

It can test part of one. In the clip, Playwright and Maestro appear only in a list of tools for deterministic software, and the speaker never says they are useless for agents. A browser automation test can still confirm that an agent's web interface loads, accepts input and shows a reply.

What a scripted test cannot do is grade whether an open-ended reply was good. A fixed input with a fixed expected output does not fit an answer that changes from run to run. That narrower point is the one the speaker's argument supports. The caption turns a claim about scope into a claim that the tools fail.

## How to build agent evals from traces

Arize's tutorial "How to build agent evals from traces," last updated August 19, 2026, describes the same loop in more detail. It frames evals as tests for AI and traces as logs for AI. The steps include:

- Turn product requirements into conditions a trace can show.
- Review representative runs end to end and record concrete failures.
- Group failures into a taxonomy and pick an evaluator type for each.
- Validate any LLM judge against human labels before trusting it.
- Save both failing and passing examples into datasets.
- Sample live traffic and promote confirmed failures into a regression suite.

The tutorial uses three kinds of evaluators. Code checks objective conditions, LLM judges handle semantic criteria, and human review covers ambiguous or high-stakes cases. That last point differs from a pass or fail assertion in a unit test.

LangChain's material separates offline evals on a fixed dataset from online evals on production traffic, which fits the same pattern.

## What the clip leaves out

The video promises "objective measurable improvement." In Arize's version, that depends on validation. The tutorial says to compare an LLM judge against human labels, look at precision and recall counts, and review disagreements before relying on the judge. Skipping that step leaves a dataset that looks objective without being checked.

The clip also says nothing about how many traces to read, how to sample them or how to define success first. Arize starts with that step. Because this is a short cut from a longer video, the full interview may cover it.

## Do AI tools help you ship a production agent?

The speaker says "AI tools is really good for vibe coding," but not great at "helping you get from like an initial agent to like a production agent." He names no study or product, so this is his opinion from experience.

Arize describes coding agents and LLM judges doing part of the trace review, which shows vendors are building tools for this step. That still fits his larger point. He says you "don't really know it's good for users" until you analyze their traces, which matches Arize's advice to read traces yourself before automating.

## The article on screen

Every frame shows an X article titled "Agent Traces: Patterns for Capturing Agent Execution" by Josh Rosen (@JoshARosen). Its summary box reads: "Agent tracing captures decisions, actions, and delegation across sessions, processes, and systems the agent modifies, as no single instrumentation point holds the full picture." The audio never mentions it, and a search for the title and author did not find the article. The second person in the interview is not named in the clip or its description.

## Bottom line

Reading traces, building an eval dataset from real failures and promoting confirmed production failures into a regression suite is a documented practice, and Arize lays it out step by step. Arize also says to validate LLM judges against human labels, a step the clip skips. Playwright still has a role in checking an agent's interface, even though it cannot grade the agent's answers.

## Sources

- [How to build agent evals from traces, Arize, last updated August 19, 2026](https://arize.com/resources/agent-evals-from-traces/): the trace-to-eval loop, judge validation and evaluator types.
- [YouTube link from the video description](https://www.youtube.com/watch?v=fNwGKZ8h5tU&feature=youtu.be): the full interview, which did not load when fetched.


## Video description and verification notes

**The clip runs 90.60 s (ffprobe container duration; the video stream is 90.58 s and the audio 90.60 s). It is 576x1024 (9:16) H.264 High profile at 24 fps, about 469.6 kbps for video and about 32 kbps HE-AACv2 stereo audio at 44.1 kHz, about 506.6 kbps overall, in an MP4 (isom) container of 5,736,946 bytes (about 5.7 MB). metadata.json gives format id h264_540p_506556-1 and lists fps as null, so the 24 fps figure comes from ffprobe. It was posted at epoch 1794420608, which is 2026-11-11 18:10:08 UTC, by @therobertta_ (nickname "Robert Ta"). The audio is credited as "original sound" by Robert Ta. The description is 483 characters, links a full YouTube video (https://youtu.be/fNwGKZ8h5tU) and carries the hashtags #ai #aiagents #evals. Engagement at archive capture (2026-11-11 21:01:21 UTC, about three hours after posting) was 435 views, 8 likes, 0 comments, 5 reposts and 7 saves. I read 5 frames: 1, 10, 20, 40 and 45, which at 2 s sampling sit at roughly 0 s, 18 s, 38 s, 78 s and 88 s. The transcript has 27 lines, 27 Whisper segments (ids 0 to 26) and roughly 290 words. Every frame I read has the same layout. The top half is a screenshot of an X article page by "Josh Rosen @JoshARosen" titled "Agent Traces: Patterns for Capturing Agent Execution", with counters 23, 59, 443 and 50K and a Summary box reading "Agent tracing captures decisions, actions, and delegation across sessions, processes, and systems the agent modifies, as no single instrumentation point holds the full picture." The bottom half is burned-in captions over a split-screen interview. Frame 1 shows both people side by side, with the caption "Playwright won't tell you if your agent works" and the word "an engineer" below it. Frames 10, 20 and 40 and 45 alternate between a long-haired man in a grey top and a man in a dark T-shirt wearing white earbuds. The captions in those frames read "All right,", "that this agent", "agent accordingly." and "to customers." No terminal, trace viewer, eval dashboard, code or tool run appears in any frame I read. The spoken argument is that engineers used to deterministic tests now work with non-deterministic agents, that traces should feed the eval dataset, and that reading traces is the skill that turns a vibe-coded agent into a production agent. The final Whisper segment (id 26, "Thank you.") is stamped 90.4 to 98.2 s against 90.60 s of audio, and its decoding temperature is 1.0 with a compression ratio of 0.56, so I treat it as invented. The other 26 segments end at 90.4 s or earlier.**

## The "two plus two" framing is a metaphor about test determinism, and it holds in outline

At 10.84 s the video says the engineer is "used to deterministic systems like unit tests, integration tests, end-to-end testing, maestro, playwright". It then says (segment 7, ending 25.06 s) "now it's a what's for dinner sort of non-deterministic land". The arithmetic sequence ("two plus two", "two times two", "two to the end's power") describes a rising count of possible states. It is a rhetorical device, not a measured claim, so there is nothing to verify numerically. The underlying point is that a scripted test with a fixed input and expected output cannot tell you whether an agent's open-ended answer was good. The Arize tutorial I fetched states a close version of the same idea. It says evals are tests for AI and traces are logs for AI, and it makes the evaluator a different kind of object from a pass/fail assertion: code checks for objective conditions, LLM judges for semantic criteria, and human review for ambiguous or high-stakes cases.

The on-screen caption "Playwright won't tell you if your agent works" is the editor's headline, not a sentence from the transcript. Playwright and Maestro appear in the spoken list as examples of tools the engineer already uses. At 14.88 s the speaker says those tools are how "you're used to compounding your software". He never says they are useless for agents. A browser-automation test can still confirm that an agent's web UI loads, submits input and renders output. What it cannot do is grade the quality of the agent's reply, which is the narrower claim the speaker's argument supports. The caption therefore compresses a claim about scope into a claim about failure, and I rate it a partial overstatement.

## "Traces are the first class citizen primitive" is consistent with vendor practice, but the video gives no source for it

At 27.36 s and after, the video says traces are "the first class citizen primitive" and "can inform what are in your evals data set". It then says (segments 12, ending 44.08 s) that this "gets you to objective measurable improvement instead of just vibing". The workflow it describes is read traces, build a dataset, evaluate, then update the agent. I found this workflow in the Arize tutorial "How to build agent evals from traces" (last updated August 19, 2026). That tutorial tells readers to review representative runs end to end and record concrete failures before grouping them. It then has them build a failure taxonomy and choose evaluator types. It also says to validate any LLM judge against human labels and save both failing and passing examples into datasets. It ends with sampling live traffic and promoting confirmed failures into a regression suite. A web search I ran also returned LangChain material that separates offline evals on a fixed dataset from online evals on production traffic, and FutureAGI and Datadog pages that describe the same trace-to-dataset loop. I did not fetch those three pages, so I rely on them only for the existence of the pattern, not for details.

Two points of difference are worth stating. First, the video says "objective measurable improvement" will follow. The Arize steps make that conditional on validation. The tutorial says to compare an LLM judge against human labels, with raw counts of precision and recall, and to review disagreements before trusting it. The video skips that step, so a viewer could read it as saying a dataset built from traces is objective on its own. Second, the video says nothing about how many traces to read, how to sample them or how to define success first. The Arize tutorial begins with exactly that, turning product requirements into conditions a trace can show. The video is a 90 s excerpt from a longer YouTube video, so the omission may be a result of cutting.

## The claim that AI tools help with vibe coding but not the move to a production agent is the speaker's opinion

At 46.98 s the speaker says "AI tools is really good for vibe coding", and at 56.42 s he says AI tools are not great at "helping you get from like an initial agent to like a production agent". No benchmark, study or product is named, and no frame shows a tool being used. I could not find a primary source that measures this. It is an experience-based opinion, and I mark it Unverified. Vendor material I found points the other way in one respect: the Arize tutorial and the Datadog "eval-trace RCA" skill listing both describe coding agents or LLM judges doing part of the trace review. The Arize docs hit in my search described annotating traces "by hand or with a coding-agent skill". That does not contradict the speaker's point that a human still has to look at the data. His own statement at 70.58 s to 78.2 s is that you "don't really know it's good for users" until you analyze their traces, and that matches the Arize advice to read traces before automating.

The statement about the Whisper output "train agent with the vowels" at 70.58 s reads as a mis-transcription, most likely "evals" ("the vowels" sounds like "the evals"). The sentence "you can sort of like train agent with the evals and say, is this good in the context" makes sense with that reading, and I did not confirm it against the captions, because no frame I read covers that second. Treat the sentence as ambiguous.

## The on-screen article could not be verified

The X article shown in every frame I read, "Agent Traces: Patterns for Capturing Agent Execution" by Josh Rosen (@JoshARosen), is visible but never named or discussed in the audio. I searched for the exact title and author and found no page for it. The search returned unrelated sources, including the arXiv paper "AgentTrace: A Structured Logging Framework for Agent System Observability". I did not fetch x.com, and I did not follow the article's text beyond the visible Summary box. I therefore cannot say what the article claims beyond the two lines readable in the frames, and its counters (23, 59, 443, 50K) are screenshot values that I could not check. The video links to a YouTube video, but my fetch of the redirected URL returned only footer text, with no title, channel, date or transcript, so I could not check what the long version adds. The identity of the second speaker is not stated in the transcript, the description or any caption I read. I do not name him.

## Key Takeaways

- **Verified:** The workflow of reading traces, building an eval dataset from them and updating the agent matches the Arize tutorial updated August 19, 2026, including promotion of confirmed production failures into a regression suite.
- **Partial correction:** The caption "Playwright won't tell you if your agent works" is the editor's wording. At 14.88 s the speaker only lists Playwright and Maestro as tools for deterministic software. He does not say they cannot test an agent's UI.
- **Partial correction:** At 44.08 s the video says trace-based evals give "objective measurable improvement". The Arize tutorial makes that depend on validating judges against human labels, which the clip does not mention.
- **Unverified:** The claim that AI tools are weak at taking an initial agent to a production agent (56.42 s) has no cited source or demo, and no frame shows any tool in use.
- **Unverified:** The Josh Rosen X article on screen could not be found by title or author, so its content beyond the visible Summary box is unchecked, as are the YouTube video's contents.
- **Context:** The final Whisper segment "Thank you." (90.4 to 98.2 s) runs past the 90.60 s audio at temperature 1.0, so I treat it as invented. Whisper also appears to have rendered "evals" as "vowels" at 70.58 s.

## Resources

- [How to build agent evals from traces, Arize, last updated August 19, 2026](https://arize.com/resources/agent-evals-from-traces/) establishes the twelve-step loop from reading traces to regression suites, the judge-validation step and the evaluator types I compare the video against.
- [LLM observability & monitoring: how to evaluate agent behavior, LangChain](https://www.langchain.com/resources/llm-monitoring-observability) appeared in my search results. I did not fetch it and cite it only as evidence that the offline/online eval split exists.
- [YouTube link from the video description](https://www.youtube.com/watch?v=fNwGKZ8h5tU&feature=youtu.be) was fetched but returned only footer text, so it establishes nothing about the long video.
