AI Agent Evaluation: Grade the Output or the Trajectory?
By Matt Cullerton, founder of Mavric and Apso
A TikTok from @farisallaf lists five things he says he learned in his first week at Deloitte about building enterprise AI agents: entity resolution, ontology, progressive disclosure, trajectory evaluation and context compression. Four of them match published sources closely. The fourth, grading an agent on the whole path it took, is something Google ships as a product and Anthropic's own eval guidance argues against.
The video is a single 71-second talking-head take with no slides or code, so the spoken list is all there is. A legal pad and pen on the desk carry "Deloitte University" branding.
What are entity resolution and ontology?
These are his first two items, and both are core terms in Palantir Foundry. He describes entity resolution as deciding that "X corporation," "XYZ123" and "XX corporation" all point to one company and picking a single name. Palantir's Gotham docs define object resolution as "the act of combining two or more Objects," for example records from two source systems "that refer to the same real-world entity, to prevent duplicate objects from being created."
He describes an ontology as typed entities plus relationships that get "the business sense out of the data" in a way a schema does not. Palantir's Foundry Ontology overview calls it "an operational layer for the organization" that goes "far beyond data cataloging or schema design solutions." Foundry splits it into semantic elements (object types, properties, link types) and kinetic elements (action types and functions). The video covers only the semantic half.
He never mentions Palantir. Deloitte announced a strategic alliance with Palantir on July 21, 2025, built around an Enterprise Operating System on Foundry and AIP with "Palantir's ontology pre-configured using Deloitte's industry leading practices." A Deloitte hire learning these two terms in week one is most likely learning a specific vendor stack, and the video presents them as general concepts.
What is progressive disclosure for AI agents?
He calls his third item "programming disclosure," and TikTok's auto-captions show the same words. The idea he describes, agents that "discover tools on demand," is what Anthropic calls progressive disclosure.
Anthropic's November 4, 2025 post on code execution with MCP says "Presenting tools as code on a filesystem allows models to read tool definitions on-demand, rather than reading them all up-front." In its example the agent lists a ./servers/ directory and reads only the tool files it needs. Anthropic reports one workload dropping "from 150,000 tokens to 2,000 tokens, a time and cost saving of 98.7%."
A November 24, 2025 post introduced the Tool Search Tool, which lets Claude search "thousands of tools without consuming its context window." Anthropic reports internal MCP evaluation gains from 49% to 74% for Opus 4 and from 79.5% to 88.1% for Opus 4.5. Those are Anthropic's own benchmarks, and I found no independent replication.
Should you evaluate an agent's trajectory or its final output?
His fourth item is to grade the agent on its entire path to the answer as well as the answer. Google ships this. Its January 24, 2025 announcement of agent evaluation in Vertex AI splits metrics into "final response and trajectory evaluation" and lists six trajectory metrics:
- Exact match requires a trajectory that "perfectly mirrors the ideal solution."
- In-order match needs all required actions in order but allows extra steps.
- Any-order match needs all required actions in any order.
- Precision and recall score the overlap with the reference trajectory.
- Single-tool use checks for one specific action.
Anthropic's January 9, 2026 post on agent evals takes the other side. It says checking "a sequence of tool calls in the right order" is "too rigid and results in overly brittle tests, as agents regularly find valid approaches that eval designers didn't anticipate." Its advice: "it's often better to grade what the agent produced, not the path it took."
Anthropic gives an example where Opus 4.5 solved a flight-booking task by finding a policy loophole. It failed the rigid trajectory check while serving the user better. The video presents trajectory grading as settled practice, and two of the largest vendors in the field disagree about it. Which approach fits likely depends on whether you are debugging a known workflow or measuring open-ended task success.
How does context compression work?
His fifth item is the best-sourced. He makes three points, and each one has a name in Anthropic's September 29, 2025 post on context engineering.
- He says you do not want context "to die as it just grows in length." Anthropic calls this context rot: as tokens increase, "the model's ability to accurately recall information from that context decreases."
- He says to externalize context. Anthropic calls this structured note-taking, where "the agent regularly writes notes persisted to memory outside of the context window."
- He says to summarize what happened. Anthropic calls this compaction: summarizing a conversation near the context limit and starting a new window with the summary.
What the list leaves out
The video never defines an agent. Anthropic's December 19, 2024 post separates agents, where "LLMs dynamically direct their own processes and tool usage," from workflows, which run "through predefined code paths." It recommends "finding the simplest solution possible, and only increasing complexity when needed."
I found no independent confirmation that he works at Deloitte beyond the branded stationery and a #deloitte hashtag.
Bottom line
Context compression, progressive disclosure, entity resolution and ontology all match published definitions, though the last two are Palantir vocabulary presented without the vendor's name. Trajectory grading is a real technique with a real counterargument. Read Anthropic's eval post before building tests that check every step an agent takes.
Sources
- Anthropic, "Demystifying evals for AI agents" - January 9, 2026, the case for grading output over path.
- Google Cloud, agent evaluation in Vertex AI - January 24, 2025, the six trajectory metrics.
- Code execution with MCP, Anthropic engineering - November 4, 2025, on-demand tool definitions and the 150,000 to 2,000 token figure.
- Introducing advanced tool use, Anthropic - November 24, 2025, the Tool Search Tool and its internal benchmarks.
- Effective context engineering for AI agents - September 29, 2025, context rot, compaction and note-taking.
- Building effective AI agents - December 19, 2024, agents versus workflows.
- Palantir Foundry Ontology overview - the ontology as an operational layer.
- Gotham object resolution basics - the entity resolution definition.
- Deloitte and Palantir alliance - July 21, 2025 press release.
- Deloitte University - the training campus named on the notepad.
Published September 21, 2026. Based on a TikTok by @farisallaf. Full video notes (markdown)