---
title: "AI Agent Evaluation: Grade the Output or the Trajectory?"
date: "2026-09-21"
video_id: "7687840916681411854"
tiktok_url: "https://www.tiktok.com/@farisallaf/video/7687840916681411854"
creator: "@farisallaf"
category: "AI"
tags: ["ai agents", "agent evaluation", "context engineering", "progressive disclosure", "palantir"]
excerpt: "A Deloitte new hire lists five agent concepts. Four match published sources. The fifth, grading an agent's full path, is something Anthropic advises against."
---
A TikTok from @farisallaf lists five things he says he learned in his first week at Deloitte about building enterprise AI agents: entity resolution, ontology, progressive disclosure, trajectory evaluation and context compression. Four of them match published sources closely. The fourth, grading an agent on the whole path it took, is something Google ships as a product and Anthropic's own eval guidance argues against.

The video is a single 71-second talking-head take with no slides or code, so the spoken list is all there is. A legal pad and pen on the desk carry "Deloitte University" branding.

## What are entity resolution and ontology?

These are his first two items, and both are core terms in Palantir Foundry. He describes entity resolution as deciding that "X corporation," "XYZ123" and "XX corporation" all point to one company and picking a single name. Palantir's Gotham docs define object resolution as "the act of combining two or more Objects," for example records from two source systems "that refer to the same real-world entity, to prevent duplicate objects from being created."

He describes an ontology as typed entities plus relationships that get "the business sense out of the data" in a way a schema does not. Palantir's Foundry Ontology overview calls it "an operational layer for the organization" that goes "far beyond data cataloging or schema design solutions." Foundry splits it into semantic elements (object types, properties, link types) and kinetic elements (action types and functions). The video covers only the semantic half.

He never mentions Palantir. Deloitte announced a strategic alliance with Palantir on July 21, 2025, built around an Enterprise Operating System on Foundry and AIP with "Palantir's ontology pre-configured using Deloitte's industry leading practices." A Deloitte hire learning these two terms in week one is most likely learning a specific vendor stack, and the video presents them as general concepts.

## What is progressive disclosure for AI agents?

He calls his third item "programming disclosure," and TikTok's auto-captions show the same words. The idea he describes, agents that "discover tools on demand," is what Anthropic calls progressive disclosure.

Anthropic's November 4, 2025 post on code execution with MCP says "Presenting tools as code on a filesystem allows models to read tool definitions on-demand, rather than reading them all up-front." In its example the agent lists a `./servers/` directory and reads only the tool files it needs. Anthropic reports one workload dropping "from 150,000 tokens to 2,000 tokens, a time and cost saving of 98.7%."

A November 24, 2025 post introduced the Tool Search Tool, which lets Claude search "thousands of tools without consuming its context window." Anthropic reports internal MCP evaluation gains from 49% to 74% for Opus 4 and from 79.5% to 88.1% for Opus 4.5. Those are Anthropic's own benchmarks, and I found no independent replication.

## Should you evaluate an agent's trajectory or its final output?

His fourth item is to grade the agent on its entire path to the answer as well as the answer. Google ships this. Its January 24, 2025 announcement of agent evaluation in Vertex AI splits metrics into "final response and trajectory evaluation" and lists six trajectory metrics:

- **Exact match** requires a trajectory that "perfectly mirrors the ideal solution."
- **In-order match** needs all required actions in order but allows extra steps.
- **Any-order match** needs all required actions in any order.
- **Precision and recall** score the overlap with the reference trajectory.
- **Single-tool use** checks for one specific action.

Anthropic's January 9, 2026 post on agent evals takes the other side. It says checking "a sequence of tool calls in the right order" is "too rigid and results in overly brittle tests, as agents regularly find valid approaches that eval designers didn't anticipate." Its advice: "it's often better to grade what the agent produced, not the path it took."

Anthropic gives an example where Opus 4.5 solved a flight-booking task by finding a policy loophole. It failed the rigid trajectory check while serving the user better. The video presents trajectory grading as settled practice, and two of the largest vendors in the field disagree about it. Which approach fits likely depends on whether you are debugging a known workflow or measuring open-ended task success.

## How does context compression work?

His fifth item is the best-sourced. He makes three points, and each one has a name in Anthropic's September 29, 2025 post on context engineering.

- He says you do not want context "to die as it just grows in length." Anthropic calls this **context rot**: as tokens increase, "the model's ability to accurately recall information from that context decreases."
- He says to externalize context. Anthropic calls this **structured note-taking**, where "the agent regularly writes notes persisted to memory outside of the context window."
- He says to summarize what happened. Anthropic calls this **compaction**: summarizing a conversation near the context limit and starting a new window with the summary.

## What the list leaves out

The video never defines an agent. Anthropic's December 19, 2024 post separates agents, where "LLMs dynamically direct their own processes and tool usage," from workflows, which run "through predefined code paths." It recommends "finding the simplest solution possible, and only increasing complexity when needed."

I found no independent confirmation that he works at Deloitte beyond the branded stationery and a #deloitte hashtag.

## Bottom line

Context compression, progressive disclosure, entity resolution and ontology all match published definitions, though the last two are Palantir vocabulary presented without the vendor's name. Trajectory grading is a real technique with a real counterargument. Read Anthropic's eval post before building tests that check every step an agent takes.

## Sources

- [Anthropic, "Demystifying evals for AI agents"](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) - January 9, 2026, the case for grading output over path.
- [Google Cloud, agent evaluation in Vertex AI](https://cloud.google.com/blog/products/ai-machine-learning/introducing-agent-evaluation-in-vertex-ai-gen-ai-evaluation-service) - January 24, 2025, the six trajectory metrics.
- [Code execution with MCP, Anthropic engineering](https://www.anthropic.com/engineering/code-execution-with-mcp) - November 4, 2025, on-demand tool definitions and the 150,000 to 2,000 token figure.
- [Introducing advanced tool use, Anthropic](https://www.anthropic.com/engineering/advanced-tool-use) - November 24, 2025, the Tool Search Tool and its internal benchmarks.
- [Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) - September 29, 2025, context rot, compaction and note-taking.
- [Building effective AI agents](https://www.anthropic.com/engineering/building-effective-agents) - December 19, 2024, agents versus workflows.
- [Palantir Foundry Ontology overview](https://www.palantir.com/docs/foundry/ontology/overview) - the ontology as an operational layer.
- [Gotham object resolution basics](https://www.palantir.com/docs/gotham/api/revdb-resources/resolution/resolution-basics) - the entity resolution definition.
- [Deloitte and Palantir alliance](https://www.deloitte.com/us/en/about/press-room/deloitte-palantir-strategic-alliance.html) - July 21, 2025 press release.
- [Deloitte University](https://www.deloitte.com/us/en/about/deloitte-university-leadership-center.html) - the training campus named on the notepad.


## Video description and verification notes

**Four of the five concepts in this 71-second video map onto published primary sources almost word for word, and the fifth one runs directly against Anthropic's own published guidance on evaluating agents.** Faris Allaf posted it on September 21, 2026, framing it as everything he learned in his first week at Deloitte about implementing enterprise-level agents. The frame is a single locked-off selfie shot in a dimly lit hotel room: a young man in a white t-shirt sits at a desk, a bed and dark headboard behind him, a curtain at the right edge. He clicks a pen through the whole take and counts the list off on his fingers. A ruled legal pad lies upside down on the desk in the foreground with handwritten notes on it, and the pad's printed header reads "Deloitte University." The pen carries the same branding. Burned-in word-chunk captions in a bold white slab serif with a black outline run along the bottom. There are no slides, no diagrams, no code, and no screen recording, so the spoken list is the entire payload.

## The five terms are Palantir vocabulary, and he never says Palantir

His first two items are entity resolution and ontology. Both are load-bearing terms in Palantir Foundry, and his definitions track Palantir's own. He describes entity resolution as deciding that "X corporation," "XYZ123," and "XX corporation" all point to one thing and picking a single name. Palantir's Gotham API docs define object resolution as "the act of combining two or more Objects. For instance, it may be useful to resolve objects from two different Source Systems that refer to the same real-world entity, to prevent duplicate objects from being created."

His ontology framing is that typed entities plus relationships get you "the business sense out of the data" in a way a plain schema does not. Palantir's Foundry Ontology overview describes the Ontology as "an operational layer for the organization" that sits on top of integrated digital assets, and states that it goes "far beyond data cataloging or schema design solutions." Foundry splits it into semantic elements (object types, properties, link types) and kinetic elements (action types and functions). The video covers the semantic half and skips the kinetic half entirely, which is the part that makes an ontology more than a well-labeled schema.

The context he leaves out matters. Deloitte announced a strategic alliance with Palantir on July 21, 2025, built around an Enterprise Operating System that runs on Foundry and AIP with "Palantir's ontology pre-configured using Deloitte's industry leading practices." The same release says Deloitte has trained the largest cohort of certified Forward Deployed Engineers outside Palantir itself and expects to more than triple that number within a year. A Deloitte hire learning entity resolution and ontology in week one is most likely learning a specific vendor stack. Presenting those two terms as general AI-engineering concepts, with no vendor attached, is the video's biggest framing gap.

## "Programming disclosure" is progressive disclosure, and both caption tracks got it wrong

At 0:34 he says what Whisper transcribes as "programming disclosure." TikTok's own burned-in caption at frame 17 reads "third is programming," so the platform's automatic captioner heard the same thing. The concept he then describes, agents that "discover tools on demand," is what Anthropic calls progressive disclosure.

Anthropic's November 4, 2025 engineering post on code execution with MCP states it plainly: "Presenting tools as code on a filesystem allows models to read tool definitions on-demand, rather than reading them all up-front." The worked example has the agent list a `./servers/` directory to find available servers and read only the tool files it needs. Anthropic reports that pattern cutting one workload "from 150,000 tokens to 2,000 tokens, a time and cost saving of 98.7%." A follow-up post on November 24, 2025 ships this as a product feature, the Tool Search Tool, which "allows Claude to use search tools to access thousands of tools without consuming its context window," and reports internal MCP evaluation gains of 49% to 74% for Opus 4 and 79.5% to 88.1% for Opus 4.5. Those are Anthropic's own benchmarks on its own evaluation suite. I found no independent replication of either the token figure or the accuracy deltas.

He also tacks on "they're not stuck with the same exact memory," which is a different technique bolted onto the same bullet. Dynamic tool loading and persistent agent memory solve separate problems, and merging them under one heading is the sloppiest moment in an otherwise tight list.

## Trajectory evaluation is shipped by Google and argued against by Anthropic

His fourth item is that you grade the agent on the entire path to the solution as well as the solution. The concept is real and productized. Google announced agent evaluation in the Vertex AI Gen AI evaluation service on January 24, 2025, and states that its metrics "can be grouped in two categories: final response and trajectory evaluation." It ships six trajectory metrics: exact match, which requires a trajectory that "perfectly mirrors the ideal solution"; in-order match, where the trajectory "needs to include all the necessary actions in the correct order, but it might also include extra, unnecessary steps"; any-order match, which "only cares that the agent's trajectory includes all the necessary actions, regardless of their order"; precision; recall; and single-tool use, which "checks for the presence of a specific action within the agent's trajectory."

Anthropic's January 9, 2026 post on evals for AI agents pushes back on exactly this instinct. It says: "There is a common instinct to check that agents followed very specific steps like a sequence of tool calls in the right order. We've found this approach too rigid and results in overly brittle tests, as agents regularly find valid approaches that eval designers didn't anticipate." Its conclusion: "So as not to unnecessarily punish creativity, it's often better to grade what the agent produced, not the path it took." The post gives a concrete failure case where Opus 4.5 solved a flight-booking task by finding a policy loophole, failing the rigid trajectory check while actually serving the user better.

So the video states as settled practice something that two of the largest vendors in the space publicly disagree about. Google sells trajectory scoring. Anthropic warns that it produces brittle tests. Neither position is wrong on its face, and the resolution probably depends on whether you are debugging a known workflow or measuring open-ended task success. The video presents only one side, with no hedge.

## Context compression is the best-sourced item in the list

His fifth point breaks into three claims, and all three have a named counterpart in a single Anthropic post from September 29, 2025. He says you do not want the context "to die as it just grows in length." Anthropic calls that context rot: "As the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases." He says you want to externalize it. Anthropic calls that structured note-taking or agentic memory, "a technique where the agent regularly writes notes persisted to memory outside of the context window." He says you want to summarize what has happened. Anthropic calls that compaction, "the practice of taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary."

Three sub-claims, three documented techniques, correct order of operations. This is the part of the video I would hand to someone as a starting point without qualification.

## What a first-week list cannot cover

The video never defines what an agent is, which leaves the whole list floating. Anthropic's December 19, 2024 post draws the line: agents are "systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks," while workflows are "systems where LLMs and tools are orchestrated through predefined code paths." The same post recommends "finding the simplest solution possible, and only increasing complexity when needed." Four of the five items here presuppose that an agent is the right architecture. Entity resolution and ontology work are expensive data-engineering projects that pay off whether or not an LLM ever touches the result.

Also absent: any cost figure, any named tool or framework, any failure mode, any security or governance consideration, and any evaluation result from an actual deployment. He flags the limitation himself by calling it week one and asking viewers for content ideas. At 979 views and 52 likes at capture, this reads as someone building an audience while learning in public, and the list is accurate enough that the audience-building is earned.

## Key Takeaways

- Entity resolution and ontology are Palantir Foundry terms, and Deloitte announced a Palantir alliance on July 21, 2025 whose Enterprise Operating System ships Foundry and AIP with a pre-configured Palantir ontology. The video presents both as vendor-neutral concepts.
- "Programming disclosure" is progressive disclosure. Anthropic documents it as reading tool definitions on demand from a filesystem rather than loading them all up front, and reports one workload dropping from 150,000 to 2,000 tokens.
- Grading the full trajectory is a live disagreement, not settled practice. Google ships six trajectory metrics in Vertex AI. Anthropic's eval guidance calls step-sequence checking "too rigid" and recommends grading the output instead.
- The context compression point is the strongest item. Context rot, structured note-taking, and compaction are all named and defined in one Anthropic post from September 2025, and he hits all three.
- Transcript quality: Whisper tracks the audio closely across all 71 seconds with no truncation and no dropped sentences. The one substantive error is "programming disclosure" at 0:34, and TikTok's own burned-in caption makes the identical error, so this may be what he actually said rather than a Whisper failure. The transcript also renders "DU" without expansion; only the frames resolve it, via the Deloitte University notepad and pen on the desk. Nothing else on screen is lost, because there is nothing else on screen.
- Unverified: I found no independent confirmation that Faris Allaf works at Deloitte beyond the branded stationery and the #deloitte hashtag. I could not verify that these five concepts reflect Deloitte's actual internal curriculum, or that he encountered them on a Palantir engagement. Anthropic's token-savings and accuracy figures are its own internal benchmarks with no replication I could find. Palantir's Foundry Entity Resolution product page returned no readable body text when I fetched it, so I cite Palantir's Gotham object resolution documentation for the definition instead.

## Resources

- [Source video: "my hair is messy icl but I hope you learned something"](https://www.tiktok.com/@farisallaf/video/7687840916681411854) - the 71-second clip, 979 views and 52 likes at capture
- [Anthropic, "Demystifying evals for AI agents"](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) - January 9, 2026; the source of "grade what the agent produced, not the path it took" and the argument that step-sequence checks are too rigid
- [Google Cloud, "Introducing agent evaluation in Vertex AI Gen AI evaluation service"](https://cloud.google.com/blog/products/ai-machine-learning/introducing-agent-evaluation-in-vertex-ai-gen-ai-evaluation-service) - January 24, 2025; defines the six trajectory metrics and splits evaluation into final response and trajectory
- [Anthropic, "Code execution with MCP"](https://www.anthropic.com/engineering/code-execution-with-mcp) - November 4, 2025; progressive disclosure of tool definitions from a filesystem, with the 150,000 to 2,000 token figure
- [Anthropic, "Introducing advanced tool use on the Claude Developer Platform"](https://www.anthropic.com/engineering/advanced-tool-use) - November 24, 2025; the Tool Search Tool and its internal MCP evaluation deltas
- [Anthropic, "Effective context engineering for AI agents"](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) - September 29, 2025; defines context rot, compaction, and structured note-taking
- [Anthropic, "Building effective AI agents"](https://www.anthropic.com/engineering/building-effective-agents) - December 19, 2024; the agent versus workflow definition the video never states
- [Palantir, Foundry Ontology overview](https://www.palantir.com/docs/foundry/ontology/overview) - defines the Ontology as an operational layer going "far beyond data cataloging or schema design," with semantic and kinetic elements
- [Palantir, Gotham object resolution basics](https://www.palantir.com/docs/gotham/api/revdb-resources/resolution/resolution-basics) - defines resolving objects from different source systems that refer to the same real-world entity
- [Deloitte, "Deloitte and Palantir Strategic Alliance" press release](https://www.deloitte.com/us/en/about/press-room/deloitte-palantir-strategic-alliance.html) - July 21, 2025; the Enterprise Operating System, the pre-configured Palantir ontology, and the Forward Deployed Engineer cohort
- [Deloitte University: The Leadership Center](https://www.deloitte.com/us/en/about/deloitte-university-leadership-center.html) - confirms DU as the Westlake, Texas training campus opened in 2011, matching the branded notepad and pen on the desk
