<- all tokdocs
2026.09.21@farisallaf

Anthropic Tells Eval Designers to Grade an Agent's Output, "Not the Path It Took." This Deloitte First-Week Recap Says Grade the Whole Path.

AI
Watch on TikTok

View on TikTok ->

Four of the five concepts in this 71-second video map onto published primary sources almost word for word, and the fifth one runs directly against Anthropic's own published guidance on evaluating agents. Faris Allaf posted it on September 21, 2026, framing it as everything he learned in his first week at Deloitte about implementing enterprise-level agents. The frame is a single locked-off selfie shot in a dimly lit hotel room: a young man in a white t-shirt sits at a desk, a bed and dark headboard behind him, a curtain at the right edge. He clicks a pen through the whole take and counts the list off on his fingers. A ruled legal pad lies upside down on the desk in the foreground with handwritten notes on it, and the pad's printed header reads "Deloitte University." The pen carries the same branding. Burned-in word-chunk captions in a bold white slab serif with a black outline run along the bottom. There are no slides, no diagrams, no code, and no screen recording, so the spoken list is the entire payload.

The five terms are Palantir vocabulary, and he never says Palantir

His first two items are entity resolution and ontology. Both are load-bearing terms in Palantir Foundry, and his definitions track Palantir's own. He describes entity resolution as deciding that "X corporation," "XYZ123," and "XX corporation" all point to one thing and picking a single name. Palantir's Gotham API docs define object resolution as "the act of combining two or more Objects. For instance, it may be useful to resolve objects from two different Source Systems that refer to the same real-world entity, to prevent duplicate objects from being created."

His ontology framing is that typed entities plus relationships get you "the business sense out of the data" in a way a plain schema does not. Palantir's Foundry Ontology overview describes the Ontology as "an operational layer for the organization" that sits on top of integrated digital assets, and states that it goes "far beyond data cataloging or schema design solutions." Foundry splits it into semantic elements (object types, properties, link types) and kinetic elements (action types and functions). The video covers the semantic half and skips the kinetic half entirely, which is the part that makes an ontology more than a well-labeled schema.

The context he leaves out matters. Deloitte announced a strategic alliance with Palantir on July 21, 2025, built around an Enterprise Operating System that runs on Foundry and AIP with "Palantir's ontology pre-configured using Deloitte's industry leading practices." The same release says Deloitte has trained the largest cohort of certified Forward Deployed Engineers outside Palantir itself and expects to more than triple that number within a year. A Deloitte hire learning entity resolution and ontology in week one is most likely learning a specific vendor stack. Presenting those two terms as general AI-engineering concepts, with no vendor attached, is the video's biggest framing gap.

"Programming disclosure" is progressive disclosure, and both caption tracks got it wrong

At 0:34 he says what Whisper transcribes as "programming disclosure." TikTok's own burned-in caption at frame 17 reads "third is programming," so the platform's automatic captioner heard the same thing. The concept he then describes, agents that "discover tools on demand," is what Anthropic calls progressive disclosure.

Anthropic's November 4, 2025 engineering post on code execution with MCP states it plainly: "Presenting tools as code on a filesystem allows models to read tool definitions on-demand, rather than reading them all up-front." The worked example has the agent list a ./servers/ directory to find available servers and read only the tool files it needs. Anthropic reports that pattern cutting one workload "from 150,000 tokens to 2,000 tokens, a time and cost saving of 98.7%." A follow-up post on November 24, 2025 ships this as a product feature, the Tool Search Tool, which "allows Claude to use search tools to access thousands of tools without consuming its context window," and reports internal MCP evaluation gains of 49% to 74% for Opus 4 and 79.5% to 88.1% for Opus 4.5. Those are Anthropic's own benchmarks on its own evaluation suite. I found no independent replication of either the token figure or the accuracy deltas.

He also tacks on "they're not stuck with the same exact memory," which is a different technique bolted onto the same bullet. Dynamic tool loading and persistent agent memory solve separate problems, and merging them under one heading is the sloppiest moment in an otherwise tight list.

Trajectory evaluation is shipped by Google and argued against by Anthropic

His fourth item is that you grade the agent on the entire path to the solution as well as the solution. The concept is real and productized. Google announced agent evaluation in the Vertex AI Gen AI evaluation service on January 24, 2025, and states that its metrics "can be grouped in two categories: final response and trajectory evaluation." It ships six trajectory metrics: exact match, which requires a trajectory that "perfectly mirrors the ideal solution"; in-order match, where the trajectory "needs to include all the necessary actions in the correct order, but it might also include extra, unnecessary steps"; any-order match, which "only cares that the agent's trajectory includes all the necessary actions, regardless of their order"; precision; recall; and single-tool use, which "checks for the presence of a specific action within the agent's trajectory."

Anthropic's January 9, 2026 post on evals for AI agents pushes back on exactly this instinct. It says: "There is a common instinct to check that agents followed very specific steps like a sequence of tool calls in the right order. We've found this approach too rigid and results in overly brittle tests, as agents regularly find valid approaches that eval designers didn't anticipate." Its conclusion: "So as not to unnecessarily punish creativity, it's often better to grade what the agent produced, not the path it took." The post gives a concrete failure case where Opus 4.5 solved a flight-booking task by finding a policy loophole, failing the rigid trajectory check while actually serving the user better.

So the video states as settled practice something that two of the largest vendors in the space publicly disagree about. Google sells trajectory scoring. Anthropic warns that it produces brittle tests. Neither position is wrong on its face, and the resolution probably depends on whether you are debugging a known workflow or measuring open-ended task success. The video presents only one side, with no hedge.

Context compression is the best-sourced item in the list

His fifth point breaks into three claims, and all three have a named counterpart in a single Anthropic post from September 29, 2025. He says you do not want the context "to die as it just grows in length." Anthropic calls that context rot: "As the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases." He says you want to externalize it. Anthropic calls that structured note-taking or agentic memory, "a technique where the agent regularly writes notes persisted to memory outside of the context window." He says you want to summarize what has happened. Anthropic calls that compaction, "the practice of taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary."

Three sub-claims, three documented techniques, correct order of operations. This is the part of the video I would hand to someone as a starting point without qualification.

What a first-week list cannot cover

The video never defines what an agent is, which leaves the whole list floating. Anthropic's December 19, 2024 post draws the line: agents are "systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks," while workflows are "systems where LLMs and tools are orchestrated through predefined code paths." The same post recommends "finding the simplest solution possible, and only increasing complexity when needed." Four of the five items here presuppose that an agent is the right architecture. Entity resolution and ontology work are expensive data-engineering projects that pay off whether or not an LLM ever touches the result.

Also absent: any cost figure, any named tool or framework, any failure mode, any security or governance consideration, and any evaluation result from an actual deployment. He flags the limitation himself by calling it week one and asking viewers for content ideas. At 979 views and 52 likes at capture, this reads as someone building an audience while learning in public, and the list is accurate enough that the audience-building is earned.

Key Takeaways

  • Entity resolution and ontology are Palantir Foundry terms, and Deloitte announced a Palantir alliance on July 21, 2025 whose Enterprise Operating System ships Foundry and AIP with a pre-configured Palantir ontology. The video presents both as vendor-neutral concepts.
  • "Programming disclosure" is progressive disclosure. Anthropic documents it as reading tool definitions on demand from a filesystem rather than loading them all up front, and reports one workload dropping from 150,000 to 2,000 tokens.
  • Grading the full trajectory is a live disagreement, not settled practice. Google ships six trajectory metrics in Vertex AI. Anthropic's eval guidance calls step-sequence checking "too rigid" and recommends grading the output instead.
  • The context compression point is the strongest item. Context rot, structured note-taking, and compaction are all named and defined in one Anthropic post from September 2025, and he hits all three.
  • Transcript quality: Whisper tracks the audio closely across all 71 seconds with no truncation and no dropped sentences. The one substantive error is "programming disclosure" at 0:34, and TikTok's own burned-in caption makes the identical error, so this may be what he actually said rather than a Whisper failure. The transcript also renders "DU" without expansion; only the frames resolve it, via the Deloitte University notepad and pen on the desk. Nothing else on screen is lost, because there is nothing else on screen.
  • Unverified: I found no independent confirmation that Faris Allaf works at Deloitte beyond the branded stationery and the #deloitte hashtag. I could not verify that these five concepts reflect Deloitte's actual internal curriculum, or that he encountered them on a Palantir engagement. Anthropic's token-savings and accuracy figures are its own internal benchmarks with no replication I could find. Palantir's Foundry Entity Resolution product page returned no readable body text when I fetched it, so I cite Palantir's Gotham object resolution documentation for the definition instead.

Resources

Published September 21, 2026. Writeup generated from a favorited TikTok.