Semantic Memory: The Agent Design Concept That Separates Knowledge From Action

Reliable agents keep factual knowledge in memory and keep actions in tools, and semantic memory is the layer that holds the facts. In this whiteboard breakdown, Doby Lanete argues that most people building agents right now have never heard of semantic memory, then defines it plainly: it stores factual, structured knowledge such as an SOP, a process document, or a set of rules that stays consistently true about how a business operates. He contrasts that stable knowledge against tools, which are the actions an agent takes, like adding a row in Airtable. The point is a clean separation that makes an agent behave predictably.

What Semantic Memory Actually Stores

Semantic memory holds knowledge that is consistently true, not a log of what happened. In the transcript, Doby describes it as "factual, structured knowledge" and gives concrete examples: a CRM or a database, both of which hold structured, factual records. He also points to a Standard Operating Procedure that is "100% truth and factual knowledge." The distinction is that this content does not change based on any single conversation. It is the ground truth an agent leans on when it makes decisions.

This lines up with how the broader agent field defines the term. Across LLM agent research, semantic memory is the agent's stable world knowledge, usually kept in external knowledge bases of documents, rules, and references, and separate from episodic memory, which records task-specific experiences with timestamps and context.

Memory Versus Tools: A Distinction Most Builders Miss

The video's sharpest moment is a correction Doby makes to himself. He starts to call Airtable a tool, then clarifies: "Airtable technically is a tool, but what we call the tool for Airtable is how we interact with it. A tool would be add a row in Airtable." Airtable-the-database can serve as external memory, while "add a row" is the action. The two are not the same thing.

That framing gives a simple rule for anyone wiring up an agent. Knowledge that informs why and when to act belongs in memory. The concrete action itself belongs in a tool. Keeping them separate is, in his words, "what makes an agent actually reliable."

The Whiteboard Map: In-Context, External, and Episodic

The diagram on the whiteboard sketches how these memory types sit together. A box labeled "Brain" represents external memory and holds the model's working "Context." Feeding into it is an "in-context memory" block and an "episodic memory" block, with an arrow running out to an "Airtable" table. As Doby explains that a business SOP can live as semantic memory, he writes a small "SOP" box inside the Brain, showing the SOP stored as external memory the agent can read.

The takeaway from the drawing is that in-context memory is the live window the agent is working in, while external memory is the persistent store it can reach into on demand. Semantic knowledge like an SOP lives in that external store so the agent can pull it as ground truth whenever a decision calls for it.

How Claude Code Fits In

Doby ties the concept to a specific tool: "Claude code can access that SOP through the external memory." An SOP written once and kept factual becomes a resource the agent reads back when it needs the rules of the business, rather than something re-explained in every prompt.

Anthropic's own memory tool works on exactly this pattern. It lets a model store and retrieve information across conversations in a directory of memory files, reading them back on demand so the active context stays focused on the current task. When someone asks whether this is the same as sub-agent logic, Doby answers no. It can overlap, but the core meaning of semantic memory stands on its own: it stores factual, structured knowledge, full stop.

Key Takeaways

Semantic memory stores stable, factual, structured knowledge such as SOPs, process documents, rules, CRMs, and databases.
It is distinct from episodic memory, which records what happened and when, and from the live in-context window.
A tool is an action the agent takes, like "add a row in Airtable," not the knowledge that informs the action.
Airtable can be external memory and also be acted on by a tool; the database and the action are separate concerns.
Store knowledge in memory and store actions in tools; that separation is what makes an agent reliable.
An SOP kept as external memory becomes ground truth an agent like Claude Code can read back when making decisions.