Karpathy's LLM Knowledge Bases: Why RAG Might Be Reading the Textbook Every Single Time

Andrej Karpathy -- co-founder of OpenAI, former head of AI at Tesla, and one of the most influential AI researchers alive -- has proposed a concept that could fundamentally reshape how retrieval-augmented generation works. He calls them LLM Knowledge Bases (also known as the "LLM Wiki" pattern), and the idea is deceptively simple: stop making the AI reread raw documents every time someone asks a question. Instead, have it build study notes once and answer from those.

The Problem with Traditional RAG

The current RAG pipeline works like this: a user asks a question, the system grabs raw document chunks that are closest to the query via vector similarity, and shoves those messy chunks into the prompt. The LLM then has to interpret the raw material on the fly while simultaneously trying to produce an answer.

Karpathy's analogy is sharp: that is like asking a chef to invent a recipe while someone reads them a cookbook out loud in another language. No wonder it hallucinates.

The LLM Knowledge Base Approach

Think about how a human would actually learn a subject. If you had a hundred research papers on your desk, you would not reread all of them every time someone asked you a question. You would read them once, make clean study notes, organize them, flag where two papers disagree, and cross-reference the concepts. After that, you would answer from your notes, not from the raw papers.

That is exactly what Karpathy proposes for AI. Have the LLM build structured markdown notes -- organized, interlinked, and pre-digested. When someone asks a question, the AI reads the notes, not the raw source material.

Approach	How It Works	Weakness
Traditional RAG	Retrieves raw chunks at query time, LLM interprets on the fly	Hallucination, poor cross-document reasoning, slow
LLM Knowledge Base	LLM pre-processes documents into structured notes, queries read notes	Incremental updates are unsolved at scale

As Karpathy himself put it: "A large fraction of my recent token throughput is going less into manipulating code and more into manipulating knowledge."

Why This Matters

Three reasons this approach is generating excitement:

Accuracy improves dramatically. The hard thinking happens when the notes are written, not during the query. The LLM is not under time pressure to both comprehend and respond.
Cross-document reasoning becomes possible. Questions that span multiple sources work because the cross-references are already baked into the notes.
Contradictions get caught early. Conflicting information between sources is identified during ingestion rather than smuggled into the final answer.

The Catch

This is still experimental. The hard unsolved problem is what happens when your documents change. Updating the notes incrementally is messy, and nobody has cracked that for large enterprise deployments yet.

But for individual builders -- anyone with research notes, a project knowledge base, or domain-specific material they want the AI to deeply understand -- this is immediately actionable. Karpathy published the entire pattern as a GitHub Gist, and the community has already built working open-source implementations. You can clone one, hook it to Claude Code via MCP, upload your documents, and have your own knowledge wiki running in under an hour.

Key Takeaways

Traditional RAG forces the LLM to reinterpret raw documents on every query -- expensive, slow, and error-prone
Karpathy's LLM Knowledge Base pattern pre-processes documents into structured markdown notes that the LLM reads at query time
The approach dramatically improves accuracy, cross-document reasoning, and contradiction detection
Incremental document updates remain an unsolved problem at scale
Working open-source implementations already exist and can be set up in under an hour

Resources

LLM Wiki Gist by Karpathy -- The original pattern description
Beyond RAG: Karpathy's LLM Wiki Pattern -- Deep dive on the architecture
LLM Wiki Skill -- Karpathy-style knowledge base agent skill for Claude Code
Karpathy LLM Wiki Stack -- Build-ready reference implementation using Obsidian and Claude Code