How to Evaluate Search Without Labels Using Synthetic Queries
By Matt Cullerton, founder of Mavric and Apso
You can start evaluating a search or RAG retrieval system without hand-labeled data by having an LLM write a question that each document answers, then checking whether your retriever returns that document. A TikTok from @hamel_husain describes this recipe, and published research supports using LLM-written queries for retrieval evaluation. The catch is that "without any labels" is not quite right: each generated question and its source document form a synthetic label.
How does the synthetic query method work?
The speaker calls it inverting the dataset:
- Take your corpus of documents.
- For each document, "ask an LLM to create a question" that the document answers, "kind of going in reverse."
- Keep each pair: "Now you have a pairing of question document."
- Run each question through retrieval and check whether the source document comes back.
He frames step 4 as a stress test: "are you able to go the other way?"
The key point is the instruction to the model. As he puts it, "You're not asking a question, you're asking to create a question."
Does research support LLM-generated queries for evaluation?
Yes, in a broader form. Rahmani, Craswell, Yilmaz, Mitra and Campos submitted "Synthetic Test Collections for Retrieval Evaluation" to arXiv on 13 May 2024 (arXiv:2405.07767). They built test collections from both synthetic queries and synthetic relevance judgments, and they report that such collections can be used reliably for retrieval evaluation. They also note a possible bias in these collections toward LLM-based models. The arXiv comments field lists SIGIR 2024.
That paper covers more than the one-direction check in the video. It supports the narrower point that LLM-written queries can serve as a basis for evaluation.
Is it really evaluation without labels?
The description opens "You can start evaluating search without any labels." The method still produces a labeled set. Each generated question sits next to its source document, and that pairing is the relevance label.
The accurate version is that you need no human annotation to get started. The labels come from the LLM and from one assumption: that the source document is the only correct answer.
Where synthetic queries can mislead you
That assumption breaks in some corpora. If several documents overlap or nearly duplicate each other, more than one may answer a generated question equally well. A retriever that returns a different valid document would be scored as a miss. The video does not address this.
The questions can also be easier than real ones. The LLM reads the document before writing the question, so its wording tends to echo the document. Real users often type short, vague or misspelled queries. A retriever that scores well on document-derived questions may still struggle on live traffic.
The speaker calls the recipe "like a playbook for search in general." The clip shows no demonstration, and the arXiv abstract does not show that one recipe fits every search system. Use the pairs as a starting set rather than a final benchmark.
What metric should you use?
The video names none. Two standard choices fit this setup:
- Recall at k: the share of generated questions whose source document appears in the top k results.
- Mean reciprocal rank: how high, on average, the source document ranks.
The video also gives no dataset, tool or result, so it presents the idea without showing it working on a corpus.
Bottom line
Generating one question per document is a quick, cheap way to get a first retrieval benchmark, and published work supports synthetic queries for evaluation. Treat the pairs as synthetic labels with known weaknesses: overlapping documents and questions that echo the source text. Measure recall at k on them, then add real user queries as soon as you have them.
Sources
- Synthetic Test Collections for Retrieval Evaluation, arXiv, submitted 2024-05-13: synthetic queries and judgments, the reliability finding and the bias caveat.
- TikTok post by @hamel_husain: the video that describes the recipe.
Published October 28, 2026. Based on a TikTok by @hamel_husain. Full video notes (markdown)