---
title: "How to Evaluate Search Without Labels Using Synthetic Queries"
date: "2026-10-28"
video_id: "7701834089715797261"
tiktok_url: "https://www.tiktok.com/@hamel_husain/video/7701834089715797261"
creator: "@hamel_husain"
category: "AI"
tags: ["rag evaluation", "synthetic queries", "retrieval", "ai evals", "llm"]
excerpt: "You can start testing search by having an LLM write a question for each document. Research supports the method, but the pairs are still synthetic labels."
---
You can start evaluating a search or RAG retrieval system without hand-labeled data by having an LLM write a question that each document answers, then checking whether your retriever returns that document. A TikTok from @hamel_husain describes this recipe, and published research supports using LLM-written queries for retrieval evaluation. The catch is that "without any labels" is not quite right: each generated question and its source document form a synthetic label.

## How does the synthetic query method work?

The speaker calls it inverting the dataset:

1. Take your corpus of documents.
2. For each document, "ask an LLM to create a question" that the document answers, "kind of going in reverse."
3. Keep each pair: "Now you have a pairing of question document."
4. Run each question through retrieval and check whether the source document comes back.

He frames step 4 as a stress test: "are you able to go the other way?"

The key point is the instruction to the model. As he puts it, "You're not asking a question, you're asking to create a question."

## Does research support LLM-generated queries for evaluation?

Yes, in a broader form. Rahmani, Craswell, Yilmaz, Mitra and Campos submitted "[Synthetic Test Collections for Retrieval Evaluation](https://arxiv.org/abs/2405.07767)" to arXiv on 13 May 2024 (arXiv:2405.07767). They built test collections from both synthetic queries and synthetic relevance judgments, and they report that such collections can be used reliably for retrieval evaluation. They also note a possible bias in these collections toward LLM-based models. The arXiv comments field lists SIGIR 2024.

That paper covers more than the one-direction check in the video. It supports the narrower point that LLM-written queries can serve as a basis for evaluation.

## Is it really evaluation without labels?

The description opens "You can start evaluating search without any labels." The method still produces a labeled set. Each generated question sits next to its source document, and that pairing is the relevance label.

The accurate version is that you need no human annotation to get started. The labels come from the LLM and from one assumption: that the source document is the only correct answer.

## Where synthetic queries can mislead you

That assumption breaks in some corpora. If several documents overlap or nearly duplicate each other, more than one may answer a generated question equally well. A retriever that returns a different valid document would be scored as a miss. The video does not address this.

The questions can also be easier than real ones. The LLM reads the document before writing the question, so its wording tends to echo the document. Real users often type short, vague or misspelled queries. A retriever that scores well on document-derived questions may still struggle on live traffic.

The speaker calls the recipe "like a playbook for search in general." The clip shows no demonstration, and the arXiv abstract does not show that one recipe fits every search system. Use the pairs as a starting set rather than a final benchmark.

## What metric should you use?

The video names none. Two standard choices fit this setup:

- **Recall at k**: the share of generated questions whose source document appears in the top k results.
- **Mean reciprocal rank**: how high, on average, the source document ranks.

The video also gives no dataset, tool or result, so it presents the idea without showing it working on a corpus.

## Bottom line

Generating one question per document is a quick, cheap way to get a first retrieval benchmark, and published work supports synthetic queries for evaluation. Treat the pairs as synthetic labels with known weaknesses: overlapping documents and questions that echo the source text. Measure recall at k on them, then add real user queries as soon as you have them.

## Sources

- [Synthetic Test Collections for Retrieval Evaluation, arXiv, submitted 2024-05-13](https://arxiv.org/abs/2405.07767): synthetic queries and judgments, the reliability finding and the bias caveat.
- [TikTok post by @hamel_husain](https://www.tiktok.com/@hamel_husain/video/7701834089715797261): the video that describes the recipe.


## Video description and verification notes

**The file is a 24.608 s vertical clip (1080x1920, 9:16) in an MP4/MOV container (isom brand). The video stream is HEVC Main at 30 fps, 738 frames, about 452 kb/s. The audio is HE-AACv2 stereo at 44.1 kHz, about 48 kb/s. The container reports 507,724 b/s overall and 1,561,761 bytes, which matches the metadata filesize and the format id bytevc1_1080p_507704-1. The post timestamp is epoch 1793223005, which is 2026-10-28 21:30:05 UTC and agrees with upload_date 20261028. The handle is @hamel_husain, the nickname is also hamel_husain, and the audio credit is "original sound - hamel_husain". The description is 3 sentences plus the hashtags #aievals #rag #llm. At capture (2026-10-29T12:01:41 UTC) the post showed 281 views, 2 likes, 1 comment, 0 reposts and 2 saves. In this pass I did not re-read any frames. The frame details below come from an earlier draft that reported reading frames 1, 4, 8 and 12 (about 0 s, 6 s, 14 s and 22 s), and I could not confirm them myself. That draft described the same layout in all four: a white upper panel with the monospaced text "Q: How to get started with evaluating search without labels?", a lower panel with a bald, bearded man in glasses and black over-ear headphones, and a burned-in caption box reading "You can take a", "can invert the dataset.", "you're asking it to" and "is it, are you". It reported no terminal, code, chart, URL or product name. The transcript has 8 segments and 81 words, which I read directly. The speaker says to take a corpus of documents, have an LLM write a question each document answers, and then test whether retrieval can "go the other way". The last segment ends at 24.32 s against an audio duration of 24.608 s, so the transcript sits inside the audio and I treat it as genuine.**

## The recipe: "invert the data set" to get question and document pairs

Segment 2 runs 5.46 s to 7.38 s: "And then you can invert the data set." Segment 3 runs 7.54 s to 11.30 s: "For a particular document, you can ask an LLM to create a question." Segment 4 finishes the sentence at 13.66 s with "That document answers, kind of going in reverse." At 15.80 s the speaker adds "You're not asking a question, you're asking to create a question." Segment 6 ends at 19.50 s with "Now you have a pairing of question document."

Published retrieval research uses this idea. Rahmani, Craswell, Yilmaz, Mitra and Campos submitted "Synthetic Test Collections for Retrieval Evaluation" to arXiv on 13 May 2024 (arXiv:2405.07767). I fetched its abstract page. The authors set out to build test collections by generating both synthetic queries and synthetic judgments, and they report that such collections can be used reliably for retrieval evaluation. The page also notes a possible bias in these collections toward LLM-based models. The arXiv comments field lists SIGIR 2024, but the page does not state the acceptance decision. That paper covers a wider setup than the video's single-direction check, so it supports only the narrower point that LLM-written queries can serve as a basis for evaluation.

A web search in the earlier draft surfaced Promptagator (arXiv:2209.11755), which prompts an LLM to write queries for documents. I did not fetch it in this pass, so I do not rely on it.

## "Without any labels": the pairs are synthetic labels

The on-screen question, as the earlier draft reported it, reads "How to get started with evaluating search without labels?" The description opens "You can start evaluating search without any labels." The procedure the video describes still produces a labeled set. Each generated question sits beside its source document, and that pairing works as the relevance label. The accurate claim is that no human annotation is required to start. The labels come from the generating LLM and from the assumption that the source document is the one correct answer.

That assumption has a weakness the video does not raise. In a corpus with near-duplicate or overlapping documents, several documents may answer a generated question equally well. A retriever that returns a different valid document would score as a miss. The video does not say how it handles this, and I found no fetched source that measures the rate, so I treat it as a caution and not a result.

## "A playbook for search in general": broader than the evidence shown

At 2.32 s the transcript says "This is like a playbook for search in general." The clip shows no demonstration, so the claim rests on the speaker's judgment. The arXiv abstract supports synthetic queries for evaluation, but it does not show that one recipe fits every search system. An LLM reads the document before writing the question, so its questions tend to echo the document's wording. Real users often type short, vague or misspelled queries. A retriever that scores well on document-derived questions may still struggle on live traffic. The pairs work better as a starting set than as a final benchmark.

## "Stress test your retrieval system": what the last step shows

From 19.50 s to 24.32 s the transcript says "You can stress test your retrieval system by saying, are you able to go the other way?" The description states the check plainly: run the questions through retrieval and see whether it finds the right document. The earlier draft reported that the caption in the last sampled frame reads "is it, are you", a restart that the Whisper text smooths into one clean sentence. That is a small transcription difference and not a factual dispute.

The video names no metric. A concrete version would be recall at k, the share of generated questions whose source document appears in the top k results, or mean reciprocal rank. The video gives no numbers, dataset, tool or result, so it offers the idea without evidence that it worked on any corpus.

## Provenance and engagement

The clip looks like an excerpt from a longer talk or Q&A, since a question prompt sits on screen. I did not locate an original session and do not claim one. Engagement at capture was 281 views, 2 likes, 1 comment and 2 saves. That is too small a sample to say anything about adoption. The comment text is not in the staged metadata, and I did not read it.

## Key Takeaways

- **Verified:** The transcript matches the description. An LLM writes a question for each corpus document, and the speaker proposes checking whether retrieval returns that document (transcript 7.54 s to 24.32 s).
- **Verified:** Rahmani et al. (arXiv:2405.07767, submitted 2024-05-13) report that LLM-generated queries and judgments can support reliable retrieval evaluation, with a noted possible bias toward LLM-based models.
- **Partial correction:** The video says "without any labels," but the generated question and its source document form a synthetic label. The labels come from the LLM and from the assumption that only the source document is correct.
- **Clarification:** The video names no metric, tool, dataset or result, and the frame observations here come from an earlier draft that I did not re-check.
- **Unverified:** The 2.32 s claim of "a playbook for search in general" has no demonstration in the clip, and I fetched no source that tests it across search systems.
- **Context:** The audio runs 24.608 s and the last segment ends at 24.32 s, so the 81-word transcript is not padded. The post date is 2026-10-28 21:30:05 UTC.

## Resources

- [Synthetic Test Collections for Retrieval Evaluation, arXiv, submitted 2024-05-13](https://arxiv.org/abs/2405.07767) establishes the authors, the generated queries and judgments, the reliability finding, the bias caveat and the SIGIR 2024 comments entry I cite.
- [Video page on TikTok by @hamel_husain, posted 2026-10-28](https://www.tiktok.com/@hamel_husain/video/7701834089715797261) is the source video. I read its staged metadata, transcript and ffprobe output, not the live page.
