An AI Agent That Tunes Your RAG Pipeline While You Sleep

Most RAG pipelines get tuned once by hand and then left alone, even though chunk size, embedding model, and keyword filtering all interact in ways nobody predicts up front. Autoretrieval, an open source project by daly2211 spotted on Hacker News, flips that: you hand an AI agent your retrieval pipeline and an eval set, and it runs experiments on its own overnight, keeping only the changes that measurably improve accuracy. The video walks through the GitHub README, which shows a progress chart of 83 experiments producing 13 kept improvements.

The Core Loop: Edit, Evaluate, Keep or Discard

The agent works on a single file, experiment.py, which contains the chunker, embedding model, keyword filtering, and retrieval logic. Everything in that file is fair game. After each edit, a fixed scoring engine (run_eval.py) computes character-level overlap between retrieved chunks and ground-truth reference highlights. If the F-beta score improved, the change stays. If not, it gets discarded and the agent tries something else. The README's progress chart makes this visible: green dots for kept experiments, gray for discarded, with the running best score climbing from roughly 0.20 to above 0.45 over the run.

Four Files, Deliberately Small

The repo is intentionally minimal. Four files matter:

• experiment.py -- the pipeline the agent edits and iterates on
• run_eval.py -- the fixed scoring engine the agent cannot touch
• program.md -- baseline instructions for the agent, edited by the human
• generate_dataset.py -- generates question and reference-highlight pairs from any corpus

That separation is the key design decision. The agent owns the pipeline, the human owns the instructions, and the eval is frozen so the agent cannot game its own scorecard.

Evals Built From Your Own Documents

Generic retrieval benchmarks tell you little about your own data. Autoretrieval sidesteps that by pointing generate_dataset.py at your corpus and using an LLM to produce question and reference-highlight pairs. The result is a domain-specific eval set, which means the optimization target reflects your actual content rather than a public benchmark.

Recall First, But Tunable

The default optimization target is F-beta with beta set to 2.0, which favors recall over precision. The reasoning: in RAG, missing relevant text usually hurts more than including extra text. You can change F_BETA in run_eval.py to favor precision (beta below 1) or use balanced F1 (beta of 1.0). All metrics use character-level overlap between retrieved chunks and ground-truth highlights, so scores stay comparable even as the agent changes its chunking strategy.

Getting It Running

Requirements are Python 3.10+ and an OpenRouter API key. The quick start shown in the video is four steps: clone the repo and install from requirements.txt, copy .env.example and add your OPENROUTER_API_KEY, run a single evaluation with python run_eval.py (or --pct 10 for a quick smoke test), then optionally generate a dataset from your own corpus. The project uses ChromaDB as the vector store and builds on techniques from Karpathy's autoresearch, Chroma's chunking evaluation, and Andrew Lucek's custom RAG evals. MIT licensed.

Key Takeaways

• Autoretrieval lets an AI agent tune a RAG pipeline autonomously: edit experiment.py, run the eval, keep improvements, discard the rest.
• The eval is generated from your own documents, so the agent optimizes for your domain rather than a public benchmark.
• The scoring engine is fixed and separate from the file the agent edits, which prevents the agent from gaming its own metric.
• Default target is F-beta at 2.0 (recall-weighted), adjustable toward precision or balanced F1 in run_eval.py.
• Setup is light: Python 3.10+, an OpenRouter API key, ChromaDB, and four files.

Resources

• daly2211/autoretrieval -- MIT-licensed open source agent that autonomously optimizes RAG retrieval pipelines against evals built from your own documents