Turn Your AI Agent Into a Self-Improving Scientific Researcher

What if your AI agent could run experiments on its own capabilities overnight and wake up better than it was yesterday? This skill, adapted from Andrej Karpathy's autoresearch repository, turns your Claude Code or OpenClaw agent into a scientific researcher that systematically experiments with and optimizes its own performance.

The Concept

Karpathy's original autoresearch project was designed to give coding agents a scientific mindset: identify a hypothesis, run an experiment, measure the result, keep or discard, repeat. The skill described here adapts that philosophy from ML training optimization to general agent self-improvement.

After installing the skill, your agent follows a structured loop:

1. Identify KPIs -- The agent determines which key performance indicators matter most for its assigned role
2. Define variables -- It identifies the levers it can adjust during experimentation
3. Run experiments -- As frequently as you want, with as many variations as you want, simultaneously
4. Compare results -- After each experiment, the agent compares the previous version against the experimental version on the target KPIs
5. Compound learning -- The agent remembers the positive and negative effects of each experiment, so knowledge compounds over time

Why This Matters

Karpathy's original results with autoresearch were striking. He ran the system for two days, it conducted 700 experiments, and discovered 20 optimizations that improved ML training speed. Those same 20 tweaks applied to a larger model yielded an 11% speedup. Shopify CEO Tobi Lutke ran 37 experiments overnight and woke up to a 0.8B parameter model outperforming his hand-tuned 1.6B model.

The principle translates beyond ML training. Any agent with measurable outputs can adopt this pattern: identify what "better" means, experiment, measure, iterate.

Key Takeaways

• Self-improvement loops give agents a scientific method for optimizing their own capabilities
• The agent retains memory of both failed and successful experiments, building compounding knowledge
• This pattern works for any agent role with measurable performance indicators
• Based on Andrej Karpathy's autoresearch, which demonstrated significant real-world improvements

Resources

• Karpathy's autoresearch -- The original autonomous ML experimentation framework
• DataCamp Guide to AutoResearch -- Detailed walkthrough of how autoresearch works