A Self-Improving Data Science Agent Running on a Local 14B Model

Jake Beau built a long-running agent that solves data science competition questions and improves its own predictive models over time. The unusual part is the brain: instead of a frontier cloud model, the agent runs on Phi 14b, a small model that fits on his own machine, so every iteration costs nothing.

The video walks through the loop the agent runs, shows a live dashboard of dozens of experiments it has executed on its own, and points to two engineering write-ups (one from Anthropic, one from OpenAI) that shaped how he kept the agent on track instead of drifting off the rails.

The Data Scientist Loop, Automated

The agent mimics what a data scientist actually does. As Jake puts it in the video: build a predictive model, run an experiment to test its performance, then improve the model based on the findings. He targets data science competition questions in the style of Kaggle, where a scored leaderboard gives the agent an objective signal to optimize against.

The video illustrates the cycle with a weather forecasting example. A model predicts 72 degrees, the actual reading comes in at 58, and the mismatch gets graded. That grade is what the agent feeds back into its next attempt.

Watching It Run Experiments

The dashboard shows each experiment as a dot in a branching tree, with the current best score (0.9121 in the frame below) pinned at the top alongside run counts and elapsed time in the range of 15 hours. After each run, the agent reads the results and decides what to try next, so the tree grows in the directions that look promising.

Jake is upfront that the scores are decent rather than record-setting. The point of the project was different: prove that an agent on a local model can keep improving across many sessions instead of degrading or going off the rails, which is the usual failure mode for long-running agents.

Small Local Brain, Zero Marginal Cost

The agent runs a plan, act, observe, reflect loop with Phi 14b at the center. The overlay in the video labels it "LOCAL 14B, $0, on-device" and shows a quality bar climbing toward a cloud-model benchmark line as iterations accumulate. The tradeoff is explicit: a small model starts weaker per step, but free iterations mean the harness can grind through many experiments and close the gap through accumulated learning.

He credits two write-ups for the harness design: Anthropic's engineering post on effective harnesses for long-running agents and OpenAI's post on harness engineering with Codex. Both appear on screen near the end of the video, and both deal with the same core problem he hit, which is keeping an agent productive across sessions that each start with no memory of the last one.

Key Takeaways

• A long-running agent needs an objective feedback signal. Competition-style scoring gives the agent something concrete to improve against each cycle.
• The loop is the product: build a model, run an experiment, read the results, decide the next attempt. Each experiment is a node in a growing search tree.
• A small local model (Phi 14b) trades per-step capability for zero-cost iteration, which suits workloads where many cheap experiments beat a few expensive ones.
• The hard problem is not raw intelligence but harness design, meaning the scaffolding that keeps the agent improving instead of drifting over long horizons.

Resources

• Effective harnesses for long-running agents (Anthropic Engineering)
• Harness engineering: leveraging Codex in an agent-first world (OpenAI, Ryan Lopopolo)
• Kaggle, the competition platform whose question format the agent targets