The Hiring Question That Reveals Real AI Fluency: "What Are You Running Overnight?"

The best test of an AI operator is not the tools they open during the day, it is the loop that keeps working after they close the laptop. In this clip, Eric Siu shares a question a founder friend now asks in interviews: "What are you running overnight?" The point is that showing off Codex, Claude Code, or open coding agents is table stakes. The real signal is whether you have a repeatable overnight eval loop that tests cheaper models and prompt variants against real data while you sleep. The on-screen document walks through a concrete example built at a company called Tenex, complete with model costs, a golden dataset, and a launchd schedule.

The Question Behind the Question

Siu opens by saying a friend who runs a company kept hearing "What are you running overnight?" during his first weeks there. The framing on screen is explicit: not "what are you working on," but "what are you running overnight." At first it sounds absurd. Then someone reframes it: if you are not running something overnight, you are leaving money on the table. Once you have paid for a plan like Codex Pro, the marginal cost of an overnight sweep is close to zero, so idle compute is wasted budget. The question doubles as a fluency test. Anyone can list tools. Fewer people can describe a process that improves their system without them in the room.

The Real Cost Problem: Useful Features That Bleed Money Monthly

The document on screen grounds the idea in a specific build. Coming from a VC-backed "do things that don't scale" mindset, the author realized that niche software carries a price tag every single month long after the build is done. A feature can be genuinely useful and still be a bad deal. The example: an email extractor and classifier that triages incoming business opportunities. It started on Sonnet with great accuracy but cost roughly $1,200 per month at production volume. Dropping to Haiku cut that to about a third, roughly $400, but Haiku alone did not hold the accuracy. That gap between "works" and "works at a price the client will tolerate" is what the overnight loop exists to close.

Building the Golden Dataset First

The transcript stresses capturing 100 to 200 real examples, and the on-screen text explains why this step matters most. A strong model drafted around 150 examples from production data. The author then reviewed every one by hand, corrected what was wrong, and loaded the clean set into Langfuse. The document calls this step unglamorous and incredibly annoying, but by far the most important. The dataset includes inputs, expected outputs, edge cases, failures, and human-corrected labels. Without it, an overnight sweep is just noise. With it, every cheaper model and prompt variant gets scored against the same trusted answers.

The Overnight Sweep and What Gets Tracked

With the dataset clean, the author kicked off the run and went to bed. The document shows Codex using the Langfuse MCP server to run experiments across gpt-oss-120b, GPT-5 mini, Kimi K2, and Qwen3-32B, with one rule: get within 95% of Sonnet's baseline. The six-step pattern is laid out on screen. Build with the best model first to establish baseline quality. Capture 100 to 200 real examples as the golden dataset. Run overnight sweeps of cheaper models, prompt variants, routing logic, and guardrails. Score on business outcomes, not just accuracy: correctness, cost per run and monthly cost, latency, failure modes, human review required, and regression risk. Promote a model only if it clears the bar, with an example rule of "must hit 95% of the expensive baseline, with no critical misses, and reduce cost by 50%+." Ship the proof as a short report covering the winning config, cost delta, failed cases, and a recommendation on whether it is safe to switch.

Making the Loop Run Without You

The final frames show the loop wired into a schedule so it runs unattended. The author asked the agent to report back at 8am the next morning on whether the eval succeeded. The response confirms a one-time report-back job writing to latest-report-back.md, plus a local macOS notification flagging SUCCESS or FAILED_OR_INCOMPLETE. The dry run through launchd exited cleanly with exit code 0. The standing schedule is visible: 11:00 PM daily for the eval run, 7:30 AM daily for the morning summary, and a one-time 8:00 AM success or failure report-back. This is the "constantly having this thing work even when you're not around" idea made concrete. Siu closes by saying he got two things from the clip: a new interview question, and an overnight eval loop of his own to build.

Key Takeaways

"What are you running overnight?" is a sharper AI-fluency test than "what tools do you use," because it measures process, not tool names.
A feature can be genuinely useful and still a bad deal. Track monthly cost per model, not just whether it works.
The example moved a classifier from Sonnet at about $1,200/month toward Haiku or a cheaper model at roughly $400/month, but only after measuring accuracy on a golden dataset.
Build the dataset first: 100 to 200 real examples, human-corrected, loaded into an eval tool like Langfuse. It is the least glamorous and most important step.
Score on business outcomes: correctness, cost per run, monthly cost, latency, failure modes, human review required, and regression risk.
Promote a cheaper model only when it clears an explicit bar, such as 95% of the expensive baseline with no critical misses and a 50%+ cost cut.
Schedule the loop (for example via launchd) so it runs overnight and reports back automatically, then ship a short proof report each run.