The real cost of a slow agent

People talk about agent latency like it is a polish problem. Ship it now, make it fast later. That framing is wrong. Latency decides what the product is for, and you cannot fix the product later by making it faster.

Speed changes the use case, not just the feel

A response that takes two seconds and a response that takes thirty seconds are not the same feature at two speeds. They are two different features.

At two seconds, people use the system inside their own flow. They ask, they read, they ask again. It becomes a tool they reach for without thinking.

At thirty seconds, people stop using it for anything interactive. They batch their questions. They go make coffee. They route around it. The slow version does not get used less. It gets used differently, for a smaller set of tasks, by fewer people.

So when you defer latency, you are not deferring polish. You are shipping a narrower product and finding out later that the audience you wanted never showed up.

Where the time actually goes

When I profile a slow agent, the time is almost never in one place I expected. It is spread across:

• Extra model calls that exist to check the output of other model calls
• Tool calls run in sequence that could have run at the same time
• A context window that grew large enough to slow every single call
• Retries that fire silently and double the work

Each one looks reasonable on its own. Together they turn a three second task into thirty.

The cost is real money, not just patience

Every one of those extra calls is also a line on the bill. A second model call to validate the first one doubles the cost of that step. A retry doubles it again. At low volume nobody notices. At real volume the slow version and the expensive version are the same version, and you are paying twice for the privilege of being slow.

What to do about it

Treat latency as a number you own from day one, the same way you own correctness. Set a target. Measure the real distribution, not the average, because the average hides the slow tail that users actually feel. Then go after the biggest contributor first.

Usually the biggest win is not a faster model. It is removing a call you did not need, running two calls at once instead of one after the other, or keeping the context small enough that every call stays fast. None of that is glamorous. All of it ships a product people will actually use in their flow, which is the only version that matters.