Most stories about a huge surprise LLM bill have the same shape. An agent or retry loop runs overnight, nobody notices, and the provider’s spend limit either doesn’t exist at the level you need or kicks in too late. Here’s why that happens, and a pattern that stops it. Why the usual safeguards miss it
- Provider limits are coarse. They’re usually per organisation or per project and per month. A loop can burn a month’s budget in an hour and still be “within limits”.
- Usage dashboards lag. Cost data often shows up minutes to hours late, so an alert based on it fires after the damage is done.
- max_tokens caps one call, not a loop. 10,000 calls at 4,000 tokens each is still 40M tokens.
- Code review doesn’t catch the runtime causes. The real culprits tend to be a leaked key, a context window that grows every turn, retries piling up during a provider outage, or a tool-call loop that only certain inputs trigger. The code looks fine; the traffic isn’t. The pattern: check the budget before every call (reserve → call → settle) Check the budget before each request goes out, not after. Because you don’t know the real cost yet, reserve the worst case first, then correct it once the bill is known: def guarded_call(budget, req):
Worst case: actual input tokens + the max output you allow
est = price(req.model, count_tokens(req.messages), req.max_tokens)
Atomic check-and-add: only succeeds if spent + reserved + est <= limit
if not budget.reserve(est): raise BudgetExceeded(f"would exceed {budget.limit}") try: resp = client.chat(req) except Exception: budget.release(est) # nothing was billed raise budget.settle(est, price_from_usage(resp.usage)) # swap estimate for actual return resp Details that matter : The reservation must be atomic. With 50 concurrent workers, “read spent → compare → call” lets all 50 pass the check before any of them records spend. Use something that checks and adds in one step:
- Redis: a Lua script, or INCRBY followed by a check and rollback.
- SQL: UPDATE budgets SET reserved = reserved + :est WHERE spent + reserved + :est <= limit and check the affected row count. Reserve the worst case, not the average. Estimating output at a typical length lets a run of long replies break the ceiling. Reserve max_tokens and give back the difference afterwards. Streams that get cut off still cost money. If the client disconnects halfway through a stream, you’ll often never receive the final token count. Charge the estimate rather than zero, or your ledger will under-count exactly during incidents. Retries spend budget too. Each retry goes through guarded_call again. Backoff with jitter plus a budget check means an outage burns at most what’s left in the budget, not your whole card. Use more than one limit. A daily cap per key catches leaked keys. A per-run or per-agent cap catches loops. A monthly team cap catches gradual drift. One global monthly cap catches none of these quickly. Watch how fast you’re spending, not just the total. “$40 in the last 5 minutes on a key that averages $2 an hour” is a better alarm than “you’ve reached 80% of your monthly budget”. Compare each key against its own baseline. Know what happens if the checker goes down. If the budget store is unreachable, do you let calls through or block them? Decide on purpose. Letting them through keeps your product up but lets a loop run unchecked. A common compromise is a small allowance per key that applies only while the store is down. Cheap wins if you can’t build all of this
- Hard-cap agent steps (max_iterations) and tool calls per run.
- Trim or summarise the conversation history so the context doesn’t grow every turn.
- Use separate API keys per environment and per service, so a leak or loop is contained and easy to trace.
- Alert on calls per minute per key. That number is available immediately, unlike the bill. Happy to go deeper on any of these, especially the concurrency side. That’s where most hand-rolled versions break. ( Disclosure : I build a cost-control proxy in this space, but everything above works with a few dozen lines of your own code.) submitted by /u/MaverikSh
Originally posted by u/MaverikSh on r/ArtificialInteligence