Going to share something that nearly killed a production deployment, because I keep seeing the same mistake in threads here. We shipped an agentic chatbot feature for a fintech client. Passed every test. Worked perfectly in staging under simulated load. Went live. Six weeks in, the API bill arrived. $400 per day per enterprise client. The feature was consuming more in token costs than it was generating in revenue. Nobody had modeled the run cost. Nobody had set guardrails. We discovered it three months in when the client’s finance team flagged the cloud spend. What went wrong (technically): Single-turn LLM calls are predictable. Agentic loops are not. When an AI is taking sequential actions, calling tools, revising its approach, each step burns tokens. Without per-workflow budgets, it burns silently until your cloud bill is a surprise. The architecture fix: Per-workflow token budgets enforced at the retrieval layer, not at the model layer. By the time the model is processing, the tokens are already being consumed by the context construction. You need to control it upstream. Prompt caching for high-frequency context patterns. If the same system context is being prepended to every call in a session, caching it reduces token consumption dramatically, 40-60% reduction on high-frequency workflows in our case. Domain-bounded retrieval. Retrieving only the context chunks relevant to the specific task category, not a broad similarity search across everything, reduces context window width and therefore token consumption per call. Cost ceiling monitoring with circuit breakers. Hard limit on daily cost per workflow type. When 70% of the ceiling is hit, alert. When 100% is hit, pause execution and notify. The principle: Token optimization is not a post-launch cleanup task. It belongs in your architecture spec before a single line of production code is written. Treating it as a “we’ll tune it later” concern is how you get the $400/day bill. submitted by /u/Individual-Bench4448
Originally posted by u/Individual-Bench4448 on r/ArtificialInteligence