Latency budgets for LLM workflows in production

Most AI reliability failures I’ve seen weren’t “the model got worse.” They were pathway failures: an unbudgeted tool call, an evaluation that only lived in a notebook, or a deploy process that treated every prompt change like a hotfix.

On an AI application platform supporting 10,000+ users, I treated generation like any other production system. That meant a clear ownership boundary for the pathway — public APIs, backend services, evaluation, monitoring, and CI/CD — not just the prompt text.

The first design move was a latency budget. If product needed a responsive developer experience, we couldn’t discover p95 regressions after a marketing push. We measured path length through services and model calls, cached only where correctness allowed, and refused “just add another call” changes that blew the budget without an explicit tradeoff.

The second move was evaluation in the release path. Pre- and post-deploy checks beat anecdotal playground wins. When reliability and availability moved — latency down ~30%, availability up ~20% — it was because operability was part of the platform, not a cleanup sprint.

The third move was deployment speed as a reliability feature. Cutting deploy time ~40% wasn’t vanity DevOps; it shortened the loop between detecting a bad behavior and shipping a fix. Feature iteration rose because the pathway was safe enough to use continuously.

Senior takeaway: if you only optimize the model call, you own half a product. Own the pathway — and make tradeoffs about latency, eval coverage, and release risk visible to product and research partners.