Search for a command to run...
Standard load-testing phases by traffic volume. For an agent with third-party upstreams and multi-call turns, attribution is the harder problem.
No comments yet. Be the first to comment.
The standard load-testing playbook phases along a single axis: traffic volume. Start with a smoke test to confirm the system runs under minimal load. Ramp to a load test to find normal operating performance. Push to a stress test to find the breaking point. Sustain with a soak test to catch memory leaks and degradation over time. This progression is well-established, well-supported by tooling, and answers its question precisely: at what traffic level does this system fail?
For an LLM agent, that's the wrong question to lead with.
A standard backend service owns its critical path. Every layer is observable. When something breaks under load, there are metrics on the application server, the database, the cache. Attribution is reconstructable after the fact.
An LLM agent has a different shape. A single user turn typically triggers multiple upstream API calls: an orchestration step, one or more model calls, potentially an external search request. The upstream provider operates under rate limits you don't control, at latency you can't optimise. Response times span an order of magnitude — a simple question completes in seconds, a research-heavy turn takes minutes.
Measure that from the outside under load and the load-profile phases will find a breaking point. What they won't tell you is whether you hit the application server ceiling, the upstream provider's rate limit, or something in between. Because the approach phases by traffic volume rather than by dependency, upstream constraints surface incidentally — a watch for this footnote rather than something the method was built to find.
The alternative is to phase not along traffic volume, but along dependency exposure.
Phase 1: mock all upstream dependencies. The application runs against local mock servers that handle every external API call without leaving the machine. The purpose is narrow: find the platform's throughput ceiling — the application server, the connection handling, the streaming layer — before any upstream dependency is in the picture. This phase is fast, cheap, generates zero real API traffic, and answers exactly the question it's designed to answer.
Phase 2: isolate one real upstream dependency. Test it directly, bypassing the application. For a model provider, that means direct API calls at increasing concurrency until a rate-limit response appears. For a search API, the same. The purpose is to characterise what each upstream contributes on its own: its actual limits, its real response times under concurrency. This is not an integration test. It's a dependency audit.
Phase 3: full system at low concurrency with real dependencies. Not a stress test — a validation test. The intent is that by this point Phases 1 and 2 have characterised every layer individually, so Phase 3 checks whether the integrated system behaves as those results predict. A surprise here should indicate an interaction effect rather than an unknown ceiling.
Databricks' documentation on load-testing an agent app does the first step: mock the model to isolate infrastructure throughput from model latency, then ramp to saturation with Locust to find peak QPS. That's still the load-profile axis — mock-then-ramp is Phase 1 folded into a traffic-volume progression. Phases 2 and 3 are where this proposal extends past where that approach stops: characterise each upstream on its own, then validate the integrated system against what those phases established.
Their guide is also candid about the limit of a mocked phase, and it's worth repeating. A mock that returns instantly makes the workload CPU-bound, while a real agent mostly sits waiting on a model endpoint. The throughput number you get describes the platform, not the agent — which is exactly what Phase 1 is for, provided nobody later quotes it as an end-to-end figure.
The load-profile approach produces a throughput number, then relies on post-hoc tracing to attribute which component broke. That's a reasonable approach. It also means the most integrated run is the most informative one: you discover which layer is the constraint after running the full system under load.
Phasing by dependency exposure reverses the direction. The aim is to establish attribution before the integrated test runs. If Phase 1 shows the platform ceiling sits well above expected production load, and Phase 2 characterises the upstream on its own, then Phase 3 becomes a confirmation pass rather than an exploration.
The load-profile axis answers: at what traffic level does this system fail? The dependency-isolation axis answers: which layer would fail first, and why?
For services with homogeneous latency and fully-owned infrastructure, the load-profile progression is the natural choice — it's mature, well-tooled, and the attribution problem is tractable after the fact. For LLM agents, with heterogeneous latency, third-party upstreams, and multi-call turn structures, the attribution question is the one worth answering first.
The argument for isolating layers first is that it costs less, catches wrong optimisations earlier, and gives you a cleaner baseline to validate production against.
All of this is a property of the scripts, not the plan.
Each phase changes one variable is easy to write in a planning document and easy to lose in implementation. A phase gets scoped down to one scenario because a full blended mix is slow to run. One phase runs locally because mocking is cheapest there, and another runs on the deployment because hosting limits only exist in the deployment. A think-time gets adjusted between runs. Each of those decisions is individually reasonable. Together they mean two runs differ in three ways, and a number produced by comparing them can't be attributed to any one of them.
Nothing catches this for you. The load-profile playbook doesn't, because it varies one thing by construction and holds the rest constant by convention rather than by design — which works right up until your phases legitimately need to run in different places. And a phase plan that isn't enforced somewhere in the harness will eventually get cited as though it were.
So if you adopt this axis, the thing worth building isn't the phases. It's a record, attached to each run, of what that run held constant. The axis you phase along should match the topology of the system you're testing — and the claim you make about each phase should be checkable against the run that produced it.
I didn't arrive at that from theory. I got there going back through a campaign of my own months afterwards and finding that two runs I'd been comparing differed in more ways than I'd remembered. The axis you phase along should match the topology of the system you're testing — and the claim you make about each phase should be checkable against the run that produced it.
If you've phased a load test this way, I'd be curious how you recorded what each run held constant. That's the part I still don't have a good answer for.
Sravan Vidiyala — I build AI systems that hold up under production load and compliance scrutiny: agent architecture, evaluation harnesses, and the performance and safety work underneath. Twenty years across data engineering, data science and AI. Based in Sydney.
Available for consulting on production AI systems, and open to new opportunities.
