Operations
A stress test where nothing breaks was not a test
We spent a day trying to break our own platform. Three things broke. That is the useful part of the report, not the numbers before it.
Most published load tests answer the wrong question. They show how many queries a system manages per second while everything is fine. What matters is what happens when it stops being fine — and whether you can see it coming.
What a stress test has to show
- Where the limit sits: at how many concurrent users the first query stops completing
- What happens at the limit: a clear rejection, or a system that hangs
- How long recovery takes: how quickly normal answers resume once the load stops
- Whether anyone had to intervene — and if so, how often
The third and fourth questions are the ones that matter in operations. A system that slows under load and picks itself up afterwards is manageable. One that hangs and needs a restart is not — however good the numbers looked beforehand.
Four scenarios, escalating
We ran against a deliberately modest installation: a single virtual machine with six cores and 24 GB, no swap, the complete platform on it, and 60 million rows as the data. Every scenario went through the normal, secured interface — with authentication, authorization and audit, not around them.
| Scenario | Load | What came out |
|---|---|---|
| User staircase | 8 → 16 → 32 → 64 concurrent analysts | Error-free up to 32; from 64 onward, rejections |
| Sustained fire | 32 users, short queries, 60 seconds | 3,176 queries, median 0.59 s |
| Memory monster | One deliberately senseless query | Aborted within seconds, everything else carried on |
| Death blow | 128 users and 8 monsters at once | 42 full analyses completed, the rest rejected |
The most important number is not the fastest one
Across every run: not a single crash of the query engine. Thirty seconds after the harshest load, an ordinary query was answered in 76 milliseconds — no intervention, no cleanup.
What makes that possible is a property load tests rarely credit: rejection. Fire beyond the promised concurrency and you get a clear message rather than a hanging system. That is not an invention but a configuration of the query engine — Trino knows hard limits for concurrently running queries and for the length of the queue; once both are reached, requests are rejected.
In the results that looks bad: 9,433 rejected requests in the harshest scenario. The number is inflated mainly because the test tool retried rejected requests immediately. What counts is the figure beside it — 42 full analyses over 60 million rows completed cleanly in the same window.
Three hits, honestly
A destruction test in which nothing breaks has not tested seriously. We had three hits:
- The authentication service failed under a login storm: 32 concurrent logins pushed it past its memory limit at the time, and the watchdog restarted it. 23 seconds to full readiness, with no intervention. The remedy was a number in the configuration.
- The engine tore on importing 60 million rows — as long as its container was set too tight. With the limit appropriate to that data volume the same import ran through in 73 seconds, more than 800,000 rows per second.
- Two findings on the side, neither a platform fault: brute-force protection kicked in at 32 parallel logins of the same account — a nuisance in the test, exactly right in operation. And our own measuring tool blew up a helper container because it was accidentally running inside it.
Sizing is not an architecture fault
The second hit is the most instructive, because it forces a distinction that routinely blurs in procurement. A system that aborts when memory is set too tight does not have an architecture problem — it has a sizing problem. The test question: does the fault disappear when you change a number in the configuration? Then it is sizing. Does it remain? Then it is architecture.
The difference is practical: sizing is solved before going live, with a measurement. Architecture is not solved at all — at least not in the project where you notice it.
What we learned about choosing a size
The obvious response to load is to add a second compute node. On the same machine that gains nothing — it shares the same six cores, and in our comparison that configuration trailed the simple one in almost every scenario. What did help a great deal: giving the one engine more memory. From 8 to 16 GB, completed analyses under peak load rose by 70 per cent and short-query throughput by 43 per cent.
The resulting rule is unremarkable and probably holds for any engine of this kind: one engine per machine, with as much memory as the hardware safely allows. Only beyond that, further machines that bring their own cores. Putting more nodes on the same hardware means managing more without computing faster.
What the test does not show
It shows one machine at one data volume. It says nothing about terabyte estates, nothing about a cluster of machines, and nothing about query patterns other than those tested. Turning such numbers into a general performance claim is exactly what makes so many load tests worthless.
What it does show is something else, and more useful for an operational decision: that the limit is known, that it is enforced by clean rejection, and that nobody has to clean up after it.
Sources
Every figure in this article is sourced. Where no defensible source exists, no figure is given.
- Trino – Resource groups (concurrency and queueing)
- lavalake – stress test of the reference installation, 6 September 2026 (own measurement)