A p99 latency figure is a property of the measurement procedure that produced it, not a property of the service. Two teams can quote different p99 values for the same server on the same afternoon and both be arithmetically correct, because the load generator’s admission policy decides which requests exist to be counted at all.

That is the whole argument. Everything below is the mechanism, the arithmetic, and the reporting rule that follows from it.

The artifact: a server that is dead 20% of the time, reported as healthy

There is a public demonstration repository by Judofyr that makes this concrete. The Go server does one thing: for eight seconds it behaves normally, then it takes a lock and sleeps for two seconds, during which every request blocks. The system is completely unresponsive for two seconds out of every ten. Twenty percent of wall-clock time is a stall.

Benchmarked with wrk at 10 connections, the reported distribution was:

Latency Distribution
  50%   90.00us
  75%  120.00us
  90%  350.82ms
  99%    1.83s

Then the same server, same duration, measured with wrk2 at a fixed rate of 70k requests per second:

Latency Distribution (HdrHistogram - Recorded Latency)
  50.000%    1.48s
  75.000%    1.85s
  90.000%    2.31s
  99.000%    2.92s

Read the medians, not just the tails. The closed-loop run says half your requests complete in 90 microseconds. The fixed-rate run says half your requests take 1.48 seconds. Those are not two estimates of the same quantity with different error bars. They are measurements of two different things, and only one of them resembles what a user experiences.

The mechanism, in two sentences

In wrk‘s execution model, each connection begins sending a request only after receiving the response to the previous one, so a high-latency response causes the generator to stop sending — and therefore stop sampling — precisely during the high-latency period. Percentiles are then computed over the requests that were sent, and because the vast majority of those requests were sent during healthy periods, the distribution is skewed toward the healthy period.

Gil Tene, who coined the term and wrote wrk2, states the mechanism in the tool’s README directly: “Since each connection will only begin to send a request after receiving a response, high latency responses result in the load generator coordinating with the server to avoid measurement during high latency periods.” The Judofyr README puts the arithmetic consequence plainly: “The percentiles are around the requests, not time spans.”

This is why the error is dangerous rather than merely annoying. It is not random noise that averages out. It is a one-directional bias: the reported number is always better than the experienced one, and it is better by an amount proportional to how bad the bad periods were. A system that never stalls is measured correctly. A system that stalls catastrophically is measured most wrongly. The methodology fails hardest exactly when you most need it to work.

Why the tail is not the only casualty

The intuition most engineers carry is that coordinated omission inflates the tail and leaves the median alone. The Judofyr numbers refute that. Under wrk the p50 was 90 microseconds; under wrk2 at a comparable throughput it was 1.48 seconds. When a stall occupies a fifth of the timeline and the generator refuses to send during it, the requests that would have been slow are simply never created, and the surviving population is dominated by the fast periods.

This matters for how you argue about it internally. If someone proposes that the median is trustworthy and only the p99 is suspect, they have the model backwards. The median is trustworthy only if the generator’s send schedule is independent of the server’s response times. In a closed loop, it is not.

Two remedies, and they are not equivalent

Schedule-first generation

The first remedy is to decide when each request should be sent before the run starts, and to measure latency from that intended send time rather than from the moment the request actually left the client. wrk2 does exactly this. From its README: “Rather than measure response latency from the time that the actual transmission of a request occurred, wrk2 measures response latency from the time the transmission should have occurred according to the constant throughput configured for the run.”

The consequence is that when the server stalls, the requests that could not be sent on schedule are still counted, and their latency includes the time spent waiting for the generator to catch up. The README is explicit that this technique generalizes: it “can be applied to variable throughput loaders” provided you have “a ‘model’ or ‘plan’ that can provide the intended start time” for each request. Constant throughput just makes that plan trivial.

The README also documents the magnitude. In one example run, a single 1.4-second disruption introduced by suspending the Apache process produced a corrected 99th percentile roughly 200 times larger than the uncorrected one. That is one stall, in one run, on a local test server.

Post-hoc correction against an expected interval

The second remedy is to keep your existing closed-loop sample set and reconstruct the missing samples. HdrHistogram supports this: the wrk2 README notes that “HdrHistogram includes a simple way to compensate for Coordinated Omission when a known expected interval between measurements exists.” The method is to take any observed sample larger than the expected interval and emit additional samples by repeatedly subtracting the interval until the remainder is non-positive.

Here is a labeled illustration of the arithmetic, not a measurement of any system: suppose your generator was configured for one request every 80 ms, and one observed sample came back at 2.0 seconds. Correction expands that single sample into a descending series — 1.92 s, 1.84 s, 1.76 s, and so on down to the remainder — because during those 2.0 seconds the generator should have issued roughly 25 requests, and the users behind those requests were waiting too. The original sample stays; the reconstructed ones are added.

This is a fallback, not an equivalent. It requires a defensible expected interval, and it fabricates samples that were never observed. If your generator’s rate drifted during the run, or if the workload was transactional rather than request-independent, the expected interval is a fiction and the correction inherits that fiction. Say so in the report.

The workload model question nobody writes down

Artillery’s engineering blog frames the boundary condition well: coordinated omission “is a measurement issue that can happen when an open system is tested with a closed workload generator,” and the tell is structural — “Any time you have a fixed number of connections, ‘threads’ or ‘iterations’, you will run into CO.”

But the same post makes the honest counterpoint that a pure open-loop generator only fits a narrow class of systems: “The only way to send requests at constant RPS is if each request is completely independent of other requests.” Real user journeys are sequential — load a page, click a repository, open the Issues tab — and imposing a constant request rate on that sequence is meaningless because each step depends on the previous response.

The resolution Artillery proposes is a hybrid: new users arrive according to a distribution (open loop), and each user then experiences back-pressure from the service (closed loop per user). That model is safe from coordinated omission because while user A waits, users B through Z are still arriving and still recording their own outsized response times.

So the test-design rule is not “always use an open-loop tool.” It is: decide the arrival model before you choose the tool, and write the model into the report. If the workload is genuinely request-independent, generate at a fixed rate and measure from intended send time. If it is transactional, model per-user closed loops inside an open arrival process, and state that you did.

Where the generator’s schedule interacts with the contract

Two protocol-level behaviors make the gap between the test number and the production number wider than the generator alone explains.

Timeouts and retries. AWS’s Builders’ Library documents the failure mode precisely: “Setting the timeout too low has two risks: Increased traffic on the backend and increased latency because too many requests are retried. Increased small backend latency leading to a complete outage, because all requests start being retried.” A closed-loop test that never exercises the retry path will report a p99 that production cannot reproduce, because production’s tail includes retried requests that the test never issued. The same article quantifies the amplification risk in layered systems: a five-deep call stack with three retries per layer multiplies database load by 243x. If your load test models one layer and production has five, the tail you measured is not the tail you will get.

Metric cardinality. OpenTelemetry’s metrics SDK enforces a default cardinality limit of 2000 unique attribute combinations per metric stream, and when the limit is reached, measurements are folded into a single overflow data point marked otel.metric.overflow=true. The documentation is careful about the consequence: “Once the stream overflows, a measurement for {url.path=/checkout, success=false} is folded into the single overflow data point {otel.metric.overflow=true}, dropping success along with url.path. A query for success=false then misses that measurement.” This is not a latency-underreporting mechanism, and it should not be described as one. But it is a real way that a production dashboard can undercount the error population whose latency you are trying to correlate against your test results. If your p99 chart and your error-rate chart are built on the same overflowing metric stream, they may not be describing the same requests.

The reporting rule

A latency percentile without its workload model and expected interval is not a reproducible number. It is a claim about a procedure that the reader cannot reconstruct. The fix is cheap and it belongs in the template, not in someone’s memory:

  • Name the model. Open, closed, or hybrid. If hybrid, state the arrival distribution and the per-user loop structure.
  • Name the rate. Requests per second, or the arrival distribution parameters. A closed-loop run with N connections has no rate until the server defines one, which is the entire problem.
  • Name the correction. If the samples were corrected post hoc, state the expected interval used and that the tail is reconstructed rather than observed.
  • Gate on the corrected figure. An SLA threshold applied to an uncorrected closed-loop sample set is a gate on the generator’s patience, not on the service.

The last point is the one that changes behavior. If your release gate says p99 < 200 ms and the number feeding it came from a closed-loop generator, then the gate passes whenever the generator was willing to wait. It fails to describe the service at all during the intervals that matter.

What I would do differently

I would stop treating the load-test report as a measurement and start treating it as a procedure with an output. The procedure has three decisions — arrival model, send schedule, correction policy — and all three need to be visible to anyone who reads the number. The cost of writing them down is a paragraph. The cost of not writing them down is a release gate that has been quietly measuring the wrong thing for as long as anyone can remember.

I would also stop accepting “the p99 is fine” as an answer to “what did the generator do during the worst interval?” If nobody on the call can answer the second question, the first one is not yet evidence.

The heuristic

Schedule before you sample. Decide when each request should have been sent before the run starts, measure latency from that intended time, and write the schedule, the rate, and the correction policy into the report next to the percentile. If you cannot state the intended send time for the worst interval in the run, you do not have a latency measurement — you have a record of how long your generator was willing to wait.