The artifact first

Here is what the graph looks like. Queue depth climbs in a straight line. Consumer concurrency is pinned flat at its ceiling. The age of the oldest message rises with the depth, and the two lines are so parallel that someone in the incident channel says “it’s linear, so it’s fine” — which is the sentence that should page a second person.

AWS documents exactly this shape for an SQS-to-Lambda pipeline: when a producer consistently generates messages faster than the function can process them, backpressure occurs, and SQS monitoring shows the age of the oldest message growing linearly along with the approximate number of messages visible (AWS Lambda Developer Guide, “Troubleshoot event source mapping issues”).

The queue is not the problem. The absence of a refusal point upstream is. An unbounded queue is not a buffer; it is a decision to defer refusal until refusal is most expensive — after the work has been accepted, serialized, paid for in storage, and promised to a caller who has already stopped waiting.

Two graphs that look identical and need different fixes

The AWS event-source-mapping page separates two conditions that produce the same rising-age curve, and the distinction matters more than most dashboards admit.

Backlog from a slow or erroring consumer. The page describes a function that throws errors: Lambda returns messages to the SQS queue and, notably, prevents the function from scaling to avoid errors at scale. The observable signature is that the age of the oldest message and the number of visible messages both increase while no messages are deleted, the function’s error count is non-zero and growing, and concurrency has actually reduced while throttling stopped. The fix is in the function, not the queue.

Backpressure from a producer that outpaces the consumer. Here the function is healthy. It is simply slower than the arrival rate, and the age of the oldest message grows linearly alongside the visible message count. The fix is a throughput or admission decision, not a bug hunt.

If your only signal is queue depth, these two are indistinguishable, and you will spend the first twenty minutes of an incident arguing about which one you have. That is a metric design failure, not an incident-response failure.

The bimodal queue, and why “add consumers” is a latency bet

AWS’s builders’ library describes queue-based systems as having two modes: with no backlog, latency is low and the system is in fast mode; once arrival rate exceeds processing rate, it flips into a mode where end-to-end latency grows and it can take a great deal of time to work through the backlog to return to fast mode (“Avoiding insurmountable queue backlogs,” AWS Builders’ Library).

The same article gives the arithmetic that should end most autoscaling arguments: recovering from an hour-long outage requires double the system’s capacity for another hour after recovery, absent other factors. And the compensating factors are not guaranteed — the article notes that other systems your consumer touches may not be prepared for the surge as you drain, which can extend recovery further.

So when someone says “just scale out the consumers,” the honest translation is: we are betting that our downstream dependencies can absorb a step change in throughput for as long as the backlog takes to drain, and that the work in the queue will still be worth doing when it gets there. Both halves of that bet are testable before the incident. Neither is tested by a queue-depth alarm.

Where refusal is cheap, part one: the client

Google’s SRE book describes client-side adaptive throttling in concrete terms. Each client task tracks, over the last two minutes, the number of requests attempted by the application layer and the number accepted by the backend. When the ratio of requests to accepts crosses a multiplier K, the client begins rejecting new requests locally, with a probability derived from that ratio — and those requests fail without reaching the network (“Handling Overload,” Site Reliability Engineering, Google).

The source is explicit about the tradeoff in choosing K: a smaller multiplier makes throttling more aggressive and wastes fewer backend resources on rejected requests, but slows the propagation of state from backend to clients. Google states a general preference for a 2x multiplier, and that is their example, not a prescription for your system.

The same chapter is candid about the limit: client-side throttling may not work well for clients that only very sporadically send requests, because each client’s view of backend state is drastically reduced, and approaches to increase that visibility tend to be expensive. If your traffic comes from cron jobs and webhooks that fire once an hour, adaptive throttling at the client is mostly theater. Know which kind of client fleet you have before you build it.

Where refusal is cheap, part two: the deadline

Deadlines are backpressure that the protocol already carries, and most services throw it away.

gRPC’s deadline guide is blunt about the server side: a server may receive RPCs with an unrealistically short deadline that would not give it enough time to ever respond, which wastes resources and in the worst case can crash the server. A gRPC server automatically cancels a call once the client’s deadline has passed — but the guide adds the sentence that separates a working system from a busy one: the server application is responsible for stopping any activity it has spawned to service the RPC, and long-running processes should periodically check whether the RPC was cancelled (gRPC documentation, “Deadlines”).

That is the whole argument in one line. The framework cancels the call; your code has to notice. A consumer that keeps computing a result nobody will read is not doing work, it is holding a database connection hostage.

Propagation is the other half. The guide notes that propagating a deadline as-is is problematic because clocks on two servers may not be synchronized, so gRPC converts the deadline to a timeout with elapsed time already deducted. If your service calls three dependencies in sequence and passes each one the original deadline, you have built a system that will happily spend three times the caller’s patience.

Retry is a backpressure surface, and it belongs in the contract

Retries are the most common way a struggling service gets kicked while it is down, and the gRPC retry design treats that as a protocol problem rather than a client-code problem.

Three details from the proposal (gRPC Proposal A6, “gRPC Retry Design”) are worth stealing even if you never touch gRPC:

  • The deadline spans all attempts. The call deadline applies across every retry for a given RPC; the operation fails after that time regardless of how many attempts were configured. A retry policy without a shared deadline is a policy for multiplying load.
  • Retry throttling is per server name. The client maintains a token count; failures decrement it, successes increment it by a ratio, and when the count falls to or below half of maxTokens, retries stop until it recovers. This is a circuit breaker expressed as configuration, and it exists because retries are a load source the server cannot see.
  • Servers can push back explicitly. The proposal defines the metadata key grpc-retry-pushback-ms, a signed 32-bit integer of milliseconds to wait before retrying, where a negative or unparseable value means do not retry at all. The server gets a vote.

The proposal also states the idempotency rule plainly: in general, only status codes indicating the service did not process the request should be retried, and a more aggressive set can be used when the service owner knows a method is idempotent. Retry policy is a statement about your method semantics. Putting it in each client’s code means it cannot be revised without a client release, cannot be throttled per server name, and cannot be audited when it goes wrong.

The database as the last queue

There is a queue inside your consumer that no dashboard shows: its own resource budget. PostgreSQL’s documentation on work_mem is a useful illustration of how a per-unit limit multiplies. A complex query may perform several sort and hash operations at the same time, each generally allowed to use as much memory as work_mem specifies before spilling to temporary files, and several running sessions may be doing this concurrently — so total memory used could be many times the value of work_mem (PostgreSQL 18 Documentation, §19.4 “Resource Consumption”).

Read that narrowly. It is not a claim that work_mem causes queue backlogs; it is a demonstration that “we cap it per query” and “we cap it per system” are different statements. The same gap appears in every consumer: a per-request timeout, a per-request memory cap, a per-request concurrency slot — each individually reasonable, collectively unbounded. When the consumer’s resource budget is the real queue, the refusal has to happen before the request is admitted, not after it has allocated.

A three-layer refusal checklist

This is my recommendation, not a finding from the sources. The design principle is that refusal should happen at the cheapest layer that can make an informed decision, and each layer must be independently observable — because a single queue-depth graph cannot tell you which layer is doing the work.

  1. Client-side cap keyed to observed accept ratio. Follow the adaptive-throttling shape: track attempted versus accepted over a rolling window, and reject locally when the ratio crosses your multiplier. Cheap, fast, and useless for sporadic clients — so instrument how many clients actually have enough recent history to make the decision.
  2. Admission control at the API boundary, with per-customer limits. The SRE chapter’s per-customer quota model is the reference: provision capacity against negotiated usage, define limits per customer, and reject out-of-quota requests quickly. AWS’s Well-Architected guidance adds the structural version of the same rule — expose APIs that manage work intake and place requests into internal queues, rather than exposing internal queues to clients (REL05-BP04, “Fail fast and limit queues”).
  3. Consumer-side deadline checks that abandon useless work. Check cancellation between units of work, not just at the start. Propagate remaining time, not the original deadline. If the result will be discarded on arrival, the correct amount of remaining work is zero.

Each layer needs its own refusal metric: local rejections at the client, quota rejections at the boundary, abandoned work at the consumer. If you cannot name all three numbers, you do not know which layer is protecting you.

Measure age, and split it

Queue depth tells you how much work exists. Queue age tells you whether the work is still worth doing. AWS’s Well-Architected guidance lists as an anti-pattern not measuring the age of messages in a queue, describing it as a measurement of latency that shows when consumers are falling behind or erroring out and causing retries.

The builders’ library goes further with a metric design worth copying: AgeOfFirstAttempt records now minus enqueue time only on the first delivery attempt, so that slow or offline devices do not pollute the signal, and a companion metric isolates the first subscriber to keep the number stable in a pub/sub fan-out. The reasoning is that retries and consumer-side slowness mask producer-side overload; separating first attempts from retries makes the producer’s behavior visible again.

Here is a clearly labeled hypothetical, arithmetic on stated parameters rather than a case record. A consumer fleet at 100 concurrent executions, 30-second invocations, batch size 1 drains roughly 3.3 messages per second. A producer at 10 messages per second builds a backlog at about 6.7 messages per second. Scaling to the concurrency quota does not change the sign of that difference; it only changes how long the queue takes to become the largest object in your account. The number that matters is not the depth — it is how many seconds of age you are willing to serve, and whether the producer can be told no.

What I would do differently

Three things, in plain terms.

First, I would put the refusal metric in the same dashboard as the queue metric before the first incident, not after. The cost of not doing this is measured in incident minutes spent distinguishing two failure modes that the vendor documentation already separates.

Second, I would write the retry policy into the service contract on day one, with a shared deadline across attempts and a server-side pushback path. The cost of retrofitting it is a client release cycle per client, and the scar is every retry storm that happened in between.

Third, I would stop treating “the queue is bounded” as a safety property. A bound that is larger than the work anyone will still want is an unbounded queue with a smaller number in the config file.

The heuristic

Put the refusal where it is cheapest, and make the queue prove it was heard.

The operational test is one question per layer: can you name the metric that shows this layer refused work in the last hour? If the answer is no for any layer, that layer is not backpressure — it is hope with a retention policy.