Backpressure when documents are requested faster than they render

Why the default answer is wrong here

Every document pipeline has a rate above which it cannot keep up, and the behaviour at that point is decided by default unless somebody decides it deliberately. The default is usually to accept everything and queue it, which converts a capacity problem into an unbounded latency problem and eventually into memory exhaustion. The alternatives, shedding load, rejecting with a retry hint, or degrading to a simpler output, are all better than an unbounded queue, and all of them require having thought about it first.

The decisions

Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.

• Bound every queue. An unbounded queue does not remove the limit, it hides it and moves the failure somewhere less recoverable.
• Decide what happens at the bound: reject, shed, or block the producer. All three are defensible and the default of growing forever is not.
• Reject with a retry hint rather than a bare error, so a well-behaved client can wait rather than hammer.
• Separate interactive traffic from bulk traffic, so a payroll run cannot starve a user waiting for a receipt. Two queues with different priorities is usually enough.
• Apply the limit at the edge rather than deep inside, so work is rejected before resources have been spent on it.
• Make the degraded mode explicit if you have one. Serving a cached or simplified document under load is a good answer when it is a decision and a bad one when it is a surprise.

In practice

A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.

// Two lanes, both bounded. Bulk cannot starve interactive.
const lanes = {
  interactive: new Semaphore(8),    // a user is waiting
  bulk: new Semaphore(4),           // a run, nobody waiting
};

async function render(spec, { lane = "interactive" } = {}) {
  const permit = await lanes[lane].tryAcquire({ timeout: 250 });

  if (!permit) {
    if (lane === "bulk") {
      // Nobody is waiting: push back on the producer instead of queueing.
      throw new Backpressure("bulk lane saturated");
    }
    // Somebody is waiting: reject with a hint rather than a bare error.
    const err = new TooBusy("try again shortly");
    err.retryAfter = 5;
    throw err;
  }

  try {
    return await pdf.post("/v1/pdf", spec);
  } finally {
    permit.release();
  }
}

/* The decision being made explicitly rather than by default:
     bounded, both lanes           an unbounded queue hides the limit
     bulk pushes back              the producer slows down
     interactive rejects with a
       Retry-After hint            a client can wait politely
   The default, accepting everything and queueing, turns a capacity
   problem into an unbounded latency problem and then into memory
   exhaustion.                                                        */

What people do instead

An unbounded queue in front of the renderer. It looks like resilience and it is deferral: the queue grows, latency grows with it, and the eventual failure is memory exhaustion in the queueing process rather than a clean rejection at the edge.

What the symptom looks like

Queue depth growing monotonically is the signal that demand exceeds capacity, and it is visible long before anything fails. Latency rising with a flat throughput says the same thing from the other direction.

Frequently asked

Related operational topics

The decisions that depend on each other, then the rest of the same group.