Operations / Correct under repetition
Designing a client that lives inside a rate limit
Building the caller so it stays within a limit rather than discovering it, which is a different question from what to do when a 429 arrives.
Why the default answer is wrong here
Most rate-limit handling is reactive: send until rejected, then back off. That works and it wastes the rejected requests, adds latency to everything behind them, and behaves badly when many workers do it at once, because they all back off and return together. A client designed around the limit paces itself, which is more predictable for everybody, and it still needs the reactive path for the cases its pacing did not anticipate.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Pace outbound requests with a token bucket sized below the limit, so the limit is a backstop rather than a control mechanism.
- Share the budget across workers rather than giving each one its own, or the effective rate is the per-worker rate multiplied by the number of workers, which changes when the fleet scales.
- Honour Retry-After when a rejection does happen. The server is telling you when it will be ready and a guess is not better than that.
- Add jitter to every wait. Synchronised backoff across a fleet produces a thundering return, which causes the next rejection.
- Give bulk work a smaller share of the budget than interactive work, so a run cannot consume the limit that a waiting user needs.
- Monitor how close to the limit you run. A client that is at ninety percent of its budget is one traffic increase away from failing, and that is knowable in advance.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
// Paced below the limit, shared across workers, so the limit is a
// backstop rather than the control mechanism.
const budget = new SharedTokenBucket({
key: "pdf-render",
ratePerSecond: 8, // deliberately below the ceiling
burst: 16,
});
async function call(spec, { lane }) {
// Bulk gets a smaller share than interactive.
await budget.take(lane === "bulk" ? 2 : 1);
try {
return await pdf.post("/v1/pdf", spec);
} catch (err) {
if (err.status !== 429) throw err;
// The server's own answer, with jitter so a fleet does not return
// in lockstep and cause the next rejection.
const after = Number(err.headers?.["retry-after"] ?? 1);
await sleep(after * 1000 * (0.8 + Math.random() * 0.4));
return call(spec, { lane });
}
}
// Knowable in advance: a client at ninety percent of budget is one
// traffic increase away from failing.
metrics.gauge("render.budget.utilisation", budget.utilisation());What people do instead
Giving every worker its own limiter. The effective rate is then the per-worker rate times the number of workers, so the pacing is correct in development with one worker and wrong in production with twelve, and it changes silently whenever the fleet scales.
What the symptom looks like
Rejections arriving in bursts rather than steadily means the fleet is synchronised, which points at missing jitter. A steady low rate of rejections means the budget is set slightly too high.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
Backpressure when documents are requested faster than they render
What the system does when demand exceeds capacity, which is a decision to make rather than a behaviour to discover.
Retrying a render safely, and which failures are safe to retry
Deciding which failures should be retried and which should not, because a blanket retry on a document endpoint is a duplication mechanism.
Planning capacity for a document peak that arrives once a month
Sizing for a workload that is idle most of the time and enormous occasionally, which is the normal shape for documents and the wrong shape for averages.
Idempotency for a document that must not be generated twice
Making a repeated request produce the same document rather than a second one, which matters more here than in most APIs because the artefact is often numbered.
Stopping the same document being requested twice at once
Collapsing concurrent identical requests, which is a different problem from idempotency because both arrive before either has finished.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.