Operations / What it costs and what it can take
Choosing how many documents to render at once
Picking a concurrency that uses the available capacity without exceeding it, which is a measured number rather than a guessed one.
Why the default answer is wrong here
Concurrency is usually set to whatever seemed reasonable and never revisited. Too low and a run takes far longer than it needs to, which matters when there is a deadline. Too high and throughput stops improving while error rates and latency climb, which looks like a capacity problem and is a self-inflicted one. The value is measurable in about half an hour and almost nobody measures it.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Measure it: increase concurrency in steps and record throughput and error rate at each. Throughput plateaus before errors begin, and the useful setting is just below the plateau.
- Set it below the point where throughput stops improving, not at it, because the plateau is where latency is already degrading.
- Make it configurable without a deploy, so it can be reduced during an incident.
- Set it per lane rather than globally, so bulk work and interactive work have separate allowances.
- Remember it interacts with your rate budget: concurrency and rate are different limits and hitting either produces the same symptom.
- Re-measure after any material change to document size or to the pipeline, since the number is a property of the whole path rather than of the renderer.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
/* Measured, not guessed. Half an hour of work.
concurrency docs/min p95 ms errors
2 28 2100 0
4 55 2200 0
8 104 2400 0
12 138 3100 0 <- plateau begins
16 141 5200 2 <- throughput flat,
24 139 11800 31 latency and errors up
Throughput stopped improving at 12 and everything else got worse
after it. Set it at 10: below the plateau, not at it. */
const CONCURRENCY = {
bulk: Number(process.env.RENDER_CONCURRENCY_BULK ?? 10),
interactive: Number(process.env.RENDER_CONCURRENCY_INTERACTIVE ?? 4),
};
// Configurable without a deploy, so it can be turned down in an incident.
const pool = new Pool(CONCURRENCY.bulk);
/* Concurrency and rate are different limits with the same symptom.
If reducing concurrency does not help, the constraint is the rate
budget rather than the parallelism. */What people do instead
Raising concurrency when a run is too slow. Past the plateau it makes the run slower rather than faster, because latency rises and errors add retries, and the intuition that more parallelism means more throughput is wrong on exactly the axis being adjusted.
What the symptom looks like
Throughput flat while latency rises is the plateau, and it is the number to set below. Errors appearing is already past it.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
Backpressure when documents are requested faster than they render
What the system does when demand exceeds capacity, which is a decision to make rather than a behaviour to discover.
Planning capacity for a document peak that arrives once a month
Sizing for a workload that is idle most of the time and enormous occasionally, which is the normal shape for documents and the wrong shape for averages.
Designing a client that lives inside a rate limit
Building the caller so it stays within a limit rather than discovering it, which is a different question from what to do when a 429 arrives.
Working out what a generated document actually costs you
Adding up the real per-document cost, which is more than the render and is the number every capacity and caching decision depends on.
Why the first document after a quiet period is slower
The latency difference between a pipeline that has been busy and one that has been idle, which for a monthly workload means every run starts cold.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.