Operations / What it costs and what it can take
Why the first document after a quiet period is slower
The latency difference between a pipeline that has been busy and one that has been idle, which for a monthly workload means every run starts cold.
Why the default answer is wrong here
Almost everything in the path gets faster with use: caches fill, connections are established, code paths are optimised. A document pipeline that runs once a month is cold every time, so the measurements taken during a busy test do not describe the run that matters. The first documents of a monthly run are the slowest, and they are also the ones somebody is watching, which makes the run look worse than it is and occasionally trips a timeout that never fires in testing.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Measure the first request separately from the rest, or the average hides it entirely.
- Expect the first document of a monthly run to be materially slower and set timeouts against that rather than against the steady state.
- Warm the path deliberately before a scheduled run: a small render a few minutes ahead costs almost nothing and moves the cold start off the critical path.
- Warm your own caches too, not just the remote ones. The template, the data queries and the connection pool are all cold after a month of idleness.
- Do not confuse cold with broken. A slow first document followed by fast ones is normal and does not need investigating; a slow tenth one does.
- Test after an idle period, not after a warm-up loop, or the test measures a state the production run never reaches.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
// Measure the first separately, or the average hides it.
let isFirst = true;
async function renderTracked(spec) {
const started = Date.now();
const res = await pdf.post("/v1/pdf", spec);
metrics.histogram("render.duration_ms", Date.now() - started, {
position: isFirst ? "first" : "steady",
});
isFirst = false;
return res;
}
// Warm the path before a scheduled run: cheap, and it moves the cold
// start off the critical path.
schedule.at("04:55", async () => {
await pdf.post("/v1/pdf", {
html: "<!doctype html><p>warmup</p>",
options: { format: "A4" },
});
await db.query("SELECT 1"); // connection pool
await templates.preload("payslip"); // our own cache
});
schedule.at("05:00", startPayrollRun);
/* A slow first document followed by fast ones is normal. A slow tenth
one is not, and the distinction is only visible if the first is
measured separately. */What people do instead
Benchmarking after a warm-up loop and setting timeouts from those numbers. The production run never reaches that state on its first document, so the timeout that was comfortable in testing is the one that fires at five in the morning once a month.
What the symptom looks like
A latency distribution with a small cluster far to the right, at a count matching the number of runs, is cold starts rather than a tail problem, and it needs warming rather than capacity.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
Measuring document latency in percentiles rather than averages
Recording the distribution rather than the mean, because document render times are skewed by document size and the tail is what people experience.
Planning capacity for a document peak that arrives once a month
Sizing for a workload that is idle most of the time and enormous occasionally, which is the normal shape for documents and the wrong shape for averages.
Setting timeouts along the whole path, not just in the client
Making the timeouts at each hop consistent with each other, because a path whose timeouts disagree fails in the least useful way available.
Working out what a generated document actually costs you
Adding up the real per-document cost, which is more than the render and is the number every capacity and caching decision depends on.
Choosing how many documents to render at once
Picking a concurrency that uses the available capacity without exceeding it, which is a measured number rather than a guessed one.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.