Operations / Knowing what it is doing
Measuring document latency in percentiles rather than averages
Recording the distribution rather than the mean, because document render times are skewed by document size and the tail is what people experience.
Why the default answer is wrong here
Document render time depends heavily on the document, and documents vary enormously: a one-page receipt and a two-hundred-page statement go through the same endpoint. That makes the distribution wide and heavily skewed, so a mean describes neither case and moves whenever the mix of document types changes. A dashboard showing average render time will look stable through a change that made the large documents twice as slow.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Record percentiles, and record the high ones. The median is the easy case and the ninety-ninth is the customer who is complaining.
- Break the measurement down by document kind, because mixing a receipt and a statement into one distribution makes both unreadable.
- Record the page count alongside the duration, so latency can be normalised per page and a slow render can be distinguished from a large one.
- Measure at the boundary you care about. Render duration answers a capacity question; time from request to the document being available answers the user's question, and they are different numbers.
- Watch the shape rather than the number. A distribution developing a second hump usually means a new document kind or a code path that is not using a cache.
- Keep enough history to compare against the same point in the previous cycle, because monthly workloads are only comparable to other month ends.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
// By kind, with the page count, so a slow render can be told apart
// from a large one.
metrics.histogram("render.duration_ms", durationMs, {
kind: "statement",
pages: bucketPages(res.pages), // 1, 2-5, 6-20, 21-100, 100+
cached: String(cached),
});
// Two different questions, two different measurements.
metrics.histogram("render.duration_ms", renderMs, { kind }); // capacity
metrics.histogram("document.time_to_available_ms", totalMs, { kind }); // the user
/* What to alert on, and what not to:
p50 rising capacity, worth watching
p99 rising a customer is already having a bad time
mean rising usually the document mix changed; not a signal
Normalising per page separates the two causes:
render.duration_ms / pages
A rise in that is a real slowdown. A rise in the raw duration with
this flat is simply larger documents. */
metrics.histogram("render.ms_per_page", durationMs / Math.max(res.pages ?? 1, 1), { kind });What people do instead
Alerting on the mean. It moves whenever the mix of document types shifts, which happens on a schedule in most businesses, so the alert fires on the first of the month for reasons that are not a problem and is ignored by the time something real happens.
What the symptom looks like
Duration rising while duration per page stays flat means documents got bigger, not slower. That distinction changes who needs to look at it, and it is only available if the page count is recorded with the timing.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
Tracing one document from request to delivery
Following a single document across the queue, the render and the delivery, which is the only way to answer where the time went or where it stopped.
Why the first document after a quiet period is slower
The latency difference between a pipeline that has been busy and one that has been idle, which for a monthly workload means every run starts cold.
Planning capacity for a document peak that arrives once a month
Sizing for a workload that is idle most of the time and enormous occasionally, which is the normal shape for documents and the wrong shape for averages.
What to log about a render, and what must never be logged
The fields that make a render diagnosable weeks later, and the one thing that must not be in the log at any level.
Setting timeouts along the whole path, not just in the client
Making the timeouts at each hop consistent with each other, because a path whose timeouts disagree fails in the least useful way available.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.