Operations / Knowing what it is doing
Setting timeouts along the whole path, not just in the client
Making the timeouts at each hop consistent with each other, because a path whose timeouts disagree fails in the least useful way available.
Why the default answer is wrong here
There is a timeout in the browser, one in the load balancer, one in the application server, one in the HTTP client making the render call, one in the queue's visibility window and one in the renderer. They are usually set independently, by different people, at different times, and the result is a path where an inner timeout is longer than an outer one. When that happens the outer hop gives up while the inner work continues, so the work is done and thrown away, the caller retries, and the system does the expensive thing twice while reporting failure both times.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Order them: each hop's timeout should be shorter than the one outside it, with enough margin for the outer hop to return a useful error.
- Write them all down in one place. They cannot be reasoned about individually and nobody knows all of them by memory.
- Set the queue's visibility window longer than the slowest render plus its retries, or a message is redelivered while it is still being processed, which produces a duplicate.
- Make a timeout produce a specific error rather than a generic one, so a later reader can tell which hop gave up.
- Budget from the outside in: decide what the user will wait, then subtract, rather than setting each hop to a comfortable default.
- Re-check them after any architectural change, because inserting a hop into the middle of a path silently invalidates the ordering.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
/* One place, because nobody knows all of these by memory and they
cannot be reasoned about one at a time.
Budgeted from the outside in.
browser / client fetch 30s
load balancer 28s
application handler 25s
render HTTP client 20s <- shorter than the handler
renderer's own limit 18s <- shorter again
Each inner hop is shorter than the one outside it, with margin for
the outer hop to return a useful error rather than being cut off.
Separately, and easy to get wrong:
queue visibility window > slowest render + all retries
If it is shorter, a message is redelivered while still being worked
on, and the result is a duplicate document rather than a timeout. */
const TIMEOUTS = {
handler: 25_000,
renderClient: 20_000,
queueVisibility: 300_000, // well above 20s x 4 attempts + backoff
};
const res = await pdf.post("/v1/pdf", body, {
timeout: TIMEOUTS.renderClient,
onTimeout: () => {
// Specific, so a later reader can tell which hop gave up.
throw new RenderTimeout("render client timed out at 20s");
},
});What people do instead
A client timeout longer than the gateway's. The gateway returns an error to the user while the render continues to completion, the work is paid for and discarded, and the retry starts a second one. The system is doing double the work and reporting a failure rate at the same time.
What the symptom looks like
Renders completing successfully in the logs for requests the user saw fail is the fingerprint of an inverted timeout ordering, and it is worth looking for specifically because it looks like two unrelated problems.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
Retrying a render safely, and which failures are safe to retry
Deciding which failures should be retried and which should not, because a blanket retry on a document endpoint is a duplication mechanism.
Tracing one document from request to delivery
Following a single document across the queue, the render and the delivery, which is the only way to answer where the time went or where it stopped.
Generating a document in the request or in the background
Whether the user waits for the document, which is a decision about the document's size and the user's expectation rather than about performance.
What to log about a render, and what must never be logged
The fields that make a render diagnosable weeks later, and the one thing that must not be in the log at any level.
Measuring document latency in percentiles rather than averages
Recording the distribution rather than the mean, because document render times are skewed by document size and the tail is what people experience.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.