Operations / Knowing what it is doing
What to do when document generation stops working
The decisions to make when rendering is unavailable, most of which are about what to tell people and what to preserve rather than about restoring service.
Why the default answer is wrong here
When rendering is down, the technical response is usually out of your hands: either the dependency recovers or it does not. What is in your hands is everything around it, and those decisions are worse when made live. Whether to keep queueing work or stop accepting it, whether to tell users now or wait, whether a scheduled run should be skipped or delayed, and whether the requests arriving in the meantime are preserved or lost. A pipeline with no plan loses the requests, which is the one outcome that is not recoverable afterwards.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Preserve the intent above all. Requests arriving during an outage should become durable work items, so nothing is lost even if nothing can be produced.
- Stop the retries early rather than letting them hammer a recovering service, which extends the outage for everybody.
- Decide in advance whether a scheduled run during an outage is skipped or delayed, because that decision has a business answer and not a technical one.
- Tell users something specific. A document that will arrive later is a very different message from one that failed, and the pipeline usually knows which.
- Plan the recovery burst before it happens: everything queued during the outage will be attempted at once when service returns, which is a load spike caused by the fix.
- Write down who decides. The judgement calls here are business decisions and the person on call at three in the morning should not be making them for the first time.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
/* The runbook, written before it is needed.
1. CONFIRM
Is it rendering, or is it us? Check the render error codes:
renderer_unavailable / timeout the dependency
invalid_request / payload_too_large a change we made
2. PRESERVE (the only irreversible decision)
Requests arriving now become durable work items, not errors.
Nothing is lost even though nothing can be produced.
3. STOP RETRYING
Pause the retry workers. A fleet retrying into a recovering
service extends the outage for everybody.
4. COMMUNICATE
"Your document will arrive shortly" if the work is preserved.
"Please try again later" only if it is not.
These are different messages and the pipeline knows which is true.
5. SCHEDULED RUNS
Skip or delay? Decided in advance, by name:
payroll delay, always. Never skip.
statements delay up to 24h, then escalate.
marketing skip.
6. RECOVERY
Everything queued will be attempted at once. Drain at a rate
below normal capacity, oldest first, and watch the error rate
rather than the queue depth.
DECIDES: <name / rota>, not the person who happens to be paged. */
// Step 2, which is the one that has to already exist.
async function generate(spec) {
try {
return await pdf.post("/v1/pdf", spec);
} catch (err) {
if (TRANSIENT.has(err.code)) {
await db.pendingWork.create({ spec: spec.identity, reason: err.code });
throw new Deferred("queued for retry"); // not a lost request
}
throw err;
}
}What people do instead
Letting requests fail during the outage rather than preserving them. Everything else on this page is recoverable afterwards and that is not: once a request has been rejected with no record, the only route back is asking users to do it again, and most of them will not.
What the symptom looks like
The error code distinguishes their problem from yours in the first minute, which is the single most useful thing at the start of an incident and the thing people spend twenty minutes establishing without it.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
Alerting on document generation, including failures that return 200
Choosing signals that catch the failures that matter, which for documents includes several that produce no error at all.
Regenerating documents after an incident without making it worse
Re-running a set of documents that were produced wrongly, which is a recovery operation with its own risks and needs planning before it is needed.
Degrading document generation instead of failing outright
Producing something useful when the full document cannot be made, which is sometimes right and frequently worse than failing.
What to log about a render, and what must never be logged
The fields that make a render diagnosable weeks later, and the one thing that must not be in the log at any level.
Tracing one document from request to delivery
Following a single document across the queue, the render and the delivery, which is the only way to answer where the time went or where it stopped.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.