Operations / Knowing what it is doing
Regenerating documents after an incident without making it worse
Re-running a set of documents that were produced wrongly, which is a recovery operation with its own risks and needs planning before it is needed.
Why the default answer is wrong here
An incident that produced wrong documents has a recovery step, and it is more dangerous than the incident. The affected set has to be identified precisely, since regenerating too many produces duplicates for customers who were fine. The regenerated documents may need to be delivered, which means people receive a second copy of something. And the pipeline is being asked to do a large burst of work at short notice, which is the same load profile that causes incidents. Doing this under pressure without a plan is how one incident becomes two.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Identify the affected set by a query against your own records, and check the count before doing anything. If the count is a surprise, the query is wrong.
- Decide whether documents are replaced or reissued. Replacing keeps one identity and reissuing creates a second, and for a numbered document that decision has consequences downstream.
- Decide whether to redeliver, separately from whether to regenerate. Often the right answer is to regenerate everything and redeliver only where somebody already opened the wrong one.
- Rate limit the reprocess below normal capacity rather than at it, because the pipeline is also doing its ordinary work and the recovery must not cause a second outage.
- Record the reprocess as its own run with its own identifier, so the regenerated documents are distinguishable from the originals afterwards.
- Write the runbook before the incident. Every step here is obvious in the abstract and none of them are obvious at two in the morning.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
// 1. Identify precisely, and look at the count before acting.
const affected = await db.documentEvents.find({
event: "generated",
templateVersion: "v43", // the broken one
at: { gte: "2026-03-14T08:00:00Z", lt: "2026-03-14T11:20:00Z" },
});
log.info("reprocess candidates: %d", affected.length); // surprise = wrong query
// 2. Its own run, so the regenerated set is distinguishable afterwards.
const runId = await db.runs.create({
kind: "reprocess",
reason: "INC-482 stylesheet not loaded",
templateVersionFrom: "v43",
});
// 3. Below normal capacity: the pipeline is also doing its day job.
const limiter = new RateLimiter({ perSecond: 2 });
for (const item of affected) {
await limiter.wait();
await queue.send({
runId,
subjectId: item.subjectId,
reprocess: true,
deliver: await shouldRedeliver(item), // separate decision
});
}
/* Separate decisions, made deliberately:
regenerate? almost always yes
replace or reissue? replacing keeps one identity; reissuing makes
a second numbered document, which downstream
systems will see
redeliver? often only where the wrong one was already opened */What people do instead
Reprocessing at full speed to get it over with. The recovery burst lands on a pipeline already doing its normal work, capacity is exceeded, and the incident extends into a second one caused by the fix.
What the symptom looks like
A candidate count that does not match expectations is the stop signal. It means the query is selecting the wrong set, and running it would produce duplicates for customers who were never affected.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
Answering what happened to one document three weeks later
Keeping enough operational history that a specific document's story can be reconstructed, which log retention alone usually cannot do.
What to do with a document that will never generate
The handling for work that has exhausted its retries, which for documents means somebody is missing something they were promised.
What to do when document generation stops working
The decisions to make when rendering is unavailable, most of which are about what to tell people and what to preserve rather than about restoring service.
What to log about a render, and what must never be logged
The fields that make a render diagnosable weeks later, and the one thing that must not be in the log at any level.
Tracing one document from request to delivery
Following a single document across the queue, the render and the delivery, which is the only way to answer where the time went or where it stopped.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.