Operations / Correct under repetition
What to do with a document that will never generate
The handling for work that has exhausted its retries, which for documents means somebody is missing something they were promised.
Why the default answer is wrong here
A dead letter queue in most systems is a place messages go to be looked at eventually. For documents it is different, because each message that lands there represents a person who did not get their payslip, their invoice or their statement. The queue is not a debugging aid, it is a list of unmet obligations, and it needs an owner and a deadline rather than a dashboard nobody reads.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Route exhausted items to a place with an owner, and make the owner a person or a rota rather than a queue name.
- Keep the failure reason with the item. A dead letter with no error is a message somebody has to reproduce before they can act.
- Distinguish poison items from environmental failures. One document failing repeatedly while everything else succeeds is bad data; everything failing is an outage and the queue is a symptom.
- Make reprocessing a supported action rather than a manual re-enqueue, because the manual version skips the idempotency the normal path has.
- Alert on the arrival rate, not on the depth. A queue with four items that has had four items for a week is a different problem from one that just received four hundred.
- Set a business deadline for clearing it, because the items are obligations. A payslip that arrives a week late has already caused its damage.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
// A dead letter here is a person who did not get their document, so it
// carries enough to act on and enough to reprocess safely.
async function toDeadLetter(item, error, attempts) {
await db.deadLetters.create({
runId: item.runId,
itemId: item.id,
subjectId: item.employeeId, // who is affected
errorCode: error.code,
errorMessage: error.message,
attempts,
firstFailedAt: item.firstFailedAt,
lastFailedAt: new Date(),
});
// Arrival rate, not depth: four items for a week is a different
// problem from four hundred in a minute.
metrics.increment("documents.dead_letter", { code: error.code });
}
// Reprocessing is a supported action, so it goes through the same
// idempotent path rather than a manual re-enqueue that skips it.
async function reprocessDeadLetter(id, { by }) {
const dl = await db.deadLetters.find(id);
await audit.record({ action: "reprocess", deadLetterId: id, by });
await db.runItems.update(dl.itemId, { state: "pending" });
await queue.send({ runId: dl.runId, employeeId: dl.subjectId });
await db.deadLetters.update(id, { reprocessedAt: new Date(), reprocessedBy: by });
}
/* Poison against outage: one item failing while others succeed is bad
data and needs a person. Everything failing is an outage and the
queue is a symptom of it, not the thing to work on. */What people do instead
Treating the dead letter queue as a technical backlog. It gets a dashboard, no owner and no deadline, and the items in it are people waiting for documents. The first real symptom is a support ticket rather than an alert.
What the symptom looks like
A single subject appearing repeatedly is bad data for that subject. Many subjects arriving at once is an upstream failure, and the queue depth is telling you about the outage rather than about the documents.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
Handling a batch where some documents succeeded and some did not
The normal outcome of a large batch, which is neither success nor failure, and which most client code has no branch for.
Retrying a render safely, and which failures are safe to retry
Deciding which failures should be retried and which should not, because a blanket retry on a document endpoint is a duplication mechanism.
Regenerating documents after an incident without making it worse
Re-running a set of documents that were produced wrongly, which is a recovery operation with its own risks and needs planning before it is needed.
Idempotency for a document that must not be generated twice
Making a repeated request produce the same document rather than a second one, which matters more here than in most APIs because the artefact is often numbered.
Stopping the same document being requested twice at once
Collapsing concurrent identical requests, which is a different problem from idempotency because both arrive before either has finished.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.