Operations / Knowing what it is doing
Answering what happened to one document three weeks later
Keeping enough operational history that a specific document's story can be reconstructed, which log retention alone usually cannot do.
Why the default answer is wrong here
Support questions about documents arrive late and specifically. Somebody asks about one invoice from six weeks ago: was it generated, when, with which template, was it delivered, to what address, and did anyone regenerate it. Logs are typically retained for less time than that and are structured for searching by time rather than by document. So the question is answerable in principle and not in practice, and the answer that gets given is a guess.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Keep a durable per-document record in your own database, separate from logs, containing what happened rather than what was logged.
- Record the events rather than the current state: generated, stored, delivered, failed, regenerated, each with a timestamp. State alone cannot answer how it got there.
- Record the template version and the option set on the record, because that is the question most often asked and the one logs lose first.
- Record who or what triggered it, including whether it was a scheduled run, a user action or a reprocess, since a regenerated document has a different explanation from an original.
- Keep it for as long as the document's own retention at minimum, because a question about a document is only answerable while the document exists.
- Keep the content out of it, for the same reason as logging: this record will outlive the document and it should not become a second copy of it.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
// Events, not state. State cannot answer how it got there.
await db.documentEvents.create({
documentKind: "invoice",
subjectId: invoice.id,
event: "generated",
at: new Date(),
templateVersion: TEMPLATE_VERSION,
options: { format: "A4" },
triggeredBy: { type: "scheduled_run", runId }, // or user, or reprocess
documentId: res.document_id,
correlationId,
});
/* The question this answers, six weeks later, in one query:
SELECT event, at, template_version, triggered_by
FROM document_events
WHERE document_kind = 'invoice' AND subject_id = ?
ORDER BY at;
generated 14 Mar 09:02 v42 scheduled_run:mar-2026
stored 14 Mar 09:02 v42
delivered 14 Mar 09:03 v42 to a@example.com
regenerated 02 Apr 11:40 v44 user:someone@company
delivered 02 Apr 11:40 v44 to a@example.com
Two versions, two deliveries, and the customer has both. That is the
answer, and it is not reachable from logs six weeks later.
No content on this record: it outlives the document and must not
become a second copy of it. */What people do instead
Relying on log retention. Logs are kept for a shorter period than documents, are indexed by time rather than by document, and are usually sampled, so the specific question about the specific file is exactly the one they cannot answer.
What the symptom looks like
Two generation events with different template versions for one document is the explanation for most support questions of the form why does mine look different from theirs.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
What to log about a render, and what must never be logged
The fields that make a render diagnosable weeks later, and the one thing that must not be in the log at any level.
Regenerating documents after an incident without making it worse
Re-running a set of documents that were produced wrongly, which is a recovery operation with its own risks and needs planning before it is needed.
How long to keep a generated document, and who decides
Deciding what happens to a document after it is delivered, which is a policy question with a cost attached and a legal floor underneath it.
Tracing one document from request to delivery
Following a single document across the queue, the render and the delivery, which is the only way to answer where the time went or where it stopped.
Measuring document latency in percentiles rather than averages
Recording the distribution rather than the mean, because document render times are skewed by document size and the tail is what people experience.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.