Operations / What it costs and what it can take
How long to keep a generated document, and who decides
Deciding what happens to a document after it is delivered, which is a policy question with a cost attached and a legal floor underneath it.
Why the default answer is wrong here
Storage decisions default to keeping everything forever, because deleting feels risky and storage is cheap per unit. Over a long retention the accumulated cost is not small, and forever is rarely the correct answer anyway: some documents have a legal minimum retention, some have a maximum beyond which holding them is a liability, and some are regenerable and need not be kept at all. Those are three different policies and a single default serves none of them.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Classify documents by whether they can be regenerated. A document that is a pure function of data you still hold does not need storing, only its inputs do.
- Find the legal floor before setting anything, because that is a constraint rather than a preference, and it differs by document kind and by jurisdiction.
- Set a maximum as well as a minimum where the document contains personal data, since holding it longer than necessary is its own exposure.
- Record the expiry on your own record rather than only in the storage layer. When storing through the API, the response carries document_expires, and a system that does not persist it cannot tell a user when a link stops working.
- Make deletion verifiable. A retention policy nobody has confirmed is running is a policy in a document rather than in the system.
- Keep the metadata after the document expires. Knowing an invoice existed and was delivered is useful long after the file itself needs to be available.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
// Three policies, not one default.
const RETENTION = {
// Legal floor: cannot delete before this.
invoice: { minDays: 2555, maxDays: null, regenerable: true },
// Personal data: holding it longer is exposure, not safety.
payslip: { minDays: 2190, maxDays: 2555, regenerable: true },
// A view of live data: no reason to keep it at all.
export: { minDays: 0, maxDays: 7, regenerable: true },
};
const res = await pdf.post("/v1/pdf", { html, store: true });
// Persist the expiry: without it, nobody can tell a user when the link
// stops working.
await db.documents.create({
subjectId: invoice.id,
documentId: res.document_id,
expiresAt: res.document_expires,
});
/* Regenerable is the question that decides most of this. If the
document is a pure function of data you still hold and a template
version you can still check out, storing the inputs is cheaper than
storing the output, and the output can be reproduced on request.
The trade is honesty about "still hold": a template that has been
deleted or a dependency that has moved makes a document
unreproducible, and for anything with legal weight the safe answer
is to keep the file rather than the recipe. */
// Verifiable, not aspirational.
schedule.daily(async () => {
const overdue = await db.documents.countExpiredNotDeleted();
if (overdue > 0) alert.warn("%d documents past expiry not deleted", overdue);
});What people do instead
Keeping everything forever because storage is cheap. It accumulates, the cost compounds over a long retention, and for documents containing personal data it converts a manageable holding into a growing liability that nobody decided to take on.
What the symptom looks like
Storage growing at the rate documents are created, with no corresponding deletion rate, means the expiry policy is not actually running whatever the configuration says.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
Working out what a generated document actually costs you
Adding up the real per-document cost, which is more than the render and is the number every capacity and caching decision depends on.
Answering what happened to one document three weeks later
Keeping enough operational history that a specific document's story can be reconstructed, which log retention alone usually cannot do.
Caching a rendered document, and when it is safe to serve one
Serving a previously rendered document instead of making a new one, which is nearly free when the inputs are unchanged and wrong when they are not.
Planning capacity for a document peak that arrives once a month
Sizing for a workload that is idle most of the time and enormous occasionally, which is the normal shape for documents and the wrong shape for averages.
Why the first document after a quiet period is slower
The latency difference between a pipeline that has been busy and one that has been idle, which for a monthly workload means every run starts cold.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.