Operations / Correct under repetition
Stopping the same document being requested twice at once
Collapsing concurrent identical requests, which is a different problem from idempotency because both arrive before either has finished.
Why the default answer is wrong here
Idempotency handles a repeat that arrives after the first completed. It does not handle two that arrive together, which is the common case in practice: a user double-clicks, a page component mounts twice, two workers pick up the same message. Both requests check for an existing document, both find none, and both proceed. The result is two renders, two charges against a quota, and possibly two documents, from a mechanism that was supposed to prevent exactly that.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Take a lock keyed on the operation, not on the request, and hold it across the check and the write. Without that, the check and the write have a gap and the gap is where duplication lives.
- Let the second caller wait for the first result rather than doing the work again, which is both cheaper and produces one document rather than two.
- Set a lock timeout longer than the slowest render you expect, because a lock that expires mid-render reintroduces the problem it exists to solve.
- Handle the case where the lock holder died: the lock expires, the second caller proceeds, and that is correct as long as the identity was allocated first.
- Deduplicate at the boundary rather than in every call site. A single function that owns document generation is the place for this.
- Distinguish this from caching. Deduplication is about concurrent work, caching is about repeated work over time, and a system usually wants both.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
// Two concurrent requests for the same document produce one render.
const inflight = new Map(); // process-local
async function getOrCreateDocument(orderId) {
const key = `invoice:${orderId}`;
// Same process, same moment: share the promise.
if (inflight.has(key)) return inflight.get(key);
const work = (async () => {
// Across processes: a lock held over the check and the write.
const lock = await locks.acquire(key, { ttl: 120_000 }); // > slowest render
try {
const existing = await db.invoices.findByOrderId(orderId);
if (existing?.documentId) return existing.documentId;
return await generateInvoice(orderId);
} finally {
await lock.release();
}
})();
inflight.set(key, work);
try {
return await work;
} finally {
inflight.delete(key);
}
}
/* The TTL matters: a lock that expires while a render is still running
lets a second caller start, which is the duplication this exists to
prevent. Set it above the slowest render you have measured, not the
average one. */What people do instead
Checking for an existing document and then creating one, with no lock between. It is correct in a test, where requests arrive one at a time, and wrong under any real concurrency. The gap is usually milliseconds and a double-click is faster than that.
What the symptom looks like
Two renders in the metrics for one order, seconds apart, is the fingerprint. It looks like a retry and it is not: a retry follows a failure, and these two both succeeded.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
Idempotency for a document that must not be generated twice
Making a repeated request produce the same document rather than a second one, which matters more here than in most APIs because the artefact is often numbered.
Caching a rendered document, and when it is safe to serve one
Serving a previously rendered document instead of making a new one, which is nearly free when the inputs are unchanged and wrong when they are not.
Retrying a render safely, and which failures are safe to retry
Deciding which failures should be retried and which should not, because a blanket retry on a document endpoint is a duplication mechanism.
What belongs in the cache key for a rendered document
The full list of inputs a rendered document depends on, which is longer than the data and is the reason document caches serve stale output.
Queue design for a payroll run or any large document batch
Structuring a job that produces thousands of documents at once, where the interesting decisions are about failure and progress rather than throughput.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.