Operations / Correct under repetition
Receiving a callback when a document is ready, and verifying it
Being told when work finished instead of asking repeatedly, which removes polling and adds an endpoint that has to be correct about who is calling it.
Why the default answer is wrong here
A callback is an inbound request from outside your system to an endpoint that changes your state, which makes it a different security proposition from an outbound call. It also arrives with the usual distributed-systems properties: it can be delivered more than once, it can arrive out of order relative to your own writes, and it can arrive for work you no longer care about. Handlers written as if it were a private, once-only, in-order message are the norm and each of those assumptions fails eventually.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Verify the callback is genuine before acting on it. The batch endpoint accepts a webhook_secret of at least sixteen characters, and a handler that does not check the resulting signature is an endpoint anyone can call.
- Make the handler idempotent, because a redelivery is normal. Key on the document or job identity and make a repeat a no-op.
- Treat the payload as a notification rather than as data. Fetch what you need rather than trusting values in an inbound request to update your records.
- Return quickly and do the work asynchronously. A slow handler causes retries, which causes more handlers, which is a self-inflicted load spike.
- Handle arrival before your own write completed, which happens when the render finishes faster than your transaction commits. The handler has to tolerate not finding the record yet.
- Give the endpoint a public https URL. Both callback_url and webhook_url are rejected if they resolve to a private or reserved address, which is a deliberate protection against being used to reach inside a network.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
app.post("/hooks/documents", async (req, res) => {
// 1. Verify before acting. Without this it is an open endpoint.
if (!verifySignature(req.rawBody, req.get("signature"), process.env.DOC_WEBHOOK_SECRET)) {
return res.sendStatus(401);
}
// 2. Return fast: a slow handler causes retries, which cause more
// handlers, which is a self-inflicted load spike.
res.sendStatus(204);
// 3. The rest is asynchronous and idempotent.
queue.send({ kind: "document-ready", payload: req.body });
});
async function onDocumentReady({ job_id, document_id }) {
const item = await db.runItems.findByJobId(job_id);
// Arrived before our own write committed: normal, and recoverable.
if (!item) return retryLater({ job_id, document_id });
// Redelivery: normal, and a no-op.
if (item.documentId === document_id) return;
await db.runItems.update(item.id, { state: "generated", documentId: document_id });
}
/* The payload is a notification, not data. Fetch what you need rather
than trusting an inbound request to set your records. */What people do instead
Acting on the payload without verifying the signature. The endpoint is public, its shape is guessable, and it changes state, so anyone who finds it can mark documents as delivered. Nothing about this fails visibly, which is why it survives.
What the symptom looks like
Duplicate handling attempts in the logs are expected and fine if the handler is idempotent. What is not fine is a handler that errors on the second delivery, which means the idempotency is missing rather than the delivery being wrong.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
Generating a document in the request or in the background
Whether the user waits for the document, which is a decision about the document's size and the user's expectation rather than about performance.
Retrying a render safely, and which failures are safe to retry
Deciding which failures should be retried and which should not, because a blanket retry on a document endpoint is a duplication mechanism.
Idempotency for a document that must not be generated twice
Making a repeated request produce the same document rather than a second one, which matters more here than in most APIs because the artefact is often numbered.
Stopping the same document being requested twice at once
Collapsing concurrent identical requests, which is a different problem from idempotency because both arrive before either has finished.
Caching a rendered document, and when it is safe to serve one
Serving a previously rendered document instead of making a new one, which is nearly free when the inputs are unchanged and wrong when they are not.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.