Operations / Correct under repetition
Queue design for a payroll run or any large document batch
Structuring a job that produces thousands of documents at once, where the interesting decisions are about failure and progress rather than throughput.
Why the default answer is wrong here
A payroll run is not a throughput problem, it is a completeness problem. Two thousand documents have to be produced and delivered, the run has to be resumable if it dies halfway, it must not produce anything twice if it is restarted, and somebody has to be able to answer how far along it is and which ones failed. A naive loop over the recipients gets the throughput right and every one of those wrong.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Enqueue one message per document, not one for the run. A per-document message is retryable, trackable and resumable; a run-level job is none of those.
- Make each message carry the identity rather than the data, so the worker fetches current data and a redelivery cannot render a stale copy.
- Record per-document state, so progress is a query rather than a guess and a resume is a filter on it.
- Make the worker idempotent, because at-least-once delivery is what queues offer and a redelivery is normal rather than exceptional.
- Separate generation from delivery into two stages with their own state, so a mail outage does not force every document to be regenerated.
- Give the run an identifier and put it on every message, so the whole run can be counted, paused, resumed and audited as one thing.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
// One message per document. Identity only, never the data.
async function startPayrollRun(period) {
const runId = await db.runs.create({ period, state: "running" });
const employees = await db.employees.activeFor(period);
await db.runItems.createMany(
employees.map((e) => ({ runId, employeeId: e.id, state: "pending" })),
);
await queue.sendBatch(
employees.map((e) => ({
body: { runId, employeeId: e.id, period }, // identity, not data
})),
);
return runId;
}
// The worker is idempotent, because at-least-once delivery is normal.
async function handlePayslip({ runId, employeeId, period }) {
const item = await db.runItems.find({ runId, employeeId });
if (item.state === "generated" || item.state === "delivered") return; // redelivery
const data = await db.payslips.build(employeeId, period); // fetched now
const res = await pdf.post("/v1/pdf", {
html: renderPayslip(data),
filename: `payslip-${period}-${employeeId}.pdf`,
store: true,
});
await db.runItems.update(item.id, {
state: "generated",
documentId: res.document_id,
});
// Delivery is a separate stage with its own state, so a mail outage
// does not force everything to be regenerated.
await deliveryQueue.send({ runId, employeeId, documentId: res.document_id });
}
// Progress is a query, not a guess.
// SELECT state, count(*) FROM run_items WHERE run_id = ? GROUP BY stateWhat people do instead
One job for the whole run, looping internally. It works until it dies at document 1,400, at which point there is no record of which ones were done, and the only options are to run it again and duplicate 1,400 documents or to reconstruct the state by hand.
What the symptom looks like
The per-item state table answers every operational question about a run: how far, which failed, why, and what a resume would do. A run without one produces the question and no way to answer it.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
How many documents to put in one batch request
Choosing how much work goes in a single call, which trades round trips against blast radius and against a real per-plan limit.
Handling a batch where some documents succeeded and some did not
The normal outcome of a large batch, which is neither success nor failure, and which most client code has no branch for.
Idempotency for a document that must not be generated twice
Making a repeated request produce the same document rather than a second one, which matters more here than in most APIs because the artefact is often numbered.
Retrying a render safely, and which failures are safe to retry
Deciding which failures should be retried and which should not, because a blanket retry on a document endpoint is a duplication mechanism.
Stopping the same document being requested twice at once
Collapsing concurrent identical requests, which is a different problem from idempotency because both arrive before either has finished.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.