Operations / Correct under repetition
Handling a batch where some documents succeeded and some did not
The normal outcome of a large batch, which is neither success nor failure, and which most client code has no branch for.
Why the default answer is wrong here
Client code tends to treat a call as succeeding or failing. A batch does neither: some of its requests produce documents and some do not, and the useful information is per item. Code written around a single outcome either treats a partial success as a failure, and retries everything including the parts that worked, or treats it as a success, and silently loses the ones that did not. Both are worse than the partial result.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Read the per-item results rather than the overall status. The response carries a result per request and that is the actual outcome.
- Record state per item immediately, before deciding what to do about the failures, so a crash during the handling does not lose what succeeded.
- Retry only the failed items, as a new smaller batch, and only if their errors are in the transient class.
- Report a partial result as a partial result. A run that produced 1,997 of 2,000 documents should say so rather than reporting success or failure.
- Keep the failures with their reasons, because a batch that failed for three different reasons needs three different responses and an aggregate count hides that.
- Decide whether the run should continue or stop on a given failure rate, and make that a threshold rather than a judgement made at three in the morning.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
const results = await renderBatch(chunk);
// Record first, decide second: a crash while handling failures must not
// lose the ones that worked.
const failures = [];
for (const [i, r] of results.entries()) {
const item = chunk[i];
if (r.error) {
await db.runItems.update(item.id, { state: "failed", error: r.error.code });
failures.push({ item, error: r.error });
} else {
await db.runItems.update(item.id, {
state: "generated",
documentId: r.document_id,
});
}
}
if (failures.length) {
// Group by reason: three different failures need three responses.
const byCode = groupBy(failures, (f) => f.error.code);
log.warn("batch partial", {
ok: results.length - failures.length,
failed: failures.length,
codes: Object.fromEntries(
Object.entries(byCode).map(([c, f]) => [c, f.length]),
),
});
// Retry only the transient ones, only the failed items.
const retryable = failures.filter((f) => TRANSIENT.has(f.error.code));
if (retryable.length) await enqueueRetry(retryable.map((f) => f.item));
// A threshold decided in advance, not at three in the morning.
if (failures.length / results.length > 0.1) await pauseRun(runId);
}What people do instead
Retrying the whole batch when any item failed. Everything that succeeded is generated a second time, which for numbered documents means a second invoice for customers whose first one was fine, and the failure that caused it was in one item.
What the symptom looks like
A failure distribution across codes is the diagnosis. All items failing with the same code is an environmental problem; a scatter of different codes is bad data in the run, and those need different people.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
How many documents to put in one batch request
Choosing how much work goes in a single call, which trades round trips against blast radius and against a real per-plan limit.
Queue design for a payroll run or any large document batch
Structuring a job that produces thousands of documents at once, where the interesting decisions are about failure and progress rather than throughput.
What to do with a document that will never generate
The handling for work that has exhausted its retries, which for documents means somebody is missing something they were promised.
Idempotency for a document that must not be generated twice
Making a repeated request produce the same document rather than a second one, which matters more here than in most APIs because the artefact is often numbered.
Retrying a render safely, and which failures are safe to retry
Deciding which failures should be retried and which should not, because a blanket retry on a document endpoint is a duplication mechanism.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.