Operations / Correct under repetition
Scheduling a monthly document run so it survives being late
Running a document job on a calendar, where the interesting failures are a run that did not happen and a run that happened twice.
Why the default answer is wrong here
Scheduled document runs fail in ways that ordinary jobs do not, because they are tied to a period rather than to a moment. A run that misses its window still has to happen, and running it late is usually correct. A run that fires twice must not produce two sets of documents. And the period boundary is a real hazard: a monthly run started at midnight on the first, in the wrong timezone, can select the wrong month's data and produce a complete, correct-looking, entirely wrong set of documents.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Key the run on the period rather than on the firing time, so a late run produces the right documents and a repeat is detectable.
- Make starting a run for a period that already has one an error rather than a second run.
- Compute the period boundaries in an explicit timezone, never the server's, because a scheduler in one zone and a business in another disagree about which month it is.
- Alert on a run that did not start, which requires expecting it. A missing run produces no error and no log line, so nothing catches it unless something is looking for it.
- Make the run resumable and re-runnable for a period, since the most common recovery is running it again after fixing data.
- Test the boundary explicitly: the last day of a month, the first, and a year end. Those are the cases where the period arithmetic is wrong.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
// Keyed on the period, not the firing time. A late run is correct; a
// second run for the same period is not.
async function startMonthlyRun(now = new Date()) {
const period = previousMonthIn("Europe/London", now); // explicit zone
const existing = await db.runs.findByPeriod(period);
if (existing) {
throw new AlreadyRun(`payroll run for ${period} already exists (${existing.id})`);
}
return startPayrollRun(period);
}
// A missing run produces no error and no log line, so something has to
// be looking for it.
async function checkRunHappened() {
const period = previousMonthIn("Europe/London", new Date());
const run = await db.runs.findByPeriod(period);
if (!run) {
alert.page("payroll run for %s has not started", period);
} else if (run.state === "running" && olderThan(run.startedAt, hours(6))) {
alert.page("payroll run for %s has been running for six hours", period);
}
}
/* The boundary cases to test, because this is where period arithmetic
goes wrong and produces a complete, plausible, wrong set:
a run firing at 00:30 on the first, from a server in another zone
a year end
a month with 28, 30 and 31 days */What people do instead
Using the firing time to derive the period. A run that fires a few minutes after midnight from a server in a different timezone selects the wrong month, produces a complete set of documents for it, and nothing about the output looks wrong until somebody reads one.
What the symptom looks like
The absence of a run is the failure that needs its own detector, because every other failure here produces something to look at and this one produces nothing.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
Queue design for a payroll run or any large document batch
Structuring a job that produces thousands of documents at once, where the interesting decisions are about failure and progress rather than throughput.
Alerting on document generation, including failures that return 200
Choosing signals that catch the failures that matter, which for documents includes several that produce no error at all.
Idempotency for a document that must not be generated twice
Making a repeated request produce the same document rather than a second one, which matters more here than in most APIs because the artefact is often numbered.
Retrying a render safely, and which failures are safe to retry
Deciding which failures should be retried and which should not, because a blanket retry on a document endpoint is a duplication mechanism.
Stopping the same document being requested twice at once
Collapsing concurrent identical requests, which is a different problem from idempotency because both arrive before either has finished.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.