Operations / Knowing what it is doing
Alerting on document generation, including failures that return 200
Choosing signals that catch the failures that matter, which for documents includes several that produce no error at all.
Why the default answer is wrong here
Standard alerting catches errors and latency, and the worst document failures are neither. A render that produced a document with a missing logo, a fallback font or an empty total returns 200 in a normal time. A scheduled run that never started produces no signal whatsoever, because nothing ran to fail. So an alerting setup built from the usual signals will report a healthy pipeline while it is producing unusable documents or none at all.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Alert on the absence of expected work, not only on failed work. A run that should have produced two thousand documents and produced none generates no errors at all.
- Alert on volume anomalies in both directions. A sudden drop is as diagnostic as a spike, and a drop is what a broken upstream looks like.
- Alert on the shape of the output, not just on the outcome: a document whose byte size or page count is far from its historical distribution is probably wrong even though it rendered.
- Alert on the dead letter arrival rate, since each item is somebody who did not get a document.
- Keep permanent-error rate separate from transient. A rise in permanent errors is a code or data change and needs a different responder from a rise in transient ones.
- Route to somebody. An alert on a document pipeline with no owner accumulates alongside the failures it describes.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
// The failures ordinary alerting misses.
// 1. Work that never happened: no error, no log line, nothing.
schedule.daily("03:00", async () => {
const period = previousMonthIn("Europe/London");
if (!(await db.runs.findByPeriod(period))) {
alert.page("payroll run for %s never started", period);
}
});
// 2. A document that rendered fine and is probably wrong.
const stats = await db.documentStats.forKind("invoice"); // rolling baseline
if (res.body.length < stats.p05Bytes || res.body.length > stats.p95Bytes) {
alert.warn("invoice %s is %d bytes, outside the usual range", invoice.id,
res.body.length);
}
if ((res.pages ?? 0) > stats.p99Pages * 2) {
alert.warn("invoice %s is %d pages", invoice.id, res.pages);
}
// 3. Volume in both directions: a drop is what a broken upstream looks like.
metrics.gauge("documents.generated.hourly", count, { kind: "invoice" });
// alert if count < 0.4 * same_hour_last_week
// alert if count > 3.0 * same_hour_last_week
// 4. Permanent errors separated from transient: different responders.
metrics.increment("render.failed", { class: PERMANENT.has(code) ? "permanent" : "transient" });What people do instead
Alerting only on the error rate. It is the signal most likely to be flat during the worst incidents this pipeline has, which are a run that did not happen and a run that produced a thousand documents with a missing stylesheet.
What the symptom looks like
A byte-size distribution that shifts sharply is the earliest available warning that something structural changed, usually a stylesheet or a font that stopped loading. It fires before anybody looks at a document.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
Scheduling a monthly document run so it survives being late
Running a document job on a calendar, where the interesting failures are a run that did not happen and a run that happened twice.
What to do with a document that will never generate
The handling for work that has exhausted its retries, which for documents means somebody is missing something they were promised.
What to do when document generation stops working
The decisions to make when rendering is unavailable, most of which are about what to tell people and what to preserve rather than about restoring service.
What to log about a render, and what must never be logged
The fields that make a render diagnosable weeks later, and the one thing that must not be in the log at any level.
Tracing one document from request to delivery
Following a single document across the queue, the render and the delivery, which is the only way to answer where the time went or where it stopped.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.