Operations / What it costs and what it can take
Watching usage against a quota before it stops the run
Tracking consumption against the allowance so a limit is a planned event rather than a failed monthly run.
Why the default answer is wrong here
A quota is invisible until it is exhausted, and the moment it is exhausted is determined by usage rather than by the calendar, so it lands at an arbitrary and usually inconvenient time. For a workload with a monthly peak, the exhaustion tends to happen during the peak, which is when the documents matter most. Watching the trend converts that from an incident into a purchasing decision made in advance.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Read usage on a schedule rather than waiting for a rejection. The API exposes usage at /v1/usage, with history and export variants for trend analysis.
- Alert on projected exhaustion, not on the current figure. Consumption plus days remaining tells you whether the month completes; the raw number does not.
- Account for the peak in the projection. A linear projection through a month with a large run at the end will predict comfort and be wrong.
- Watch the failure classes too. A rise in permanent errors consumes nothing, but a rise in retries against transient errors consumes allowance for documents that were never produced.
- Know the refund behaviour so the accounting is right: a render that succeeded and then failed a later step refunds the metered unit, so a failure after the render does not silently consume quota.
- Separate usage by key so a runaway job can be identified. Usage by key is available and a single aggregate figure cannot tell you which job caused a jump.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
// Read on a schedule; do not wait for a rejection.
schedule.hourly(async () => {
const usage = await pdf.get("/v1/usage");
const used = usage.used;
const limit = usage.limit;
const dayOfPeriod = daysSince(usage.period_start);
const daysLeft = daysUntil(usage.period_end);
// Projected, not current. And add the known peak rather than
// projecting linearly through it.
const dailyRate = used / Math.max(dayOfPeriod, 1);
const projected = used + dailyRate * daysLeft + expectedPeakVolume(daysLeft);
metrics.gauge("quota.used_fraction", used / limit);
metrics.gauge("quota.projected_fraction", projected / limit);
if (projected > limit * 0.9) {
alert.warn("projected to reach %d%% of quota this period", Math.round(projected / limit * 100));
}
});
/* Two things worth knowing for the accounting:
A render that succeeded and then failed a later step refunds the
metered unit, so a failure after the render does not silently
consume allowance.
Usage is available per key, so a jump can be attributed to a job
rather than being an aggregate nobody can explain. */What people do instead
Projecting linearly through a month that contains a large scheduled run. The projection is comfortable for twenty-five days and the run consumes the remaining allowance in an afternoon, at which point the documents that mattered most are the ones that did not get produced.
What the symptom looks like
A step change in usage that does not correspond to a business event is usually a retry loop against a permanent error, which consumes allowance producing nothing.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
Working out what a generated document actually costs you
Adding up the real per-document cost, which is more than the render and is the number every capacity and caching decision depends on.
Planning capacity for a document peak that arrives once a month
Sizing for a workload that is idle most of the time and enormous occasionally, which is the normal shape for documents and the wrong shape for averages.
Retrying a render safely, and which failures are safe to retry
Deciding which failures should be retried and which should not, because a blanket retry on a document endpoint is a duplication mechanism.
Why the first document after a quiet period is slower
The latency difference between a pipeline that has been busy and one that has been idle, which for a monthly workload means every run starts cold.
Choosing how many documents to render at once
Picking a concurrency that uses the available capacity without exceeding it, which is a measured number rather than a guessed one.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.