Operations / What it costs and what it can take
Planning capacity for a document peak that arrives once a month
Sizing for a workload that is idle most of the time and enormous occasionally, which is the normal shape for documents and the wrong shape for averages.
Why the default answer is wrong here
Document workloads are not steady. Invoicing fires on the first, payroll at month end, statements overnight, and nothing much in between. Planning against average throughput produces a system that is idle for twenty-nine days and fails on the thirtieth. Planning against the peak produces one that is expensive and idle. The useful question is not how much capacity, it is how long the peak is allowed to take, and that has a business answer.
The decisions
Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.
- Start from the deadline, not the throughput. Two thousand payslips in four hours and in fifteen minutes are different systems, and only the business can say which is required.
- Measure the actual peak from history rather than estimating it, including the peak of peaks, which is usually a month end that coincided with something else.
- Plan for the peak growing with the business, and know which multiple of today's peak the current design stops working at.
- Spread the peak where the deadline allows. Starting a run earlier is free and it is the cheapest capacity available.
- Separate the peak workload from interactive traffic, so the monthly run cannot degrade the experience of somebody waiting for a receipt.
- Check the whole path, because the constraint is often not the renderer. Delivery, storage or your own database can be the limit and adding render capacity will not move it.
In practice
A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.
/* Start from the deadline, because it is the only input the business
can actually give you.
2,000 payslips, must be delivered by 09:00
run starts 05:00 -> 4 hours
2,000 / 14,400s = 0.14 documents/sec -> trivial
run starts 08:45 -> 15 minutes
2,000 / 900s = 2.2 documents/sec -> a real number
Same work, two different systems, and the difference is a start time
rather than any technical decision. Starting earlier is the cheapest
capacity available. */
// Measure the real peak from history, including the peak of peaks.
// SELECT date_trunc('hour', at) h, count(*)
// FROM document_events WHERE event = 'generated'
// GROUP BY h ORDER BY count(*) DESC LIMIT 10;
// Know the multiple at which the current design stops working.
const HEADROOM = {
measuredPeakPerHour: 2400,
designedForPerHour: 6000,
multiple: 2.5, // review when the business grows past this
};
/* And check the whole path: the constraint is frequently not the
renderer. Time each stage during a real peak before adding capacity
to the stage you assumed was the problem. */What people do instead
Sizing on average throughput. The system is comfortable for twenty-nine days, which is reassuring, and the thirtieth is the only day it is asked to do anything, which is the day it does not.
What the symptom looks like
The ratio between peak and average is the number that describes this workload. If it is above about ten, capacity planning against the average is meaningless and the deadline is the only useful input.
Frequently asked
Is this worth doing for a small volume of documents?
Some of it, and the cheap parts are the ones that matter. Allocating a document's identity before rendering it costs nothing and prevents duplicates forever. Logging the template version costs one field and answers most support questions. Queue design, backpressure and capacity planning are genuinely for scale and can wait until there is some.
Why is duplication treated as more serious than latency here?
Because a document usually carries an identity. A duplicated read is harmless and a duplicated invoice is a second numbered document for one event, which somebody has to reconcile by hand and which may already have been sent. That asymmetry is why the correctness half of this cluster is larger than it would be for most APIs.
How does this relate to the troubleshooting pages?
Those start from a symptom you are looking at right now and work back to a cause. These start from a decision made before the symptom exists. The two meet in the middle: a decision skipped here usually appears there some months later as a problem with no obvious explanation.
Related operational topics
The decisions that depend on each other, then the rest of the same group.
Choosing how many documents to render at once
Picking a concurrency that uses the available capacity without exceeding it, which is a measured number rather than a guessed one.
Backpressure when documents are requested faster than they render
What the system does when demand exceeds capacity, which is a decision to make rather than a behaviour to discover.
Working out what a generated document actually costs you
Adding up the real per-document cost, which is more than the render and is the number every capacity and caching decision depends on.
Why the first document after a quiet period is slower
The latency difference between a pipeline that has been busy and one that has been idle, which for a monthly workload means every run starts cold.
How long to keep a generated document, and who decides
Deciding what happens to a document after it is delivered, which is a policy question with a cost attached and a legal floor underneath it.
Every operational topic
The full list, grouped by correctness, visibility and cost.
What this API actually does
The options and endpoints these decisions are built on, one page each.
Most of these decisions are cheaper to make before the first production run than after the first incident, and none of them need a large system to be worth making.