Testing documents / Running it in CI
Load testing document generation without inventing numbers
Measuring what your pipeline does under the load it will actually see, which for document generation is almost never a steady rate.
Why the obvious approach does not work
Document load is bursty in a way that a steady-rate load test does not represent. A payroll run generates every document in the company in a few minutes and nothing for a month. An invoicing cycle fires on the first of the month. A statement run fires overnight. Testing at an average rate measures a situation that never occurs, and the interesting question, what happens when the burst exceeds capacity, is exactly the one a steady test avoids.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Model the real shape: a burst of the real size, at the real concurrency, not a sustained average.
- Measure the whole path rather than the render call, because the queue, the storage write and the delivery step are all part of the burst and one of them is the actual limit.
- Record percentiles rather than a mean. The mean of a bursty workload describes nothing, and the tail is what a person waiting for a document experiences.
- Deliberately exceed the limit and observe the behaviour, because that is the case the system will meet and the one nobody has seen. A 429 carries a Retry-After header, and what your client does with it is the thing under test.
- Run it against an environment you are allowed to saturate, with its own key, and tell whoever needs to know before you start.
- Write down the numbers you measured with the date and the configuration, because a load result with no context is quoted for years after it stopped being true.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
// A burst, not a rate. This is what a payroll run looks like.
export const options = {
scenarios: {
payroll_run: {
executor: "shared-iterations",
vus: 20, // real concurrency
iterations: 2400, // every document, at once
maxDuration: "10m",
},
},
thresholds: {
// Percentiles, because the mean of a burst describes nothing.
"http_req_duration{status:200}": ["p(95)<8000", "p(99)<15000"],
"checks": ["rate>0.999"],
},
};
export default function () {
const res = http.post(`${__ENV.BASE}/v1/pdf`, JSON.stringify({
html: payslipHtml(__ITER),
options: { format: "A4" },
}), { headers: { Authorization: `Bearer ${__ENV.KEY}` } });
// The case worth testing: what happens past the limit.
if (res.status === 429) {
const wait = Number(res.headers["Retry-After"] ?? 1);
sleep(wait); // honour it; do not hammer
return;
}
check(res, { "rendered": (r) => r.status === 200 && r.body.length > 1000 });
}
/* Record with the result: the date, the concurrency, the document
size, and which environment. A number without those is quoted for
years after it stopped being true. */What people do instead
Load testing at a steady rate because that is what the tool does by default. It produces a reassuring graph of a situation that never happens, and it never enters the region where the system's behaviour actually becomes interesting.
What a failure actually tells you
The failure mode at saturation is the result, more than any number. Whether the system queues, sheds, retries politely or falls over is the thing you learned, and it is worth writing down next to the percentiles.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Which document tests run on every commit and which run nightly
Splitting the suite by cost so the fast tier fits in a pull request, because a test that reports a day late reports to nobody.
Testing what happens when a render fails, not just when it works
Exercising the failure branches of a document pipeline, which are the least-tested code in most systems and the code that runs on the worst day.
Keeping a document suite fast enough to stay in the pull request
The specific levers that reduce the cost of a rendering suite, which are different from the ones that speed up ordinary tests.
Running document tests in CI without a browser in the image
Testing a rendering pipeline on a runner that has no renderer, which is the normal situation once rendering is a service rather than a dependency.
Catching pagination mistakes without rendering the document
Asserting on the rules that govern page behaviour rather than on the pages themselves, which is instant and catches the omissions rather than the outcomes.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.