Testing documents / Running it in CI
Keeping a document suite fast enough to stay in the pull request
The specific levers that reduce the cost of a rendering suite, which are different from the ones that speed up ordinary tests.
Why the obvious approach does not work
The usual advice for slow suites, parallelise and mock, applies awkwardly here. Parallelising renders multiplies concurrent load against a shared service rather than using idle local cores, and mocking the render removes the thing under test. The levers that work are specific: render fewer times, render several documents in one call, and reuse a rendered document across the assertions that examine it.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Render once per fixture and run every assertion against that one document. Suites routinely re-render for the page count, then again for the text, then again for the images.
- Batch the renders into one call. POST /v1/pdf/batch takes a requests array, so a suite needing eight documents makes one round trip rather than eight.
- Cache rendered fixtures keyed by a hash of the template, the fixture and the options, so a run that changed nothing about a document reuses the last one.
- Cut the fixture matrix rather than the assertions. Fewer fixtures with more assertions each is faster and catches more than many fixtures with one assertion each.
- Keep the no-render tier genuinely free of rendering, and check periodically that nothing crept in, because one accidental render in a loop is the usual cause of a suite that got slow without anybody changing it.
- Measure before optimising. The slow part is nearly always the number of round trips rather than the assertions.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
// Render once, assert many times. The common waste is re-rendering
// per assertion.
describe("invoice, many lines", () => {
let pdf, text, pages;
beforeAll(async () => {
pdf = await renderFixtureCached("invoice", "many-lines");
text = await extractText(pdf);
pages = await pageCount(pdf);
});
test("paginates", () => expect(pages).toBe(4));
test("has the total", () => expect(text).toContain("5,016.00"));
test("has no holes", () => expect(text).not.toContain("{{"));
test("repeats the header", async () =>
expect(await extractText(pdf, { page: 2 })).toContain("Description"));
});
// One round trip for the whole fixture set.
async function renderAll(fixtures) {
const res = await client.post("/v1/pdf/batch", {
requests: fixtures.map((f) => ({ html: f.html, options: f.options })),
});
return res.results;
}
// Cache on the inputs, so an unchanged document is not re-rendered.
const key = hash([templateVersion, fixtureName, JSON.stringify(options)]);What people do instead
Parallelising the renders to make the suite faster. It moves the cost rather than removing it, it can hit a rate limit and turn a slow suite into a failing one, and the actual win was available from rendering each fixture once instead of four times.
What a failure actually tells you
A suite that got slower without new tests being added usually gained a render inside a loop or lost its fixture cache. Counting the render calls in a run finds it immediately.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Which document tests run on every commit and which run nightly
Splitting the suite by cost so the fast tier fits in a pull request, because a test that reports a day late reports to nobody.
Running document tests in CI without a browser in the image
Testing a rendering pipeline on a runner that has no renderer, which is the normal situation once rendering is a service rather than a dependency.
Making a failed document test diagnosable without a local rerun
What a failure has to emit so somebody can act on it from the CI log, since reproducing a document failure locally is expensive and often impossible.
Catching pagination mistakes without rendering the document
Asserting on the rules that govern page behaviour rather than on the pages themselves, which is instant and catches the omissions rather than the outcomes.
Testing the shape of the data a template expects
Asserting that the data reaching a template has the fields the template reads, which catches the failure that produces a document full of blanks.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.