Testing documents / Making two runs comparable
Fixtures at the page boundary, where pagination bugs actually live
Deliberately constructed data that lands content exactly at a page break, which is the only way to test the behaviour that breaks most often.
Why the obvious approach does not work
Pagination defects live in a window one row wide. A totals block splits when the line items finish four lines from the bottom, and not otherwise. A table header stops repeating when the table crosses a boundary, and a short table never does. A block that should not break only breaks when it lands across a break. Ordinary fixtures are comfortably short or comfortably long and never enter the window, so the suite passes and the defect ships to whichever customer happens to order the right number of things.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Construct fixtures that land at the boundary rather than hoping. Count the rows that fit and build one fixture at that count and one at one more.
- Recount when the layout changes. The boundary moves with the font, the margins and the row height, so a boundary fixture is a maintained thing rather than a written-once thing.
- Cover both directions: content that just fits, and content that just does not. The failures are different.
- Add a boundary fixture for each block that must not split: the totals, the signature, the figure with its caption. Each has its own window.
- Assert the page count on these more strictly than anywhere else, because the count is the whole point of the fixture.
- Keep a fixture that crosses three boundaries, not two, because a header-repeat defect can hide on page two and show on page three.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
// The boundary is a property of the layout, so derive it rather than
// hard-coding it, and let the test tell you when it moves.
const ROWS_ON_FIRST_PAGE = 28; // recount when the layout changes
test("a table that exactly fills page one does not spill", async () => {
const pdf = await render(invoice({ rows: rows(ROWS_ON_FIRST_PAGE) }));
expect(await pageCount(pdf)).toBe(1);
});
test("one row more moves to a second page and keeps the header", async () => {
const pdf = await render(invoice({ rows: rows(ROWS_ON_FIRST_PAGE + 1) }));
expect(await pageCount(pdf)).toBe(2);
const page2 = await extractText(pdf, { page: 2 });
expect(page2).toContain("Description"); // thead repeated
});
test("the totals block does not split", async () => {
// The window: items finishing a few lines from the bottom.
for (let n = ROWS_ON_FIRST_PAGE - 4; n <= ROWS_ON_FIRST_PAGE; n += 1) {
const pdf = await render(invoice({ rows: rows(n) }));
const last = await extractText(pdf, { page: await pageCount(pdf) });
expect(last).toContain("Subtotal");
expect(last).toContain("Total due"); // both on the same page
}
});What people do instead
Writing the boundary fixture once and never recounting. The boundary is a consequence of the font, the margins and the row height, so a change to any of them moves it, the fixture stops being at the boundary, and the test keeps passing while covering nothing.
What a failure actually tells you
A boundary failure is almost always a real defect rather than drift, because these fixtures exist for one reason. When one fails, the thing it tests broke.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Building fixture data that actually exercises a document
The sample data a document test renders, where a copy of a real record covers the easy case and nothing else.
Asserting on the page count, the cheapest test that catches layout
One integer per document, which is nearly free to compute and catches a surprising share of real layout regressions.
Catching pagination mistakes without rendering the document
Asserting on the rules that govern page behaviour rather than on the pages themselves, which is instant and catches the omissions rather than the outcomes.
Seeding dates and IDs so two renders produce the same document
The prerequisite for every comparison in this cluster: making a document depend only on its input, when by default it depends on the clock and on a random source.
Why a document test passes locally and fails in CI
The specific sources of non-determinism in a document pipeline, which are a short and identifiable list rather than a mystery.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.