Testing documents / Choosing what to assert
Asserting on the page count, the cheapest test that catches layout
One integer per document, which is nearly free to compute and catches a surprising share of real layout regressions.
Why the obvious approach does not work
There is no problem with this test, which is why it deserves its own page: it is the most underused assertion in document testing. Almost anything that breaks a layout also changes how much content fits on a page. A margin that grew, a font that fell back to something wider, a table that stopped repeating its header, a block that stopped being allowed to break: every one of those moves the page count, and none of them is visible to a text assertion. It costs one number and a comparison.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Assert an exact page count, not a range. A range is a test that tolerates the thing it exists to catch.
- Assert it on fixtures long enough to paginate. A one-page fixture always has one page and the assertion is noise.
- Keep a fixture at each interesting boundary: just under a page break, just over, and well past. The boundary cases are where pagination bugs live.
- Update the expected count deliberately when it changes, and treat the change as a review question rather than a maintenance chore. A page count moving is a fact about the document worth one person's attention.
- Record the count alongside the golden text, so one file carries both and one diff shows both.
- Where a document's length legitimately depends on data, assert the count for each fixture rather than abandoning the assertion.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
// One integer. It catches a margin change, a font fallback, a table
// that stopped repeating its header, and a block that stopped breaking.
test.each([
["invoice", "one-line", 1],
["invoice", "fits-exactly", 1], // the boundary: last row on page one
["invoice", "one-over", 2], // one row more than fits
["invoice", "many-lines", 4],
])("%s/%s renders %i page(s)", async (doc, fixture, expected) => {
const pdf = await renderFixture(doc, fixture);
expect(await pageCount(pdf)).toBe(expected);
});
/* The two middle fixtures are the point. A pagination bug lives in the
one-row window between "fits" and "does not fit", and a suite whose
fixtures are all comfortably short or comfortably long never enters
that window. */What people do instead
Only having short fixtures. A suite where every document is one page has a page-count assertion that can never fail, which feels like a passing test and is an absent one. The fixtures that matter are the ones near a boundary, and they have to be constructed deliberately because real sample data rarely lands there.
What a failure actually tells you
A page count change says the layout moved without saying how. That is a weak signal on its own and a strong trigger: it is the cheapest possible reason to go and look, and it fires on most of the regressions that a text assertion sleeps through.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Asserting on extracted text rather than pixels, and its blind spots
Pulling the text out of the rendered document and asserting on that, which is the cheapest useful assertion available and the one to start with.
Fixtures at the page boundary, where pagination bugs actually live
Deliberately constructed data that lands content exactly at a page break, which is the only way to test the behaviour that breaks most often.
Catching pagination mistakes without rendering the document
Asserting on the rules that govern page behaviour rather than on the pages themselves, which is instant and catches the omissions rather than the outcomes.
Golden file testing for a PDF, and what the golden file should be
Keeping a known-good output in the repository and comparing against it, where the mistake is storing the PDF itself.
Diffing two PDFs meaningfully when the bytes always differ
Comparing two renders when a byte comparison is guaranteed to fail, which means deciding what counts as a difference before you can look for one.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.