Testing documents / Making two runs comparable
Building fixture data that actually exercises a document
The sample data a document test renders, where a copy of a real record covers the easy case and nothing else.
Why the obvious approach does not work
Fixtures usually start as a record somebody copied out of a database because it was in front of them, and that record is typical, which is precisely why it exercises nothing. It has three line items, a short customer name, a positive total and no unusual characters. Every layout failure in the document lives outside it: the long description that wraps, the customer with no second address line, the ninety-line order, the negative total, the name with a character the font does not have. A fixture set built from real records is a fixture set of easy cases.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Build fixtures deliberately rather than sampling. Each one should exist to exercise a specific property of the layout, and its name should say which.
- Cover the shape of the data, not the plausibility of it: empty, one, many, and far too many. Those four cover most pagination behaviour.
- Cover the extremes of every variable-length field, using the longest value the schema allows rather than the longest value you have seen.
- Include the awkward characters your real data contains, which for names and addresses is more than the Latin alphabet.
- Keep fixtures as data, not as rendered markup, so a template change applies to all of them at once.
- Name them for what they test. A fixture called typical is a fixture nobody can reason about; one called address-missing-second-line is a specification.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
// Named for what they exercise, not for what they resemble.
export const fixtures = {
"one-line": { rows: rows(1) },
"fits-exactly": { rows: rows(28) }, // last row on page one
"one-over": { rows: rows(29) }, // the boundary
"many-lines": { rows: rows(120) },
"empty": { rows: [] },
"long-description": { rows: [{ ...row, desc: "x".repeat(400) }] },
"address-missing-line2": { customer: { ...c, line2: null } },
"negative-total": { rows: [{ ...row, amount: -620 }] },
"long-customer-name": { customer: { ...c, name: "y".repeat(120) } },
"non-latin-name": { customer: { ...c, name: "Ærlend Þórsdóttir" } },
"zero-total": { rows: [{ ...row, amount: 0 }] },
};
/* The lengths come from the schema, not from the data you have. If the
name column is 120 characters, the fixture is 120 characters, because
that is what will eventually arrive. */What people do instead
One fixture called something like sample, copied from production, used by every test. The suite then has real coverage of one shape of document and none of the others, and every layout defect ships because the fixture that would have caught it was never written.
What a failure actually tells you
Which fixture failed is most of the diagnosis. A failure on long-description is a wrapping problem, on one-over is a pagination problem, and on address-missing-line2 is a conditional-field problem. That is why the names matter.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Fixtures at the page boundary, where pagination bugs actually live
Deliberately constructed data that lands content exactly at a page break, which is the only way to test the behaviour that breaks most often.
Seeding dates and IDs so two renders produce the same document
The prerequisite for every comparison in this cluster: making a document depend only on its input, when by default it depends on the clock and on a random source.
Asserting on the page count, the cheapest test that catches layout
One integer per document, which is nearly free to compute and catches a surprising share of real layout regressions.
Why a document test passes locally and fails in CI
The specific sources of non-determinism in a document pipeline, which are a short and identifiable list rather than a mystery.
Testing a document at every page size it is issued at
Running the same document through each format it ships in, which is cheap and routinely skipped because the developer only ever sees one.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.