Testing documents / Making two runs comparable
Testing a document at every page size it is issued at
Running the same document through each format it ships in, which is cheap and routinely skipped because the developer only ever sees one.
Why the obvious approach does not work
A document issued at both A4 and Letter is two layouts, and almost nobody tests the second one. The sizes differ in both dimensions, so a table tuned to fit the width of one can overflow the other, and a page that fits the height of one paginates differently at the other. Because the developer's own locale determines which one they look at, the untested size is whichever their customers use and they do not, which is the worst possible allocation of attention.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Enumerate the formats the document is actually issued at, from configuration rather than memory, and run the suite over each.
- Assert a page count per format. They will differ, and the difference is the thing worth knowing.
- Test the width-sensitive elements at the narrower format: tables, wide figures, anything with fixed column widths.
- Test the height-sensitive behaviour at the shorter format, which is where a block that must not split runs out of room first.
- Use relative units where the layout should adapt and absolute ones where it must not, and let the cross-format test tell you which you got wrong.
- Include any label or receipt size in the sweep, because those are dramatically different shapes and the same template sometimes serves them.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
const FORMATS = ["A4", "Letter"]; // from config, not from memory
describe.each(FORMATS)("invoice at %s", (format) => {
test("fits the width", async () => {
const pdf = await render(invoice(fixtures["long-description"]), { format });
// Nothing should be clipped: the extracted text still has the whole value.
expect(await extractText(pdf)).toContain(fixtures["long-description"].rows[0].desc);
});
test("paginates as expected", async () => {
const pdf = await render(invoice(fixtures["many-lines"]), { format });
expect(await pageCount(pdf)).toBe(EXPECTED_PAGES[format]);
});
test("keeps the totals block together", async () => {
const pdf = await render(invoice(fixtures["one-over"]), { format });
const last = await extractText(pdf, { page: await pageCount(pdf) });
expect(last).toContain("Subtotal");
expect(last).toContain("Total due");
});
});
const EXPECTED_PAGES = { A4: 4, Letter: 5 }; // they differ; that is the pointWhat people do instead
Testing only the format the developer's own locale uses. It is invisible as a decision, because nobody chose it, and it means the format most of the customers receive is the one with no coverage.
What a failure actually tells you
A failure at one format and not the other says the layout depends on a dimension it should not, which is usually a fixed width somewhere that should have been a proportion, or a block sized against the taller page.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Building fixture data that actually exercises a document
The sample data a document test renders, where a copy of a real record covers the easy case and nothing else.
Fixtures at the page boundary, where pagination bugs actually live
Deliberately constructed data that lands content exactly at a page break, which is the only way to test the behaviour that breaks most often.
Testing a document in every language it is issued in
Running the layout suite per locale, because translation changes string lengths and string lengths change the layout.
Seeding dates and IDs so two renders produce the same document
The prerequisite for every comparison in this cluster: making a document depend only on its input, when by default it depends on the clock and on a random source.
Why a document test passes locally and fails in CI
The specific sources of non-determinism in a document pipeline, which are a short and identifiable list rather than a mystery.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.