Testing documents / Making two runs comparable
Testing a document in every language it is issued in
Running the layout suite per locale, because translation changes string lengths and string lengths change the layout.
Why the obvious approach does not work
Localisation is treated as a content task and it is a layout risk. A translated label is frequently longer than the original, and the strings with the least room, the column headers, are the ones that grow most. A header that wraps to two lines makes the header row taller, which removes a row from every page, which changes the pagination of the whole document. None of that is visible in the language the developer reads, and none of it is caught by a suite that runs in one locale.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Run the page-count assertions per locale, and expect the counts to differ. A difference is information, not a failure.
- Assert that no column header wrapped past the space reserved for it, which is the specific mechanism behind the pagination change.
- Include the longest translation you actually ship rather than a representative one, and take it from the translation files rather than estimating.
- Test a right-to-left locale if you issue one, because the layout mirrors and the amount column changes side.
- Test the number and date formatting per locale as content assertions, since a separator or an ambiguous date is a defect in the values rather than the layout.
- Add a pseudo-locale that lengthens every string by a fixed proportion, as a cheap way to find the elements with no slack before a real translation arrives.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
const LOCALES = ["en-GB", "de-DE", "fr-FR", "ar-EG"];
describe.each(LOCALES)("invoice in %s", (locale) => {
test("paginates", async () => {
const pdf = await render(invoice(fixtures["many-lines"], { locale }));
expect(await pageCount(pdf)).toBe(EXPECTED_PAGES[locale]);
});
test("formats the amounts for the locale", async () => {
const pdf = await render(invoice(fixtures["typical"], { locale }));
const text = await extractText(pdf);
expect(text).toContain(formatAmount(4180.5, locale));
});
test("no header wrapped", async () => {
// The mechanism behind the pagination change: a two-line header row
// removes a row from every page.
const pdf = await render(invoice(fixtures["typical"], { locale }));
expect(await headerRowHeight(pdf)).toBeLessThanOrEqual(MAX_HEADER_HEIGHT);
});
});
/* A pseudo-locale finds the elements with no slack before a real
translation arrives:
"Description" -> "[Descriptionxxxxx]"
Everything that wraps under it will wrap under some real language. */What people do instead
Testing the layout in the source language and treating translation as content that arrives later. By the time it arrives the column widths are settled, and the only remaining levers are shrinking the type or abbreviating, both of which are bad answers to a problem that was cheap to avoid.
What a failure actually tells you
A page count that differs by more than a page or two between locales says something is reflowing rather than merely rendering wider, and the header row is the first place to look.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Testing a document at every page size it is issued at
Running the same document through each format it ships in, which is cheap and routinely skipped because the developer only ever sees one.
Building fixture data that actually exercises a document
The sample data a document test renders, where a copy of a real record covers the easy case and nothing else.
Asserting on extracted text rather than pixels, and its blind spots
Pulling the text out of the rendered document and asserting on that, which is the cheapest useful assertion available and the one to start with.
Seeding dates and IDs so two renders produce the same document
The prerequisite for every comparison in this cluster: making a document depend only on its input, when by default it depends on the clock and on a random source.
Fixtures at the page boundary, where pagination bugs actually live
Deliberately constructed data that lands content exactly at a page break, which is the only way to test the behaviour that breaks most often.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.