Testing documents / Choosing what to assert
Page-level visual regression for documents, and its real cost
Rendering each page to an image and comparing against a baseline, which is the only assertion that sees appearance and the most expensive one to keep.
Why the obvious approach does not work
Pixel comparison sees everything, which is its value and its cost. It sees a colour change, a border, a shifted block and a font fallback, and it also sees the antialiasing difference between the machine that made the baseline and the machine running the test. That second category produces failures nobody can act on, and a suite that fails for reasons nobody can act on gets re-baselined until it stops meaning anything. Making this technique survive requires a tolerance, a stable rendering environment, and restraint about how many documents get it.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Compare per page, not per document. A whole-document image is one enormous assertion that tells you nothing about where the change is.
- Set a tolerance that absorbs antialiasing and nothing else. A small per-pixel threshold plus a small proportion-of-pixels threshold, and treat a failure at those settings as real.
- Pin the rendering environment for the comparison images. If the baseline was made on a different machine from the test run, the suite is measuring the machine.
- Apply it to a handful of documents rather than all of them. The ones where appearance is the product: the customer-facing invoice, the certificate, the letterhead. Not every internal report.
- Store baselines outside the git history if they are numerous, because a repository that gains a set of page images per commit becomes unclonable.
- Show the diff image on failure. A failure that only reports a percentage costs the reviewer a local reproduction, and a failure that shows the changed region costs them ten seconds.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
test("invoice looks right", async () => {
const pdf = await renderFixture("invoice", "typical");
const pages = await pageCount(pdf);
for (let p = 1; p <= pages; p += 1) {
const image = await renderPage(pdf, p, { scale: 2 });
await expect(image).toMatchBaseline(`invoice--typical--p${p}.png`, {
// absorbs antialiasing between runs, not a moved block
threshold: 0.02, // per-pixel colour distance
maxDiffRatio: 0.001, // proportion of pixels allowed to differ
});
}
});
/* Three things that make this survivable:
pin the renderer version used for the comparison images, or the
suite measures the runner rather than the document
apply it to documents where appearance is the product, not to all
write the diff image on failure, so the reviewer sees the region
rather than a percentage */What people do instead
Applying it to every document in the suite. The cost is per page and documents have many pages, so the run time grows past the point where it can be part of a pull request, and it moves to nightly, and a nightly visual suite reports yesterday's regression. Narrow it until it fits in the pull request.
What a failure actually tells you
A failure with unchanged text and an unchanged page count is the most specific signal available: the words are right, the length is right, and something visual moved. That is worth looking at every time. A failure that comes with a text or page-count failure tells you nothing new.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Diffing two PDFs meaningfully when the bytes always differ
Comparing two renders when a byte comparison is guaranteed to fail, which means deciding what counts as a difference before you can look for one.
Why a document test passes locally and fails in CI
The specific sources of non-determinism in a document pipeline, which are a short and identifiable list rather than a mystery.
Which document tests run on every commit and which run nightly
Splitting the suite by cost so the fast tier fits in a pull request, because a test that reports a day late reports to nobody.
Golden file testing for a PDF, and what the golden file should be
Keeping a known-good output in the repository and comparing against it, where the mistake is storing the PDF itself.
Asserting on extracted text rather than pixels, and its blind spots
Pulling the text out of the rendered document and asserting on that, which is the cheapest useful assertion available and the one to start with.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.