Testing documents / Choosing what to assert
Snapshot testing the markup before it becomes a document
Asserting on the HTML your template produces rather than on the PDF, which is fast, precise and testing a different thing from what the reader receives.
Why the obvious approach does not work
Snapshotting the markup is attractive because it is instant, deterministic and diffable, and it catches template regressions with no rendering at all. The risk is believing it covers the document. It does not: it proves the template produced the markup you expected, and every layout property, every page break and every font decision happens after that point. A suite made entirely of markup snapshots is a suite that will pass while the document is broken.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Use it for what it is good at: proving the data reached the template, the conditionals took the right branch, and the loop produced the right number of rows.
- Snapshot a normalised form. Collapse whitespace and sort attributes, or the snapshot churns on formatting changes that mean nothing.
- Keep the snapshots small by snapshotting a component of the document rather than the whole page. A whole-page snapshot fails on every change and gets approved without reading.
- Never let it be the only assertion. Pair every markup snapshot with at least a page count on the rendered output, so the layout has some coverage.
- Assert the absence of unrendered placeholders explicitly, since that is the template failure a snapshot is most likely to bless by capturing it.
- Treat an approved snapshot change as a code review item. The whole value is a person reading the diff.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
// Fast and precise, and testing the input rather than the output.
test("invoice template renders the line items", () => {
const html = renderInvoiceBody({ rows, total });
expect(normaliseHtml(html)).toMatchSnapshot();
});
// The pairing that makes it safe. Without this, the suite can pass
// while the document is broken.
test("invoice paginates", async () => {
const pdf = await render(renderInvoice({ rows, total }));
expect(await pageCount(pdf)).toBe(4);
});
function normaliseHtml(html) {
return html
.replace(/>\s+</g, "><") // formatting churn, not content
.replace(/\s{2,}/g, " ")
.trim();
}
/* What a markup snapshot cannot see, all of it after this point:
which page anything lands on
whether a table repeated its header
whether a font loaded
whether a block broke where it should not */What people do instead
Treating a green markup suite as document coverage. It is the most common way a team ends up with a hundred passing tests and a broken invoice, because every failure mode that matters to a reader is downstream of the assertion.
What a failure actually tells you
A failure says the template produced different markup. That is a precise, local signal and it usually points at one component. It says nothing about whether the resulting document is correct.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Asserting on extracted text rather than pixels, and its blind spots
Pulling the text out of the rendered document and asserting on that, which is the cheapest useful assertion available and the one to start with.
Testing the shape of the data a template expects
Asserting that the data reaching a template has the fields the template reads, which catches the failure that produces a document full of blanks.
Asserting on the page count, the cheapest test that catches layout
One integer per document, which is nearly free to compute and catches a surprising share of real layout regressions.
Golden file testing for a PDF, and what the golden file should be
Keeping a known-good output in the repository and comparing against it, where the mistake is storing the PDF itself.
Diffing two PDFs meaningfully when the bytes always differ
Comparing two renders when a byte comparison is guaranteed to fail, which means deciding what counts as a difference before you can look for one.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.