Testing documents / Choosing what to assert
Golden file testing for a PDF, and what the golden file should be
Keeping a known-good output in the repository and comparing against it, where the mistake is storing the PDF itself.
Why the obvious approach does not work
The instinct is to commit the PDF and compare bytes. That fails on the first run, because a PDF carries a creation date and a document identifier, so two renders of identical input are never byte-identical. Teams that discover this usually respond by normalising the bytes, which means stripping the timestamp and the identifier out of a binary format with a regular expression, and that works until a font subset changes and the file differs for a reason nobody can read. The golden file has to be something diffable, and a PDF is not.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Store a derived artefact as the golden file, not the PDF. Extracted text, a page count, a structural summary: something a person can read in a pull request diff.
- Keep the PDF as a build output rather than a checked-in file, so the repository does not grow a binary per test per commit.
- Make the golden file a text format, so a change to it is reviewable. The value of a golden test is somebody looking at the diff and saying yes or no, and a binary diff denies them that.
- Include the page count in the golden file even when the primary assertion is text, because it is one integer and it catches pagination changes the text cannot see.
- Regenerate golden files with an explicit command rather than a flag on the test run, so updating them is a deliberate act that shows up in the diff.
- One golden file per document per fixture, named for both, so a failure names the case without anybody reading the test.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
// The golden file is text, so a change is reviewable in the diff.
// tests/golden/invoice--many-lines.txt
//
// pages: 4
// ---
// Invoice
// Wynne and Hall Ltd
// ...
import { extractText, pageCount } from "./pdf-helpers";
test("invoice with many lines", async () => {
const pdf = await renderFixture("invoice", "many-lines");
const actual = [
`pages: ${await pageCount(pdf)}`,
"---",
(await extractText(pdf)).trim(),
].join("\n");
expect(actual).toMatchGolden("invoice--many-lines.txt");
});
/* Regeneration is a separate, explicit command:
npm run test:golden -- --update
Not a flag people leave on. A golden file that updates itself is a
test that asserts the code does what it currently does. */What people do instead
Committing the PDF and normalising the bytes. It works for a while and then a dependency changes how a font is subset, every golden file in the suite fails at once, and nobody can tell from the diff whether the documents actually changed. The suite gets regenerated wholesale and stops meaning anything from that day.
What a failure actually tells you
A failure says the document's text or its length changed. It does not say the layout changed, because text extraction is blind to position, so a golden test passing is not evidence the document looks right. It is evidence the content is the same, which is a different and still valuable claim.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Asserting on extracted text rather than pixels, and its blind spots
Pulling the text out of the rendered document and asserting on that, which is the cheapest useful assertion available and the one to start with.
Asserting on the page count, the cheapest test that catches layout
One integer per document, which is nearly free to compute and catches a surprising share of real layout regressions.
Reviewing a golden file change instead of rubber-stamping it
The workflow question underneath every snapshot and golden technique: what happens when the file legitimately changes, and who decides it was legitimate.
Diffing two PDFs meaningfully when the bytes always differ
Comparing two renders when a byte comparison is guaranteed to fail, which means deciding what counts as a difference before you can look for one.
Page-level visual regression for documents, and its real cost
Rendering each page to an image and comparing against a baseline, which is the only assertion that sees appearance and the most expensive one to keep.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.