Testing documents / Choosing what to assert
Reviewing a golden file change instead of rubber-stamping it
The workflow question underneath every snapshot and golden technique: what happens when the file legitimately changes, and who decides it was legitimate.
Why the obvious approach does not work
Every golden technique has the same failure mode, and it is social rather than technical. The file changes, the test fails, somebody runs the update command, and the diff goes into a pull request where it is approved along with everything else. After that has happened a few times the golden files are a record of what the code does rather than a statement of what it should do, and the suite has become expensive decoration. Preventing that is a matter of making the update visible and the review possible.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Make the golden file readable. A text diff can be reviewed and a binary diff cannot, which is most of why the golden file should not be the PDF.
- Update through an explicit command, never a flag that can be left on in a local config.
- Keep the diff small by scoping golden files narrowly. A reviewer will read a twelve-line diff and will not read a four-hundred-line one.
- Require the pull request to say why the document changed. One sentence, in the description, and the reviewer checks the diff against it.
- Fail the build if golden files changed in a commit that touched nothing capable of changing them, which catches an accidental regeneration.
- Review the count of changed golden files as a signal in itself. One changed file is a change; forty changed files is a font, a margin or a dependency, and it deserves a different conversation.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
# Explicit, separate, and visible in the shell history.
npm run test:golden -- --update
# What the reviewer sees, and can act on:
#
# tests/golden/invoice--many-lines.txt
# - pages: 4
# + pages: 5
# ---
# Invoice 2026-118
# ...
#
# One line, and it is the question: why is this document a page longer?
# A guard worth having in CI: golden files changed, but nothing that
# could legitimately change them did.
changed_golden=$(git diff --name-only origin/main... -- tests/golden | wc -l)
changed_source=$(git diff --name-only origin/main... -- templates lib | wc -l)
if [ "$changed_golden" -gt 0 ] && [ "$changed_source" -eq 0 ]; then
echo "Golden files changed with no template or library change."
exit 1
fiWhat people do instead
An update flag in the local test configuration. Once it is on, every run silently rewrites the baselines, the suite passes permanently, and nobody discovers it until a regression reaches a customer. It is worth grepping for periodically.
What a failure actually tells you
The number of golden files in a diff is itself the signal. One is a feature. Forty is an environmental change, and the right response is to find out what moved rather than to read forty diffs.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Golden file testing for a PDF, and what the golden file should be
Keeping a known-good output in the repository and comparing against it, where the mistake is storing the PDF itself.
Why a document test passes locally and fails in CI
The specific sources of non-determinism in a document pipeline, which are a short and identifiable list rather than a mystery.
Page-level visual regression for documents, and its real cost
Rendering each page to an image and comparing against a baseline, which is the only assertion that sees appearance and the most expensive one to keep.
Diffing two PDFs meaningfully when the bytes always differ
Comparing two renders when a byte comparison is guaranteed to fail, which means deciding what counts as a difference before you can look for one.
Asserting on extracted text rather than pixels, and its blind spots
Pulling the text out of the rendered document and asserting on that, which is the cheapest useful assertion available and the one to start with.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.