Testing documents / Running it in CI
Which document tests run on every commit and which run nightly
Splitting the suite by cost so the fast tier fits in a pull request, because a test that reports a day late reports to nobody.
Why the obvious approach does not work
Document tests are slow enough that the whole suite will not fit in a pull request, so something has to move to a slower tier. The decision is usually made by accident: the suite grows until somebody complains, then all of it moves to nightly, and from that point every regression is discovered the following morning by whoever reads the failure, who is not the person who caused it. Choosing the split deliberately keeps the highest-value assertions in front of the author while the change is still in their head.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Put everything that needs no render in the commit tier. Contract tests, markup snapshots and static pagination checks cost milliseconds and catch a large share of defects.
- Put a small, chosen set of rendered assertions in the commit tier too: page counts and extracted text on a handful of representative fixtures. Batch them into one call so the cost is one round trip.
- Put the expensive and the exhaustive in the nightly tier: pixel comparison, the full fixture matrix, every locale and every page size.
- Make the nightly failure land on somebody. A failure with no owner is a failure that accumulates, and a nightly suite with twelve known failures is off.
- Report the commit tier's duration as a number people can see, so its growth is noticed before it becomes a complaint.
- Re-examine the split when the numbers change. It is a budget decision and the budget moves.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
# Commit tier: fits in a pull request, runs on every push.
# contract tests no render
# markup snapshots no render
# static pagination checks no render
# page count + text one batched render call over ~8 fixtures
#
# Nightly tier: exhaustive, expensive, owned by somebody.
# pixel comparison per page
# the full fixture matrix
# every locale and every page size
# load test against the render endpoint
# The batched call is what keeps the rendered part affordable: one
# request rather than one per fixture.
# POST /v1/pdf/batch { "requests": [ { html, options }, ... ] }
# Print the budget so its growth is visible before somebody complains.
- name: Document tests
run: npm run test:documents -- --reporter=duration
timeout-minutes: 4What people do instead
Moving the whole suite to nightly the first time it gets slow. It solves the complaint and removes the feedback, and the assertions that would have caught a regression in review now catch it after it merged.
What a failure actually tells you
A commit-tier failure is attributable to the change in front of you. A nightly failure is attributable to one of a day's changes, which is why the split matters and why the cheap high-value assertions belong in the first tier.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Keeping a document suite fast enough to stay in the pull request
The specific levers that reduce the cost of a rendering suite, which are different from the ones that speed up ordinary tests.
Running document tests in CI without a browser in the image
Testing a rendering pipeline on a runner that has no renderer, which is the normal situation once rendering is a service rather than a dependency.
Page-level visual regression for documents, and its real cost
Rendering each page to an image and comparing against a baseline, which is the only assertion that sees appearance and the most expensive one to keep.
Catching pagination mistakes without rendering the document
Asserting on the rules that govern page behaviour rather than on the pages themselves, which is instant and catches the omissions rather than the outcomes.
Testing the shape of the data a template expects
Asserting that the data reaching a template has the fields the template reads, which catches the failure that produces a document full of blanks.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.