Testing documents / Running it in CI
Running document tests in CI without a browser in the image
Testing a rendering pipeline on a runner that has no renderer, which is the normal situation once rendering is a service rather than a dependency.
Why the obvious approach does not work
A suite written against a local browser needs that browser in the CI image, and that image then carries a browser, its shared libraries and its patching burden for the sole purpose of running tests. Once rendering moves to a service the dependency should leave the image with it, but the tests still need a document from somewhere. The question is which tests need a real render at all, and how the ones that do get one without reintroducing the browser.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Split the suite by what it needs. Template tests, data-contract tests and pagination estimates need no render at all and belong in the fast tier that runs on every commit.
- Let the tests that need a document call the render service, the same way production does. That also tests the integration, which a local browser never did.
- Keep one API key scoped to CI, so its usage is separable and it can be revoked without touching production.
- Batch the renders. POST /v1/pdf/batch takes a requests array, so a suite that needs twenty documents makes one call rather than twenty, which is the difference between a suite that fits in a pull request and one that does not.
- Cache rendered fixtures by a hash of their input, so a run that changed nothing about a document does not re-render it.
- Have a plan for the service being unreachable from CI. A suite that fails red on a network problem trains people to ignore red, so classify that failure differently from an assertion failure.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
# The image no longer carries a browser. It carries a client.
FROM node:22-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
# No browser binary, no shared libraries for one, no shm sizing, no reaper.
# The suite splits by what it needs:
# npm run test:unit templates, contracts, estimates. No render.
# npm run test:documents renders, batched, against the service.What people do instead
Keeping the browser in the image so the tests do not have to change. The image stays large, the patching burden stays, and the tests keep exercising a renderer that is no longer the one production uses, which means the suite is now testing something that does not ship.
What a failure actually tells you
A failure in the fast tier is a code defect. A failure in the document tier is either a document defect or an integration problem, and separating the two tiers is what lets you tell which at a glance.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Keeping a document suite fast enough to stay in the pull request
The specific levers that reduce the cost of a rendering suite, which are different from the ones that speed up ordinary tests.
Which document tests run on every commit and which run nightly
Splitting the suite by cost so the fast tier fits in a pull request, because a test that reports a day late reports to nobody.
Testing what happens when a render fails, not just when it works
Exercising the failure branches of a document pipeline, which are the least-tested code in most systems and the code that runs on the worst day.
Catching pagination mistakes without rendering the document
Asserting on the rules that govern page behaviour rather than on the pages themselves, which is instant and catches the omissions rather than the outcomes.
Testing the shape of the data a template expects
Asserting that the data reaching a template has the fields the template reads, which catches the failure that produces a document full of blanks.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.