Testing documents / Running it in CI
Making a failed document test diagnosable without a local rerun
What a failure has to emit so somebody can act on it from the CI log, since reproducing a document failure locally is expensive and often impossible.
Why the obvious approach does not work
A document test that fails with an assertion message and nothing else forces a local reproduction, which means installing the fixture, running the render and opening the file. On a shared runner the failure may not even reproduce, because the cause was environmental. So the artefacts a failure emits are not a convenience, they are the difference between a failure somebody fixes today and one that gets re-run until it passes.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Attach the rendered PDF as a build artefact on failure. It is the primary evidence and it costs nothing to keep for a few days.
- Attach the diff, not just the result: the changed pages as images, or the text diff, so the reviewer sees what moved without opening anything.
- Include the fixture name and the exact inputs in the failure message, so the case is identifiable without reading the test.
- Emit the page count and the embedded font list alongside any failure, since those two facts explain a large share of document defects and cost nothing to collect.
- Keep artefacts for both the failing run and the baseline, because a reviewer comparing two files needs both and the baseline is usually only in the repository as a derived form.
- Never make the failure message the only output. A message says an assertion failed; an artefact says what the document looked like.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
afterEach(async function attachEvidence() {
if (!this.currentTest?.failed) return;
const { pdf, fixture } = this.currentTest.ctx;
if (!pdf) return;
// The primary evidence.
await artefacts.write(`${fixture}--actual.pdf`, pdf);
// The two facts that explain most document defects, free to collect.
await artefacts.write(`${fixture}--facts.txt`, [
`pages: ${await pageCount(pdf)}`,
`fonts: ${(await embeddedFonts(pdf)).join(", ")}`,
].join("\n"));
// The diff, so the reviewer does not have to open anything.
const baseline = await loadBaseline(fixture);
if (baseline) {
await artefacts.write(`${fixture}--baseline.pdf`, baseline);
for (const page of await changedPages(baseline, pdf)) {
await artefacts.write(
`${fixture}--p${page}--diff.png`,
await diffImage(baseline, pdf, page),
);
}
}
});What people do instead
Relying on a local reproduction. It is slow when it works and impossible when the cause was environmental, which is precisely the case where the failure is telling you something important about production.
What a failure actually tells you
If a failure cannot be diagnosed from the artefacts, the artefacts are the thing to fix rather than the test. A suite whose failures are routinely re-run rather than read is a suite with a diagnostics problem, not a flakiness problem.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Why a document test passes locally and fails in CI
The specific sources of non-determinism in a document pipeline, which are a short and identifiable list rather than a mystery.
Keeping a document suite fast enough to stay in the pull request
The specific levers that reduce the cost of a rendering suite, which are different from the ones that speed up ordinary tests.
Page-level visual regression for documents, and its real cost
Rendering each page to an image and comparing against a baseline, which is the only assertion that sees appearance and the most expensive one to keep.
Running document tests in CI without a browser in the image
Testing a rendering pipeline on a runner that has no renderer, which is the normal situation once rendering is a service rather than a dependency.
Catching pagination mistakes without rendering the document
Asserting on the rules that govern page behaviour rather than on the pages themselves, which is instant and catches the omissions rather than the outcomes.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.