Testing documents / Making two runs comparable
Why a document test passes locally and fails in CI
The specific sources of non-determinism in a document pipeline, which are a short and identifiable list rather than a mystery.
Why the obvious approach does not work
Document suites go flaky in ways that generic advice about test isolation does not address, because the causes are particular to rendering. A font that is installed on a developer machine and not on the runner. A timezone difference that moves a date. An image fetched over the network that was cached locally and is slow in CI. Antialiasing that differs between machines. A render that finished before a script did on a fast machine and did not on a loaded one. Each of those has a specific fix, and treating them collectively as flakiness leads to a retry wrapper instead.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Never resolve a font from the system. Declare every face with @font-face at a URL, so the render does not depend on what is installed where.
- Pin the timezone in the test environment and freeze the clock, which removes two independent sources at once.
- Vendor or inline every asset the fixture needs, so no test depends on a network fetch succeeding within a timeout.
- Pin the rendering environment for visual baselines, or accept that pixel comparison measures the runner.
- Make readiness explicit rather than timing-dependent. A document that waits for a script is a document whose output depends on machine load.
- Do not add a retry. A retried document test hides exactly the class of defect that makes a document occasionally wrong in production, which is the most expensive kind.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
/* The short list, with the fix for each. Work through it before
reaching for a retry, because a retry hides the production defect.
font resolved from the system
-> @font-face at a URL, never a bare family name
timezone differs between laptop and runner
-> process.env.TZ = "UTC" in the test setup
date moved overnight
-> freeze the clock in the fixture
remote image cached locally, slow in CI
-> inline it as a data URI, or serve it from the fixture server
antialiasing differs between machines
-> pin the renderer for baselines, or raise the pixel tolerance
script had not finished on a loaded runner
-> make readiness explicit; do not race the render */
// A guard that catches the first cause before it causes a flake.
test("no fixture relies on a system font", () => {
const css = readFileSync("templates/print.css", "utf8");
const families = [...css.matchAll(/font-family:\s*([^;]+);/g)]
.flatMap((m) => m[1].split(",").map((f) => f.trim().replace(/["']/g, "")));
const declared = new Set(
[...css.matchAll(/@font-face[^}]*font-family:\s*["']([^"']+)["']/g)]
.map((m) => m[1]),
);
const generic = new Set(["serif", "sans-serif", "monospace", "ui-monospace"]);
for (const f of families) {
expect(declared.has(f) || generic.has(f)).toBe(true);
}
});What people do instead
Wrapping the suite in a retry. It turns a red build green and it removes the only evidence you had that the pipeline is non-deterministic, which is the same non-determinism that will eventually produce a wrong document for a customer with nobody watching.
What a failure actually tells you
A test that passes locally and fails in CI is naming an environmental dependency. Which one is usually identifiable in a minute from the list above, and the fix removes a production risk rather than a test annoyance.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Seeding dates and IDs so two renders produce the same document
The prerequisite for every comparison in this cluster: making a document depend only on its input, when by default it depends on the clock and on a random source.
Page-level visual regression for documents, and its real cost
Rendering each page to an image and comparing against a baseline, which is the only assertion that sees appearance and the most expensive one to keep.
Running document tests in CI without a browser in the image
Testing a rendering pipeline on a runner that has no renderer, which is the normal situation once rendering is a service rather than a dependency.
Building fixture data that actually exercises a document
The sample data a document test renders, where a copy of a real record covers the easy case and nothing else.
Fixtures at the page boundary, where pagination bugs actually live
Deliberately constructed data that lands content exactly at a page break, which is the only way to test the behaviour that breaks most often.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.