Testing documents / Running it in CI
Testing the shape of the data a template expects
Asserting that the data reaching a template has the fields the template reads, which catches the failure that produces a document full of blanks.
Why the obvious approach does not work
A template is a consumer of a data shape and almost never declares what it needs. When a field is renamed upstream, the template reads undefined, renders an empty string, and produces a document that is structurally perfect and missing a value. Nothing throws. The rendered-document assertions catch it only if they happen to assert on that specific value, and there are more fields than assertions. A contract makes the expectation explicit and fails at the boundary rather than on the page.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Declare the shape the template needs as a schema, and validate the data against it before rendering rather than after.
- Fail the render on a contract violation rather than rendering a document with holes in it. A loud failure is recoverable and a quiet blank is not.
- Assert that no rendered document contains the string patterns that mean a failure: an unrendered placeholder, the literal text of a null, an empty currency value.
- Version the contract alongside the template, so a change to either is visible as a change to both.
- Test the contract against real production shapes periodically, not only against fixtures, because the fixtures were written from the contract and cannot disagree with it.
- Treat an optional field as a decision. A field the template can do without needs a fixture where it is absent, which is where the empty-line defects come from.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
import { z } from "zod";
// The template's requirements, stated once and enforced at the boundary.
export const InvoiceData = z.object({
number: z.string().min(1),
issuedAt: z.date(),
customer: z.object({
name: z.string().min(1),
line1: z.string().min(1),
line2: z.string().nullable(), // optional: needs its own fixture
postcode: z.string().min(1),
}),
rows: z.array(z.object({
desc: z.string(),
qty: z.number(),
amount: z.number(),
})).min(1),
total: z.number(),
});
export function renderInvoice(raw) {
const data = InvoiceData.parse(raw); // throws rather than rendering blanks
return template(data);
}
// The backstop, on the rendered document, for anything the schema missed.
test("no document ships with a hole in it", async () => {
const text = await extractText(await renderFixture("invoice", "typical"));
for (const pattern of ["{{", "undefined", "null", "NaN", "[object Object]"]) {
expect(text).not.toContain(pattern);
}
});What people do instead
Relying on the rendered assertions to catch a missing field. They only catch the fields somebody thought to assert on, which is a fraction of the fields on the document, and the one that goes missing is reliably one of the others.
What a failure actually tells you
A contract failure names the field and fails before any rendering happens, which makes it the cheapest and clearest failure available. A backstop failure on the rendered text means the contract has a gap.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Snapshot testing the markup before it becomes a document
Asserting on the HTML your template produces rather than on the PDF, which is fast, precise and testing a different thing from what the reader receives.
Building fixture data that actually exercises a document
The sample data a document test renders, where a copy of a real record covers the easy case and nothing else.
Testing what happens when a render fails, not just when it works
Exercising the failure branches of a document pipeline, which are the least-tested code in most systems and the code that runs on the worst day.
Running document tests in CI without a browser in the image
Testing a rendering pipeline on a runner that has no renderer, which is the normal situation once rendering is a service rather than a dependency.
Catching pagination mistakes without rendering the document
Asserting on the rules that govern page behaviour rather than on the pages themselves, which is instant and catches the omissions rather than the outcomes.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.