Testing documents / Running it in CI
Asserting the right fonts actually got into the document
Checking the font list inside the finished PDF, which is a one-line assertion against the failure that changes every page and raises no error.
Why the obvious approach does not work
A font that fails to load produces a complete, readable document in a different typeface. Nothing errors, the text extracts identically, the page count may not even change much, and the only signal is that it looks wrong to somebody who knows what it should look like. Since the font is fetched over a network at render time, this failure is also intermittent, which means it can pass in CI and fail in production. The finished PDF carries a list of the fonts embedded in it, and asserting on that list closes the whole class.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Assert the expected families are present in the embedded font list of the rendered document. It is one assertion and it covers every page.
- Assert that unexpected families are absent, which is how a fallback is detected: the fallback appears in the list rather than the font you wanted.
- Run it on a document that uses every weight and style you load, or the assertion only covers the faces that fixture happened to use.
- Include it in the fast document tier, since it needs one render and one inspection and catches a failure that affects everything.
- Assert the count as well as the names, because a document carrying nine embedded faces when you loaded three is a size problem worth knowing about.
- Do not assert on the exact subset name, which varies between renders and is not a stable value.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
test("the document embeds the fonts we asked for", async () => {
// A fixture that uses every face: regular, bold and italic.
const pdf = await renderFixture("invoice", "all-weights");
const fonts = await embeddedFonts(pdf); // names from the PDF itself
const families = new Set(fonts.map(stripSubsetPrefix));
// Present: what we declared.
expect(families).toContain("BrandSans");
expect(families).toContain("BrandSans-Bold");
expect(families).toContain("IBMPlexMono");
// Absent: the fallbacks. A fallback in this list is the failure.
expect(families).not.toContain("Helvetica");
expect(families).not.toContain("Arial");
expect(families).not.toContain("Times");
// Count, as a size guard: three faces loaded, three embedded.
expect(families.size).toBe(3);
});
// Subset prefixes vary between renders and are not stable values.
function stripSubsetPrefix(name) {
return name.replace(/^[A-Z]{6}\+/, "");
}What people do instead
Assuming a visual test covers it. A pixel comparison does catch a font change, on the documents that have a visual baseline, which is usually a handful. This assertion covers every document that runs it, for a fraction of the cost, and it names the offending family rather than showing you a red region.
What a failure actually tells you
A fallback family in the embedded list is unambiguous: the font did not load. That is either a URL problem, a network problem or a licence-driven change somebody made, and the assertion tells you which family to go and look at.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Why a document test passes locally and fails in CI
The specific sources of non-determinism in a document pipeline, which are a short and identifiable list rather than a mystery.
Asserting on extracted text rather than pixels, and its blind spots
Pulling the text out of the rendered document and asserting on that, which is the cheapest useful assertion available and the one to start with.
Testing that the links in a PDF exist and point somewhere real
Checking the link annotations in the finished document, which are invisible to text extraction and to a pixel comparison alike.
Running document tests in CI without a browser in the image
Testing a rendering pipeline on a runner that has no renderer, which is the normal situation once rendering is a service rather than a dependency.
Catching pagination mistakes without rendering the document
Asserting on the rules that govern page behaviour rather than on the pages themselves, which is instant and catches the omissions rather than the outcomes.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.