Testing documents / Choosing what to assert
Asserting on extracted text rather than pixels, and its blind spots
Pulling the text out of the rendered document and asserting on that, which is the cheapest useful assertion available and the one to start with.
Why the obvious approach does not work
Extraction is fast, deterministic and comparable, which makes it the natural default. The catch is what it cannot see. Extracted text has no positions, so a total in the wrong column extracts identically to one in the right column. It has no styling, so a heading that lost its weight is unchanged. And extraction order is not always reading order: a two-column layout can extract as two interleaved columns, which makes a naive string comparison fail for a reason that is not a defect.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Assert on content that must be present rather than on the whole string, at least to begin with. The total, the invoice number, the customer name: those are the claims worth failing a build over.
- Normalise whitespace before comparing. Extraction inserts line breaks at layout boundaries, so a paragraph that reflowed produces a different string with identical words.
- Test extraction order explicitly on any multi-column layout, once, and then know whether you can trust a full-string comparison on it.
- Assert that something is absent as well as present. A draft watermark that should not be on a final document is invisible to a positive assertion.
- Remember this is the assertion that proves the document says the right thing, and pair it with a page count, which proves it is the right length. Neither proves it looks right.
- Extract from the finished PDF, not from the HTML you sent. Testing the input tests your template engine, which is a different and less interesting claim.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
const text = normalise(await extractText(pdf));
// The claims worth failing a build over.
expect(text).toContain("Invoice 2026-118");
expect(text).toContain("Wynne and Hall Ltd");
expect(text).toContain("5,016.00");
// Absence matters too, and no positive assertion covers it.
expect(text).not.toContain("DRAFT");
expect(text).not.toContain("{{"); // an unrendered placeholder
expect(text).not.toContain("undefined"); // a missing value, stringified
function normalise(s) {
return s
.replace(/\s+/g, " ") // extraction breaks lines at layout boundaries
.trim();
}
/* The blind spots, stated so nobody assumes otherwise:
position a total in the wrong column extracts the same
styling a heading that lost its weight is unchanged
order a two-column page may extract as interleaved columns
Pair this with a page count, and with a pixel comparison on the few
documents where appearance is the product. */What people do instead
Asserting on the full extracted string as an equality. It fails whenever a paragraph reflows, which happens for reasons that are not defects, and the team responds by loosening the assertion until it checks nothing. Containment assertions on the values that matter survive reflow and keep their meaning.
What a failure actually tells you
A failure says a value is missing, wrong, or newly present. That is the highest-value failure in the suite, because a wrong number on an invoice is worse than a misaligned one, and it is the class of defect a visual test is least likely to notice.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Asserting on the page count, the cheapest test that catches layout
One integer per document, which is nearly free to compute and catches a surprising share of real layout regressions.
Golden file testing for a PDF, and what the golden file should be
Keeping a known-good output in the repository and comparing against it, where the mistake is storing the PDF itself.
Diffing two PDFs meaningfully when the bytes always differ
Comparing two renders when a byte comparison is guaranteed to fail, which means deciding what counts as a difference before you can look for one.
Page-level visual regression for documents, and its real cost
Rendering each page to an image and comparing against a baseline, which is the only assertion that sees appearance and the most expensive one to keep.
Snapshot testing the markup before it becomes a document
Asserting on the HTML your template produces rather than on the PDF, which is fast, precise and testing a different thing from what the reader receives.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.