Testing documents / Running it in CI
Testing that the links in a PDF exist and point somewhere real
Checking the link annotations in the finished document, which are invisible to text extraction and to a pixel comparison alike.
Why the obvious approach does not work
A link in a PDF is an annotation: a rectangle on a page with a destination attached. Text extraction returns the visible words and not the destination, and a pixel comparison sees the underline and not what it points at. So a document whose links all point at a staging host, or at a URL with an unrendered placeholder in it, passes every other assertion in the suite. Because links are usually added late and tested by clicking one, this is a defect that ships in the documents nobody clicked.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Extract the annotations from the rendered document and assert on the destinations, not on the visible text.
- Assert every destination is absolute and uses the scheme you expect, which catches a relative URL that will not resolve from a reader.
- Assert no destination points at a development or staging host, which is the failure that reaches production most often.
- Assert no destination contains an unrendered placeholder, since a templated URL fails silently in exactly the same way a templated value does.
- Count the links and assert the count, so a link that stopped being generated is caught as well as one that is wrong.
- Do not fetch the destinations in the test suite. That makes the suite depend on the internet and on somebody else's uptime; validate the shape, not the reachability.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
test("links point somewhere real", async () => {
const pdf = await renderFixture("invoice", "typical");
const links = await extractLinks(pdf); // annotations, not text
expect(links.length).toBe(3); // a missing link is a defect too
for (const { url } of links) {
expect(url).toMatch(/^https:\/\//); // absolute and secure
expect(url).not.toMatch(/localhost|staging|\.test\b/);
expect(url).not.toContain("{{"); // unrendered placeholder
expect(url).not.toContain("undefined");
}
expect(links.map((l) => new URL(l.url).host))
.toEqual(["pdfpipe.xyz", "pdfpipe.xyz", "assets.example.com"]);
});
/* Deliberately not here: fetching each destination. That makes the
suite depend on the internet and on somebody else's uptime, and it
fails for reasons that are not defects in this document. Validate the
shape; monitor the reachability somewhere else. */What people do instead
Testing links by clicking one during review. It confirms that one link on that one document, and it is the reason a whole run of invoices goes out pointing at a staging host after an environment variable was missed.
What a failure actually tells you
A staging host in a link destination says an environment variable did not reach the template. That is usually a configuration problem rather than a template problem, and it frequently affects more than the document you were looking at.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Asserting the right fonts actually got into the document
Checking the font list inside the finished PDF, which is a one-line assertion against the failure that changes every page and raises no error.
Asserting on extracted text rather than pixels, and its blind spots
Pulling the text out of the rendered document and asserting on that, which is the cheapest useful assertion available and the one to start with.
What can and cannot be asserted about a document's accessibility
Checking the structural properties a machine can see, and being clear that the property which matters most is not one of them.
Running document tests in CI without a browser in the image
Testing a rendering pipeline on a runner that has no renderer, which is the normal situation once rendering is a service rather than a dependency.
Catching pagination mistakes without rendering the document
Asserting on the rules that govern page behaviour rather than on the pages themselves, which is instant and catches the omissions rather than the outcomes.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.