Testing documents / Making two runs comparable
Seeding dates and IDs so two renders produce the same document
The prerequisite for every comparison in this cluster: making a document depend only on its input, when by default it depends on the clock and on a random source.
Why the obvious approach does not work
A document generated today says today's date, and one generated tomorrow does not. An invoice number from a sequence is different on every run. A reference generated from a random source is different every time. None of that is a defect in production and all of it makes comparison impossible, so the fixture has to control the clock and the identifier source. The trap is doing it by substituting values in the assertion instead of at the source, which leaves the document non-deterministic and papers over it.
What to do instead
Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.
- Inject the clock rather than reading it. A function that takes the current time as a parameter is testable and one that calls the system clock is not.
- Inject the identifier source the same way, so a fixture can supply a fixed sequence.
- Freeze to a fixed date rather than to the current one, so the golden file is stable across days, and choose a date that is not near a month or year boundary unless you are testing those.
- Test the boundaries deliberately with their own fixtures: end of month, end of year, a leap day. Those are real defects and a frozen clock hides them unless you go looking.
- Set the timezone explicitly in the test environment, or a suite that passes locally fails on a runner in a different zone.
- Do not normalise dates out of the comparison. That makes the date untested, and the date is on the document.
In practice
Test code, with the thing people write instead kept in a comment where it is the more instructive half.
// Injected at the source, so the document is a function of its input.
function buildInvoice({ rows, issuedAt, invoiceNumber }) {
return { rows, issuedAt, invoiceNumber, total: sum(rows) };
}
// Production
buildInvoice({ rows, issuedAt: new Date(), invoiceNumber: await nextNumber() });
// Test: fixed clock, fixed sequence, explicit zone.
process.env.TZ = "UTC";
const invoice = buildInvoice({
rows: fixture.rows,
issuedAt: new Date("2026-03-14T00:00:00Z"),
invoiceNumber: "2026-118",
});
/* Boundaries get their own fixtures rather than being hidden by the
freeze, because they are real defects:
2026-01-31 end of month, and a payment term that lands on the 31st
2026-12-31 end of year, and a financial period rollover
2028-02-29 a leap day
What not to do: strip the date out of the comparison. That leaves it
untested, and it is printed on the document. */What people do instead
Normalising the volatile values out of the extracted text before comparing. It makes the test pass and removes the date, the invoice number and the reference from coverage, which are three of the most important values on the document and the ones a customer will phone about.
What a failure actually tells you
If this is not done, nothing else in the cluster works, so the signal is indirect: a suite that cannot be made to pass twice in a row has a determinism problem rather than a document problem, and looking for a date is the first thing to try.
Frequently asked
Why not just compare the PDF bytes?
Because two renders of identical input are not byte-identical. A PDF carries a creation timestamp and a document identifier, and font subsetting can differ between runs, so a byte comparison fails on the first run and keeps failing. The techniques here all pick a projection of the document that is stable across runs and still says something about whether it is correct.
How much of this is worth doing for one document?
The cheap end, and it is genuinely cheap: a page count and a handful of containment assertions on extracted text will catch most regressions for the cost of one render. Visual comparison and the full fixture matrix earn their keep when the document is customer-facing and numerous, and not before.
Does any of this need a browser in the CI image?
No. Rendering happens over an API call, so the runner needs a client rather than an engine, and the tests that need no document at all, the contract checks and the static pagination checks, need nothing. Keeping a browser in the image to run tests means testing a renderer that is not the one shipping.
Related testing techniques
The techniques that pair with this one, then the rest of the same group.
Building fixture data that actually exercises a document
The sample data a document test renders, where a copy of a real record covers the easy case and nothing else.
Why a document test passes locally and fails in CI
The specific sources of non-determinism in a document pipeline, which are a short and identifiable list rather than a mystery.
Golden file testing for a PDF, and what the golden file should be
Keeping a known-good output in the repository and comparing against it, where the mistake is storing the PDF itself.
Fixtures at the page boundary, where pagination bugs actually live
Deliberately constructed data that lands content exactly at a page break, which is the only way to test the behaviour that breaks most often.
Testing a document at every page size it is issued at
Running the same document through each format it ships in, which is cheap and routinely skipped because the developer only ever sees one.
Every document testing technique
The full list, grouped by what to assert, how to make runs comparable, and how to run it in CI.
What goes wrong in a rendered document
The failure modes these tests exist to catch, described from the symptom rather than the assertion.
Every technique here needs a document to assert on. Render one of your real templates first, then decide which projection of it is worth a test.