Reference
Testing and CI for generated documents
A generated document is a bad fit for every default testing technique, and the reasons are specific. A unit test asserts on a value, and a document is a file whose interesting property is what a person sees. A snapshot test asserts the bytes did not change, and two renders of the same document never produce the same bytes, because a PDF carries a creation timestamp and an identifier.
A visual comparison works and costs more per assertion than anything else in the suite. And the failure that matters most, a table splitting in the wrong place, is invisible to all three unless the fixture is long enough to paginate, which the fixture somebody copied out of production never is.
So the material here is about picking an assertion cheap enough to run on every commit and specific enough to catch something, and about making two runs comparable at all, which is a prerequisite nobody mentions. For the failures these tests exist to catch, described from the symptom, see what goes wrong.
Choosing what to assert
The first decision, and the one that determines whether the suite is useful or ignored. Every assertion on a document trades specificity against cost, and the cheap ones catch different failures from the expensive ones.
- Golden file testing
Keeping a known-good output in the repository and comparing against it, where the mistake is storing the PDF itself.
- Diffing two PDFs
Comparing two renders when a byte comparison is guaranteed to fail, which means deciding what counts as a difference before you can look for one.
- Asserting on extracted text
Pulling the text out of the rendered document and asserting on that, which is the cheapest useful assertion available and the one to start with.
- Testing the page count
One integer per document, which is nearly free to compute and catches a surprising share of real layout regressions.
- Visual regression per page
Rendering each page to an image and comparing against a baseline, which is the only assertion that sees appearance and the most expensive one to keep.
- Snapshot testing a template
Asserting on the HTML your template produces rather than on the PDF, which is fast, precise and testing a different thing from what the reader receives.
- Approving a golden file change
The workflow question underneath every snapshot and golden technique: what happens when the file legitimately changes, and who decides it was legitimate.
Making two runs comparable
Before any assertion is meaningful, two runs of the same input have to produce the same output. They do not by default, and the reasons are mundane: a timestamp, an identifier, a date that moved. This is the prerequisite that makes everything else possible.
- Deterministic dates and identifiers
The prerequisite for every comparison in this cluster: making a document depend only on its input, when by default it depends on the clock and on a random source.
- Fixture data for documents
The sample data a document test renders, where a copy of a real record covers the easy case and nothing else.
- Boundary fixtures
Deliberately constructed data that lands content exactly at a page break, which is the only way to test the behaviour that breaks most often.
- Flaky document tests
The specific sources of non-determinism in a document pipeline, which are a short and identifiable list rather than a mystery.
- Testing across page sizes
Running the same document through each format it ships in, which is cheap and routinely skipped because the developer only ever sees one.
- Testing localised documents
Running the layout suite per locale, because translation changes string lengths and string lengths change the layout.
Running it in CI
Where the tests run, how long they take, what happens when one fails and who looks at it. A document suite that takes twenty minutes runs nightly, and a test that runs nightly catches a regression a day after it shipped.
- CI without a browser in the image
Testing a rendering pipeline on a runner that has no renderer, which is the normal situation once rendering is a service rather than a dependency.
- Testing pagination without rendering
Asserting on the rules that govern page behaviour rather than on the pages themselves, which is instant and catches the omissions rather than the outcomes.
- Testing the data contract
Asserting that the data reaching a template has the fields the template reads, which catches the failure that produces a document full of blanks.
- Testing that fonts are embedded
Checking the font list inside the finished PDF, which is a one-line assertion against the failure that changes every page and raises no error.
- Testing links and annotations
Checking the link annotations in the finished document, which are invisible to text extraction and to a pixel comparison alike.
- Testing document accessibility
Checking the structural properties a machine can see, and being clear that the property which matters most is not one of them.
- Testing the error paths
Exercising the failure branches of a document pipeline, which are the least-tested code in most systems and the code that runs on the worst day.
- Testing stored documents and expiry
Exercising the store-and-retrieve path, including the part that only happens after time passes, which is the part nobody tests.
- Testing a document that is emailed
Asserting on what actually leaves the system, since a correct document attached to the wrong message, or to the wrong recipient, is a worse failure than a malformed one.
- Pull request against nightly
Splitting the suite by cost so the fast tier fits in a pull request, because a test that reports a day late reports to nobody.
- Keeping document test runs fast
The specific levers that reduce the cost of a rendering suite, which are different from the ones that speed up ordinary tests.
- Debugging a failing document test
What a failure has to emit so somebody can act on it from the CI log, since reproducing a document failure locally is expensive and often impossible.
- Load testing a render endpoint
Measuring what your pipeline does under the load it will actually see, which for document generation is almost never a steady rate.
If you only do three things
Freeze the clock and the identifier source, so two runs of the same input produce the same document. Assert an exact page count on a fixture long enough to paginate, which is one integer and catches a large share of layout regressions. And assert on extracted text for the values that would cost you money if they were wrong: the total, the reference, the name. That is one render and three assertions, and it covers more than most document suites manage with far more machinery.