Diffing two PDFs meaningfully when the bytes always differ

Why the obvious approach does not work

Two PDFs of the same document differ in bytes for reasons that have nothing to do with what is on the page: a creation timestamp, a modification timestamp, a document identifier, the order objects happen to be written in, and how a font subset was built. So the question is not how to diff a PDF, it is which projection of a PDF you want to diff, and there are three worth knowing, each catching a different class of change.

What to do instead

Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.

• Diff extracted text to catch content changes. Cheapest, fastest, blind to layout, and the one to run on every commit.
• Diff the page count to catch pagination changes. One integer, effectively free, and it catches a large share of real regressions because almost anything that breaks layout also changes how much fits on a page.
• Diff rendered images per page to catch appearance changes. Expensive, sensitive to antialiasing differences between machines, and the only projection that sees a colour, a border or a shifted block.
• Never diff the raw bytes. There is no threshold at which that is informative.
• Run the three in that order and stop at the first failure, because a text difference explains a pixel difference and looking at the pixels first wastes the reviewer's time.
• Where a difference is expected, such as a date that legitimately changed, fix it in the fixture rather than in the comparison. A comparison with exceptions in it stops being a comparison.

In practice

Test code, with the thing people write instead kept in a comment where it is the more instructive half.

// Three projections, cheapest first. Stop at the first difference.
async function comparePdfs(a, b) {
  const [pagesA, pagesB] = [await pageCount(a), await pageCount(b)];
  if (pagesA !== pagesB) {
    return { level: "pages", detail: `${pagesA} vs ${pagesB}` };
  }

  const [textA, textB] = [await extractText(a), await extractText(b)];
  if (textA !== textB) {
    return { level: "text", detail: firstDifference(textA, textB) };
  }

  for (let p = 1; p <= pagesA; p += 1) {
    const diff = await compareImages(
      await renderPage(a, p),
      await renderPage(b, p),
      { threshold: 0.02 },          // tolerate antialiasing, not layout
    );
    if (diff.ratio > 0.001) return { level: "pixels", page: p, diff };
  }

  return null;   // same document by every projection that matters
}

/* What is deliberately not here: a byte comparison. A PDF carries a
   creation timestamp and a document identifier, so two renders of the
   same input never match and there is no threshold that helps. */

What people do instead

Going straight to pixel comparison because it catches everything. It does, including the antialiasing differences between the machine that made the baseline and the machine running the test, so the suite fails on a runner upgrade and everybody learns to re-baseline rather than to read the failure.

What a failure actually tells you

Which projection failed tells you where to look. A page-count difference is pagination. A text difference with the same page count is content. A pixel difference with identical text and page count is styling, which is the narrowest and most useful signal of the three, because it says the words and the length are right and something visual moved.

Frequently asked

Related testing techniques

The techniques that pair with this one, then the rest of the same group.