Testing the delivery of a document, not just its generation

Why the obvious approach does not work

Document suites stop at the bytes. Delivery is a separate system, usually tested separately if at all, and the join between them is where the expensive failures live: the right document sent to the wrong address, the wrong document attached to the right message, an attachment named in a way that reveals something, or a document inlined into a mail body where it can be forwarded without the surrounding context. None of those are defects in the document, and all of them are defects in the thing the customer receives.

What to do instead

Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.

• Assert on the outbound message as a whole: recipient, subject, body and attachment together, in one test, because the failure is in the combination.
• Assert the attachment is the document for the recipient it is going to. A test that checks an attachment exists will pass while the wrong one is attached.
• Assert the filename separately, since it appears in the recipient's mail client and in their downloads folder and travels further than the message does.
• Test the batch case, where several documents are generated and several messages sent, because that is where a loop variable error sends everybody the same document.
• Capture messages in a test double rather than sending them, and assert on the captured set rather than on a count.
• Test the failure ordering: whether the document is generated before the message is sent, and what happens if the second fails. Sending a message that references an attachment that was never made is a common and visible defect.

In practice

Test code, with the thing people write instead kept in a comment where it is the more instructive half.

test("each customer gets their own invoice", async () => {
  const invoices = [inv("A", "a@example.com"), inv("B", "b@example.com")];
  await sendInvoiceBatch(invoices);

  const sent = mailer.captured();
  expect(sent).toHaveLength(2);

  // The failure this catches: a loop variable error sending everybody
  // the same document. A test that only checks "an attachment exists"
  // passes while that happens.
  for (const invoice of invoices) {
    const msg = sent.find((m) => m.to === invoice.customerEmail);
    expect(msg).toBeDefined();
    const text = await extractText(msg.attachments[0].content);
    expect(text).toContain(invoice.number);
    expect(text).not.toContain(other(invoices, invoice).number);
  }
});

test("the attachment filename reveals nothing", async () => {
  await sendInvoice(inv("A", "a@example.com"));
  const [msg] = mailer.captured();
  expect(msg.attachments[0].filename).toBe("invoice-2026-118.pdf");
  expect(msg.attachments[0].filename).not.toMatch(/@|\bacct\b|\d{8,}/);
});

test("no message is sent if the document could not be made", async () => {
  client.post.mockRejectedValue(new ApiError(503, "renderer_unavailable"));
  await expect(sendInvoice(inv("A", "a@example.com"))).rejects.toThrow();
  expect(mailer.captured()).toHaveLength(0);
});

What people do instead

Testing generation and delivery in separate suites that never meet. Each passes, the join between them is untested, and the resulting defect is the one class where a customer receives somebody else's document.

What a failure actually tells you

A cross-matched attachment in this test is the single most serious failure in the cluster. It is a disclosure rather than a layout problem, and it is worth treating as a different severity from everything else here.

Frequently asked

Related testing techniques

The techniques that pair with this one, then the rest of the same group.