What can and cannot be asserted about a document's accessibility

Why the obvious approach does not work

Accessibility in a document rests on a structure tree that describes headings, lists, table cells and reading order. Some of that is machine checkable: whether a tree exists, whether a language is declared, whether images carry alternative text. The part that matters most is not. A checker can confirm a reading order exists and cannot confirm it matches the document, so a file where every paragraph is tagged as a heading passes cleanly and is useless. A suite that reports a green accessibility check is making a narrower claim than the word suggests.

What to do instead

Each of these is a decision with a wrong answer, and the wrong answer usually produces a suite that passes rather than one that fails.

• Assert the checkable properties in CI: a declared document language, alternative text on images that carry meaning, and a structure tree if one is required.
• Assert on the source markup as well, since semantic markup is the raw material anything downstream works from: real headings, real lists, real table headers.
• Be explicit in the test names about what is being claimed, so a green run is not read as more than it is.
• Keep reading order as a human review item on a schedule rather than pretending it is automatable, and put it on the documents that matter rather than all of them.
• Check what this API does and does not control before promising anything: there is no tagging option on the render, so whether output carries a structure tree is not something a caller sets here, and the honest test is to inspect a real file rather than to assume either answer.
• Where a conformance requirement exists, validate with a real validator on the finished file rather than asserting on the markup, because the requirement is about the file.

In practice

Test code, with the thing people write instead kept in a comment where it is the more instructive half.

// The checkable half, on the source markup.
test("markup is semantic", () => {
  const doc = parse(renderInvoice(fixtures.typical));

  expect(doc.querySelector("html")?.getAttribute("lang")).toBe("en-GB");

  for (const img of doc.querySelectorAll("img")) {
    // Decorative images declare themselves; meaningful ones describe themselves.
    const alt = img.getAttribute("alt");
    const decorative = img.getAttribute("role") === "presentation" || alt === "";
    expect(decorative || (alt && alt.length > 0)).toBe(true);
  }

  // Real table headers, not styled cells.
  for (const t of doc.querySelectorAll("table")) {
    expect(t.querySelectorAll("th").length).toBeGreaterThan(0);
  }
});

/* What this suite does not claim, stated so a green run is not
   over-read:
     that the reading order matches the document
     that a structure tree is present in the output, which is not
       something a render option controls here
     that the file satisfies a conformance standard, which needs a
       validator run against the finished file

   Reading order is a scheduled human review on the documents that
   matter. It is not automatable and pretending otherwise is worse than
   not checking.                                                      */

What people do instead

Reporting the automated check as accessibility coverage. It covers the mechanical half, the half a checker can see, and the half that determines whether a screen reader user can actually use the document is the other one.

What a failure actually tells you

A failure here is a real and cheap fix, usually a missing alt attribute or a language declaration. A pass means the mechanical properties are in place and says nothing about the experience.

Frequently asked

Related testing techniques

The techniques that pair with this one, then the rest of the same group.