What your test report doesn't say

A green report is silent about what it skipped

Picture the report that lands before a release: hundreds of checks, every one passed, one green bar. It reads like a verdict on the product. It is really a verdict on the questions somebody thought to ask. Whatever nobody thought to ask — the checkout in a second currency, the form a keyboard user cannot reach, the page in the browser nobody opened — is not in the report at all. Not red, not green. Absent.

That absence is the problem. A reader sees a clean report and fills the silence with the most comfortable assumption: that everything important was covered. Nobody lied. The report simply had no way to say "and here is what we did not look at", so it did not.

Testing has never been able to prove absence

This is not a new insight. In one of his early notes on program reliability, Edsger Dijkstra wrote that testing can be used very effectively "to show the presence of bugs but never to show their absence". Every check is a sample. A thousand passing samples make a strong statement about the thousand things sampled and no statement at all about the rest.

Teams know this in principle. What gets lost is the boundary. Over a few release cycles, "the suite passed" quietly turns into "the product works", and the edge of what was sampled stops being visible to anyone who did not write the suite.

One team measured the gap

In 2017 the UK Government Digital Service built a web page with 143 deliberate accessibility failures and ran ten automated testing tools against it. 29% of the barriers were found by none of the ten tools. Their conclusion was that such tools cannot be relied on by themselves, and work best combined with manual checking, an audit and user testing.

The useful lesson is not that the tools were weak. It is that a run of any one of them would have produced a report with nothing in it about the barriers it could not detect. The gap was only visible because the team knew the answer in advance. On your product, nobody does. We looked at the accessibility side of this in more detail in the WCAG checks automation cannot make.

"Not covered" is an answer

So we think a release report needs a section that most reports leave out: not covered. Not as small print, and not as a generic disclaimer, but as a specific list for this release — the flows, browsers, devices and requirements nothing checked this time.

That list does three things a green bar cannot. It tells the person signing off exactly what they are accepting on trust. It turns a vague unease ("did anyone look at the refund flow?") into a decision someone can make in a minute. And it keeps the tool honest: a checker that must list what it skipped cannot let a gap pass for a pass.

We hold ourselves to the same rule. ShipperAG runs its browser checks in Chromium, including phone and tablet sizes by emulation; Firefox, Safari and real devices are not covered, and our reports say so rather than leaving you to assume otherwise. If you want to see the idea on your own site today, our free website readiness check separates what a machine verified from what needs a person and what nothing checked.

What we're building

ShipperAG checks a release against what it is supposed to do, re-checks what it finds, and hands your team evidence it can sign — including a plain account of what was not covered. The product is in a private pilot through 2026; you can see how it works at a glance, or score your next release yourself with the free release readiness checklist.

Sources, checked 28 September 2026: Edsger W. Dijkstra, "On the reliability of programs" (EWD303), E. W. Dijkstra Archive, University of Texas at Austin; "What we found when we tested tools on the world's least-accessible webpage", GOV.UK Accessibility blog, 24 February 2017.