Automated Accessibility Testing: What It Catches and What It Misses

If you read our marketing copy, you will find this sentence: "Automated tools catch around 30-40% of WCAG issues."

We say this because it is true, and because we think you should know it before you pay for a scanner.

This post breaks down exactly what falls in that 30-40%, what falls outside it, and what the practical implications are for your accessibility programme.

Where the 30-40% number comes from

WCAG 2.2 Level AA contains 55 success criteria. Of those:

Roughly 20-25 can be fully tested by automation. These produce definitive pass/fail results with no human judgement required.
Roughly 15-20 can be partially tested. Automation can flag potential issues, but a human must confirm whether the issue is real.
Roughly 10-15 cannot be tested by automation at all. They require understanding context, intent, or the experience of a real user.

The "30-40%" figure refers to the proportion of real-world accessibility barriers on a typical site that automated tools can definitively identify. The exact percentage varies by site complexity, content type, and which tool you use.

What automation catches well

These are the categories where automated scanning is reliable and comprehensive:

Structural HTML issues

Missing or empty alt attributes on images
Form inputs without associated labels
Missing document language declaration
Duplicate element IDs
Empty headings, links, and buttons
Improper heading hierarchy (skipping levels)
Missing page titles
Tables without headers

ARIA issues

Invalid ARIA roles and attributes
ARIA references pointing to non-existent IDs
Redundant ARIA roles on native HTML elements
Missing required ARIA properties
ARIA hidden applied to focusable elements

Visual design issues

Text colour contrast below 4.5:1 (normal text) or 3:1 (large text)
Non-text contrast below 3:1 for UI components
Target sizes below the 24x24 CSS pixel minimum

Media issues

Autoplaying audio or video without a pause mechanism
Missing captions on embedded video (detection of caption tracks, not quality)

These categories account for a significant majority of the issues found on most websites. If your site has never been scanned, running an automated tool will likely surface dozens of real, fixable problems.

What automation catches partially

These are areas where tools can flag potential issues but cannot make a definitive judgement:

Alt text quality

Automation can detect that an image has alt text. It cannot determine whether that alt text is meaningful. An image with alt="image" technically passes the "alt attribute present" check but fails the intent of WCAG 1.1.1. Lumi's AI analysis engine attempts to evaluate alt text quality, but this is inherently less reliable than checking for presence.

Keyboard accessibility

Most scanners check whether interactive elements are in the tab order. Fewer actually tab through the page. Lumi's keyboard engine does navigate pages, but it cannot test every possible interaction path, especially in complex single-page applications with conditional modals and dynamically loaded content.

Focus management

Automation can check whether a focus indicator exists. It is harder to determine whether that indicator is sufficient - whether a user can actually see it against the specific background, whether it appears on all interactive states, and whether focus is managed correctly when dynamic content appears.

Error handling

Automation can check whether error messages exist and are associated with form fields. It cannot determine whether the error message is actually helpful. "Invalid input" passes the automated check but fails the user.

What automation cannot catch

These categories require human testing, typically with assistive technology:

Screen reader experience

Does the page make sense when read linearly? Is the reading order logical? Do announcements accurately convey what is on screen? Does the page provide adequate context for someone who cannot see the layout? No automated tool can answer these questions because they require understanding the content's meaning.

Cognitive accessibility

Is the content written in plain language? Are instructions clear? Is the navigation predictable? Are processes broken into manageable steps? These criteria (WCAG 3.1.5, 3.2.3, 3.2.4) require human judgement about the audience and context.

Meaningful alternatives

Is the audio description on a video adequate? Does the long description of a complex chart actually convey the data? Is the transcript of a podcast accurate? Automation can check that these alternatives exist but not that they work.

Interaction quality

Does a custom widget behave as expected? Does a drag-and-drop interface provide an accessible alternative? Can a user complete a multi-step process without losing context? These require testing with real assistive technology on real devices.

Context-dependent decisions

Should this image be decorative (alt="") or informative? It depends on the surrounding content. Is this link text descriptive enough? It depends on the page context. Automation cannot make judgement calls that depend on meaning.

Implications for your accessibility programme

Understanding these boundaries leads to a practical strategy:

Use automated scanning for what it does well. Run it continuously, on every deployment, across every page. The 30-40% of issues it catches are real barriers that affect real users, and they are the easiest and cheapest to fix. A multi-engine scanner like Lumi catches more of this 30-40% than a single-engine tool.

Don't stop at automated scanning. If your organisation has legal, regulatory, or ethical obligations around accessibility (and in 2026 most do), automated scanning alone is not sufficient. Pair it with manual testing.

Manual testing does not mean testing everything manually. The efficient approach is: let automation handle what it can, then focus human testing time on the areas automation cannot cover. This means screen reader testing, keyboard flow testing, and content quality review.

The two are complementary, not competing. Automated scanning runs continuously and catches regressions immediately. Manual audits run periodically (quarterly or biannually) and catch the deeper issues. Neither replaces the other.

What Lumi does about this

We built a five-engine scanner specifically to maximise the coverage within that 30-40% automated window. Where a single engine might catch 20-25% of all issues, five engines with deduplication push closer to the 35-40% ceiling.

But we do not claim to catch everything, and we will not.

For the remaining 60-70%, the Livana team runs manual audits with NVDA, JAWS, VoiceOver, and keyboard-only testing. The combination of continuous automated monitoring and periodic human audit is the most effective accessibility programme we have seen.

Scan your site free at lumi.livana.io to see what automation can find. Then decide what comes next.