Accessibility testing ensures software applications and websites can be used by people with disabilities. With over 1 billion people worldwide living with disabilities, accessibility is crucial.
Globally, over 2.2 billion people have vision impairment or blindness. Another 466 million people have disabling hearing loss. [1] Beyond sensory disabilities, between 93 and 150 million people globally live with intellectual disabilities. [2]
These large disabled demographics rely on accessible digital experiences. For example, 62% of US adults with disabilities say they own desktops or laptops. [3] Ensuring websites and apps work for assistive technologies like screen readers is critical.
| Type of disability | Estimated global population |
|---|---|
| Vision impairment or blindness | 2.2 billion [1] |
| Hearing loss | 466 million [1] |
| Intellectual disability | 93 – 150 million [2] |
Table 1. Global disability demographics and statistics
Traditionally, accessibility testing was conducted manually. But with today‘s complex web applications, manual testing is no longer scalable or efficient. This is where test automation comes in.
In this guide, we‘ll explore what automated accessibility testing is, its benefits, how to implement it, top use cases, limitations, and the future of accessibility test automation.
What is Accessibility Testing?
Accessibility testing evaluates how easily people with disabilities can use a software application or website. It aims to identify and remove barriers that prevent access for people with:
- Vision impairments: Cannot fully see content. Rely on screen readers.
- Hearing impairments: Cannot hear audio content. Require captions.
- Motor/mobility disabilities: Cannot use a mouse or keyboard. Need voice control.
- Cognitive disabilities: Have trouble understanding navigation. Require clear simple interfaces.
Testing checks compliance with standards like WCAG 2.1. This involves testing:
- User interfaces – buttons, forms, navigation
- Presentation/styling – color contrast, text legibility
- Content – images, tables, multimedia
- Functionality – all features usable without vision/hearing/mobility
Why Automate Accessibility Testing?
Automating accessibility testing has several key benefits:
- Efficiency: Automation tests faster than manual testing. Batch automation can test 1000s of pages in hours.
- Accuracy: Automated tests are consistent and can catch more issues. They don‘t get tired or miss steps.
- Cost savings: Automation reduces manual testing effort. Fully manual testing of large sites would be cost-prohibitive.
- Early feedback: Issues caught earlier in CI/CD pipelines are cheaper to fix.
- Ongoing regression testing: Automation enables continuous accessibility monitoring.
Large complex web apps like SaaS admin UIs can have 1000s of dynamic pages. Manual testing of every component combination is impractical. Automation makes continuous accessibility testing at scale feasible.
Sample Automated Accessibility Test Types
Here are some examples of common automated accessibility test cases:
- Color contrast: Programmatically check text colors vs background colors meet minimum ratios.
- Headings hierarchy: Validate heading tags used sequentially (e.g. no h4 without h3).
- ARIA attribute validation: Check custom UI components use ARIA roles/states correctly.
- Link text: Ensure links make sense out of context of surrounding text.
- Keyboard navigation: Test all UI elements can receive focus and are operable via keyboard.
- Image alt text: Confirm all informative images have descriptive alt text.
- Form labels: Validate all form inputs have associated labels.
These tests can be scripted to run continuously against an application as it is developed.
How to Automate Accessibility Testing
Here are the key steps to implement automated accessibility testing:
Choose a Testing Tool
Many tools are available like Axe, Google Lighthouse, and Wave. Compare tools to pick one fitting your tech stack and testing needs.
Configure Testing Environment
Install, integrate, and configure the testing tool in your development environment. Set up needed dependencies and libraries. For example, Axe can be added to Jest or Cypress for automated testing.
Identify Test Scenarios
Decide what aspects of accessibility to test. Examples include keyboard navigation, color contrast, screen reader compatibility, etc. Reference standards like WCAG 2.1 to ensure coverage.
Write Automated Tests
Write scripts to automatically run tests using the tool‘s API. Tests should programmatically check for compliance and report results like pass/fail/warning.
For example, this Lighthouse Node.js snippet runs accessibility audits:
const lighthouse = require(‘lighthouse‘);
const sitemap = [‘/‘, ‘/about‘];
sitemap.forEach(page => {
const opts = {
accessibility: {
auditRefs: [
‘accesskeys‘,
‘aria-allowed-attr‘,
‘aria-required-attr‘
]
}
}
lighthouse(page, opts).then(results => {
// log results
});
});Integrate into CI/CD
Include automated accessibility tests in continuous integration and delivery pipelines via GitHub Actions, Jenkins, etc. This bakes accessibility testing into the software delivery lifecycle.
For example, this Jenkins declarative pipeline fragment runs Lighthouse on each Git commit:
node {
stage(‘Lighthouse Accessibility Tests‘) {
sh "lighthouse http://myapp.com --config-path=lighthouserc.json --output-path=report.html"
// additional steps to analyze report
}
}Analyze Results and Fix Issues
Review test reports to understand failures and create issues to fix them. Trace failures back to code locations. Prioritize failures by severity.
For example, Axe produces detailed reports like:
Axe accessibility testing report with violation details
Top Use Cases for Automated Accessibility Testing
Some top use cases where automation delivers value:
Regression Testing
Run with each build to catch accessibility regressions early. Alert if UI changes introduce new issues.
For example, adding a new JavaScript framework might break keyboard navigation. Automated regression testing will catch this quickly before users are impacted.
Large Scale Testing
Test large complex web apps like SaaS platforms where manual testing alone would be impractical and inadequate.
For example, testing a university website with thousands of pages across different sites. Automation can spider the entire site in hours.
Compliance Testing
Schedule periodic runs to verify all pages conform to standards like Section 508 and WCAG 2.1 criteria. Alert if compliance drops.
For example, set up nightly automated WCAG 2.1 AA level assessments across an e-commerce site to maintain compliance.
Agile/CI Testing
Include in continuous integration workflows so accessibility is continually measured with each code change. Alert developers of new issues before merging PRs.
For example, require passing accessibility tests before merging feature branch pull requests into main.
Challenges and Limitations of Automation
Automated testing alone isn‘t sufficient for full accessibility testing. Challenges include:
-
Incomplete test coverage: Manual testing required to catch subtle user interaction issues automation might miss.
-
JavaScript heavy apps: Dynamic complex UIs are harder to auto-crawl and test accurately.
-
False positives: Some automatically flagged issues aren‘t meaningful real-world problems.
-
Reporting limitations: Tools may lack details needed for efficient debugging/issue prioritization.
-
Talent gaps: Developer and QA training required on core accessibility principles.
-
Design thinking: Automated testing focuses on technical compliance. Holistic accessible design requires a deeper human-centric approach.
Automation complements rather than replaces manual and user testing. Combining testing approaches is ideal to ensure a site functions properly for all users.
Implementing Automated Accessibility Testing
Here is a sample workflow for rolling out automated accessibility tests:
-
Document overall site accessibility compliance goals and target standards/regulations.
-
Research leading accessibility testing tools, compare, and select tool(s) to pilot.
-
Install, configure, and integrate tools into development pipelines.
-
Identify critical site pages and flows for initial test coverage.
-
Develop automated test cases mapped to target compliance criteria.
-
Review and analyze initial automated test results. Tune tests and thresholds as needed.
-
Fix high priority errors and warnings. Track ongoing compliance metrics.
-
Expand test coverage and playbooks for addressing common failures.
-
Train developers, designers, and QA staff on core accessibility principles.
-
Complement with manual testing during development and via ongoing user research.
The Future of Accessibility Testing Automation
Accessibility testing tools continue advancing. Emerging capabilities include:
Smarter automation via AI and ML to better test complex dynamic interfaces and identify subtle issues.
Mobile app testing support to automate accessibility on Android, iOS, and hybrid apps.
Author focused tools to provide real-time design feedback and assist with creation of accessible components.
Cognitive accessibility checking interfaces for simplified navigation, clear messaging, plain language, and ease of understanding.
Accessibility performance monitoring to continually measure how real users with disabilities experience digital properties.
As technology improves, automated testing will become faster, smarter, and more comprehensive. But human insight will still be critical to ensure accessibility for all.
References
-
World Health Organization, Disability and health
-
Centers for Disease Control and Prevention, Data and Statistics on Intellectual Disability
-
Pew Research Center, Americans with disabilities less likely than those without to own some digital devices
Conclusion
Automated accessibility testing allows large complex apps to be continuously assessed for compliance. This prevents accessibility regressions and ensures an inclusive user experience.
However, automation is only one piece. Training, better design thinking, manual testing and user research are also key to create truly accessible products.
As tools continue maturing, automated testing will become faster, smarter and more accurate. This will further shift accessibility left – making it a natural part of building excellent software for all.