BlogATAhon2021Entry
Accessibility Testing and Automation in DevOps env
by Vinda Parab
Accessibility Testing
To ensure web or mobile app is accessible to its every user in true sense, it is very important to cater to the needs of all including the differently abled users. Accessibility issues can be identified and thereby addressed through accessibility testing ensuring app is accessible to everyone including those with disabilities such as auditory, physical, speech, cognitive and neurological. Accessibility testing is also important to ensure legal compliance such as:
US federal government’s Section 508 legislation
Americans with Disabilities Act of 1990 (ADA), as amended
Refer Web Accessibility Laws & Policies for world-wide governmental policies related to web accessibility.
Web Content Accessibility Guidelines (WCAG)
Accessibility of an application is measured based upon the internationally recognized W3C’s Web Content Accessibility Guidelines (WCAG).
WCAG 2.0
WCAG 2.0 are organized as per 4 Principles, 12 Guidelines and 61 Testable Success Criteria.
WCAG 2.0 has 3 Priority Conformance Levels: A[25], AA[13] and AAA[23]
- A: Essential: nonconformance to this means, assistive technology may not be able to read, understand, or fully operate the page/ view.
- AA: Required for multiple government and public body websites.
- AAA: This is typically reserved for parts of websites and web apps that serve a specialized audience.
WCAG 2.0 Principles and Guidelines
PRINCIPLES [4] | Perceivable Web content is made available to the senses - sight, hearing, and/or touch | Operable Interface forms, controls, and navigation are operable | Understandable Information and the operation of user interface must be understandable | Robust Content can be interpreted by a wide variety of user agents, including assistive technologies |
GUIDELINES [12] | 1.1 Text Alternatives 1.2 Time-based Media 1.3 Adaptable 1.4 Distinguishable | 2.1 Keyboard Accessible 2.2 Enough Time 2.3 Seizures 2.4 Navigable | 3.1 Readable 3.2 Predictable 3.3 Input Assistance | 4.1 Compatible |
TESTABLE SUCCESS CRITERIA [61] | Total: 22 A[9] AA[5] AAA[8] | Total: 20 A[9] AA[3] AAA[8] | Total: 17 A[5] AA[5] AAA[7] | Total: 2 A[2] |
FOCUS AREAS | Sight, Hearing, Touch, Transformability, Content vs. Style and Presentation | Input Methods, Interaction Methods, User Control Over Timing and Time Limits, Error Recovery | Meaning, Language, Alternative or supplemental representations such as summary, illustrations etc., Functionality | Functionality Across Current and Future Technologies, Using Technologies According to Specification |
WCAG 2.1
While WCAG 2.0 remains a W3C Recommendation, the W3C advises the use of WCAG 2.1 to maximize future applicability of accessibility efforts.
WCAG 2.1 extends WCAG 2.0 by adding 17 new success criteria around:
- Provision for mobile
- Provision for low vision
- Provision for cognitive and learning disabilities
- Conformance with EN 301 549
Accessibility Testing of Mobile Apps
With the growing popularity of mobile devices, it is critical to test mobile content that can be deployed to users through native apps, mobile websites, or hybrid apps. The WCAG for desktop also applies to Mobile devices. Widely used mobile operating systems, iOS and android provides multiple built-in accessibility features, these features should be utilized for extending accessibility testing.
Challenges with Accessibility Testing
The 2 major challenges with accessibility testing are:
Evaluation in accordance with WCAG guidelines requires considerable efforts and time especially for large and dynamic applications and wide range of mobile devices.
Adding an accessibility automation tool, though saves on manual efforts and time, on an average only 57% of WCAG A and AA conformance can be validated using automation tool.
Automating Accessibility Testing
Multiple open-source as well commercial tools are available for automating accessibility testing. Organizations are utilizing multiple accessibility test tools to supplement manual checks for evaluating an application accessibility. Automation tools generally identify potential accessibility issues, as well provide remediation/s suggestions.
Given the wide range of compliances the application needs to adhere to meet the WCAG, it has been observed that automation accounts for around 50% of the reported issues.
Automating Accessibility Testing in the DevOps env
An old software engineering saying, “Test early, test often” and this holds true for accessibility testing too. In modern world where the delivery is at its fast pace, it is even more critical to test an application for accessibility early by leveraging the automation tool/s. With this the developer get the clear feedback on the most critical accessibility issues right at the beginning of development thereby handling those in time in addition to taking due care in future development. Also, the issues reported with the automation tools can be taken care prior to passing the build for test, thereby reducing on further test efforts.
Accessibility Test Automation for Continuous Integration using Lighthouse
Lighthouse CI (LHCI) is Google Chrome’s free and open-source suite of tools for Continuous Integration.
Integrating LHCI in CI pipeline, ensures it will run on project’s urls on every commit, automatically assert Lighthouse audits pass, and upload the reports for manual inspection.
Along with accessibility checks, LHCI also analyzes Performance, SEO, Best Practices and Progressive Wed App in one go.
By default, LHCI audits only the list of urls configured for assessment and for now doesn’t support authentication mechanism directly. These 2 aspects can be easily enhanced by adding 2 major functionalities to adhere to our testing needs. There are multiple ways to handle these 2 aspects. Listing here the options incorporated by me in our project.
- Handling of pages behind authentication.
- To cater this needs a puppeteer script is utilized to handle the login.
- The control is first passed to puppeteer to perform all the perquisite actions before the LH audit begins.
- Auditing of dynamically crawled list.
- A web crawler I utilized to crawl the domain of the test site which generates a text file with list of crawled urls. This text file is than served as input to LHCI.
Accomplishments
- For every commit – analyze the impact of change at an early stage in the build cycle
- Lighthouse CI asserts threshold fail builds if they don’t meet pre-defined criteria
- Configuration as a code and hence can be tracked through version control
- Accessibility and performance in one test
- Audit pages behind authentication as well audit dynamically crawled web pages
- View and analyze current and historical reports on locally hosted LHCI server
- Identify regressions in category score or metric performance to specific commits
- View detailed diffs between any two Lighthouse reports to find the root cause of a regression
Conclusion
To deliver an accessible experience that reassures everyone user, accessibility testing should be considered at an early stage and on regular basis during software development.
Automated accessibility tests supplement the manual assessment. Integrating accessibility testing in CI pipeline, ensures it will run on project’s urls on every commit, automatically assert and provide reports for manual inspection.
Our ability to reach unity in diversity will be the beauty and the test of our civilization.
~ Mahatma Gandhi
As an accessibility tester, let’s contribute making our world more inclusive.
References
https://github.com/GoogleChrome/lighthouse-ci
https://www.w3.org/WAI/standards-guidelines/wcag/
https://www.w3.org/WAI/WCAG21/Understanding/intro#understanding-the-four-principles-of-accessibility
About Author
Vinda Parab
Sr. Lead Engineer Quality, Persistent Systems
Experienced automation and manual quality engineer. At present, deep diving into next gen AI/ML based automation tools. Focussed on enabling performance, accessibility, and security into DevOps env mainly through free and open-source tools.