Automating Accessibility Testing for Android

By Nishanth Nagella

Accessibility is an important aspect of any application users interact with. By being considerate about how the app would behave with accessibility services, you improve your app's usability and make it particularly helpful for people with disabilities to use the app.

Broadly, there are three ways to test an application's accessibility.

1. Manual Testing: To test, we interact with the app using Accessibility services, as a person with disabilities would. On Android, Talk Back and Switch Access are the most commonly used services by the disabled.
2. Testing with Analysis Tools: Run the app manually through Testing tools to discover opportunities to improve your app's accessibility.
3. Automated Testing: Write automated tests which can run on CI environments to detect accessibility issues. This helps catch accessibility issues early on in the development cycle. This document mostly focuses on Automated Testing.

1. Manual Testing

Manual Accessibility Testing is an important step and can catch most issues, compared to the other two forms of testing. The Manual Accessibility Testing Guide from Android provides good information on the official tools available to enable testing.

2. Testing with Analysis Tools

There are a few analysis tools available that can identify and suggest accessibility improvements. Some of the tools are desktop software that connects to your test device and the others are mobile apps that run on top of your app. For all these tools to test your app, you need to manually run the app and go through all the screens that need testing.

The most popular and widely used tool in the Android ecosystem is the Accessibility Scanner App. These quick steps can get you started:

• Download the app from the Play Store and open it.
• In the device's Settings app, follow the prompts to turn on Accessibility Scanner.
  • Accessibility > Accessibility Scanner > Use service.
  • You will see a floating button on the screen, which is persistent all the time.
• Go to the app you wish to test, and click on the Floating button to start recording.
• Go through all the screens you wish to test and the service automatically scans everything in the background. When you are done with all the screens, press "Stop Recording".
• Open the Accessibility Scanner app and you should see the report. You can group the errors by screen and also export and share the report.

More details and step-by-step instructions on how to use this service can be found in this Android Accessibility Help Link.

3. Automated Testing

Google's Accessibility Test Framework for Android (ATF) is an open-source library that provides a good set of Accessibility Checks which can be used to automate testing. It is the underlying set of tools used by the official Google Accessibility Scanner app described above. Espresso internally uses these same ATF checks for performing Accessibility checks. With espresso, there is very limited customization possible, but by using ATF directly, we can pick the checks we want to run on each screen, extend them or write new checks from scratch.

There are a few other tools available in the market (both free and paid) for this, but considering the official support and the wide variety of Accessibility Checks it has, ATF is a good choice to get your app Automated for Accessibility. Since the underlying set of tools is the same, one other advantage of using ATF is that if a Test Case fails in automation, we could use the Accessibility Scanner app to get more details about the test failure.

Implementation

ATF Accessibility checks happen at the root view of every screen. At runtime, it traverses through all the child views to find any issues. If you have any Screen Tests within your app, ATF is easy to integrate. Once the Screen Test opens a screen, we can add these checks to be run on that screen. These ATF checks can run together with any other UI checks you might have on that screen.

The table below lists all the checks performed by the ATF framework. The Source Code for these checks can also be referred to for more details on how each of these checks is performed.

Check Name	Description	Guidelines and Notes
SpeakableTextPresentCheck	Check that items which require speakable text have it defined. Checks for Content Description, Hints, and Labels wherever necessary.	Content Labels
EditableContentDescCheck	Check to ensure that an editable TextView is not labeled by a Content Description. Defining an android:contentDescription on any EditText or editable TextView may interfere with an accessibility service's ability to describe, navigate, and interact with text that a user enters within the item.	Editable View Labels
TouchTargetSizeCheck	Check to ensure touch targets have a minimum size, 48x48dp by default.	Touch Target Size
DuplicateSpeakableTextCheck	If two Views in a hierarchy have the same speakable text, that could be confusing for users. WARNING and INFO. Not ERROR	Duplicate Descriptions
TextContrastCheck	Check that ensures text content is readable with sufficient contrast against its background.	Color Contrast
ClickableSpanCheck	Only for Android Versions below 8.0. In Android versions lower than 8.0 (API level 26), it may be difficult for some accessibility services to detect and activate ClickableSpan reliably.	Clickable Links
DuplicateClickableBoundsCheck	Developers sometimes have containers marked clickable when they don't process click events. When a container shares its bounds with a child view, that is a clear error. This class catches that case.	Duplicate Clickable Views
RedundantDescriptionCheck	Checks for speakable text that may contain redundant or inappropriate information. Throws a WARNING when any redundant and duplicate descriptions are found.	Items Labeled with Type or State
ImageContrastCheck	Check that ensures image foregrounds have sufficient contrast against their background	Color Contrast
ClassNameCheck	Checks that the type of the UI element (ViewHierarchyElement#getAccessibilityClassName()) is supported by accessibility services.	Unsupported Item Type
TraversalOrderCheck	Check to detect problems in the developer specified accessibility traversal ordering. Throws a WARNING message, if the order is not as expected.	Traversal Order
LinkPurposeUnclearCheck	Check to warn about a link (ClickableSpan) whose purpose is unclear.	Link Purpose Unclear
TextSizeCheck	Looks for text that may have visibility problems related to text scaling. Checks for min text size and developer errors of not using scalable text.	Text Scaling
UnexposedTextCheck	BETA: Still under development. Check for finding those OCR recognized texts which are not exposed to Accessibility service. The OCR results are provided via a Parameters object.

Using ATF Checks in Screen Tests

The ATF checks would run on top of the Screen Testing framework you may have set up for your app. Typically, in a Screen Test, you open up the screen you'd like to test, check that all the UI elements show up as expected, and pass/fail the test case accordingly. We mock all the other layers of the application to open up the screen for the exact use case you want to test.

We add a new Test below that takes the RootView of the fragment (Notes on Jetpack Compose screens in the next section) and runs all the Accessibility Tests on the entire View Hierarchy.

@RunWith(AndroidJUnit4::class)
class MyScreenComponentTests {

   @Test
   fun my_screen_accessibility_checks() {
       // Open the app to MyScreen. Get the RootView of the Fragment/Activity.
       runAccessibilityChecks(screenRootView)
   }

   private fun runAccessibilityChecks(view: View) {
       val checks = getAccessibilityChecksList()
       val hierarchy = AccessibilityHierarchyAndroid.newBuilder(view).build()
       val results = mutableListOf<AccessibilityHierarchyCheckResult>()
       checks.forEach {
           results.addAll(it.runCheckOnHierarchy(hierarchy))
       }
       val groupedResults = groupAndPrintResults(results)
       val errorResults =
           groupedResults[AccessibilityCheckResult.AccessibilityCheckResultType.ERROR]
       if (!errorResults.isNullOrEmpty()) {
           // We have accessibility errors, fail the test case with proper error message.
           val errorMessages = mutableListOf<String>()
           errorResults.forEach {
               val messageFromError = it.getMessage(Locale.getDefault())
               val logMessage =
                   "Check ${it.sourceCheckClass.simpleName} failed. On View with Bounds ${it.element?.boundsInScreen}. Error Message = $messageFromError"
               errorMessages.add(logMessage)
           }
           throw AssertionError("${errorResults.count()} Errors found: $errorMessages")
       }
   }

   private fun getAccessibilityChecksList(): List<AccessibilityHierarchyCheck> {
       // Here, you can choose to configure the list of checks you need to test on this screen.
       // You can extend any of the Check classes to customize it or just return all default checks like below.
       return AccessibilityCheckPreset.getAccessibilityHierarchyChecksForPreset(
           AccessibilityCheckPreset.LATEST
       ).toList()
   }

   /**
    * Utility method to group ATF results, and log them in an easy to read format.
    * @param results The results from an ATF run on a ViewHierarchy.
    * @return the passed results grouped by type of the result. The NOT_RUN result types are filtered out.
    */
   private fun groupAndPrintResults(results: List<AccessibilityHierarchyCheckResult>): Map<AccessibilityCheckResult.AccessibilityCheckResultType, List<AccessibilityHierarchyCheckResult>> {
       val groupedResults = results.groupBy { it.type }
       groupedResults.forEach { entry ->
           println(" ----- Accessibility Checks. Result Type ${entry.key.name}. Count = ${entry.value.count()}")
           if (entry.key == AccessibilityCheckResult.AccessibilityCheckResultType.NOT_RUN) {
               return@forEach
           }
           entry.value.forEach { result ->
               println(
                   "${entry.key.name}: Check ${result.sourceCheckClass.simpleName}. On View with Bounds ${result.element?.boundsInScreen}. Message =  ${
                       result.getMessage(
                           Locale.getDefault()
                       )
                   }"
               )
           }
       }
       return groupedResults
   }
}

Notes for Jetpack Compose Accessibility

Jetpack Compose is the new way of building native Android UI Components. When rendering UI components on the screen, it does NOT have the traditional android View and ViewGroup hierarchy.

All the Accessibility Tests described above need AccessibilityNodeInfo to be populated on each view. Each of the accessibility services (and in turn the testing tools described above) read the information in AccessibilityNodeInfo objects of every view to verify if it's good enough.

Jetpack Compose builds something called a semantic tree in parallel when it's rendering the composable. And when the accessibility services request the AccessibilityNodeInfo it constructs it from the semantic tree.

Limitations of ATF with Jetpack Compose

1. With Jetpack Compose, only the RootView of the fragment is a traditional view (ComposeView), and we can run these checks only on the RootView. (not very useful!)
2. Whenever there is a warning/error from the accessibility checks, the developer needs more info to evaluate and identify the exact Composable that caused this issue. But there are some challenges as described below:
3. The error/warning response from checks does not have a ViewId (Compose generates a random viewId, which cannot be traced back to the exact composable)
4. The accessibilityClassName is also created by Compose, from the Role defined in semantics. This ClassName does not always match the Compose which rendered it.
5. The bounds of the inaccessible view are returned, which can be used to pinpoint the inaccessible region of the RootView.

There is an open issue on the ATF GitHub Repo to add support for Compose screens. For Jetpack Compose screens there are tools like Deque Axe Mobile Tools which provide Automation Checks.

Final Words, TLDR

Making your app accessible is important. While there are tools out there to check your app for accessibility, automating accessibility tests gives you the advantage of catching some of these issues early and during development. If your app is built on Activities and Fragments, Google's ATF is the best framework to integrate and make your app better!

For more information about Mobile Accessibility, you can check out Mobile Accessibility at W3C