Important information to note about this tool AATT requires the use of a command line interface (CLI) such as Terminal or Windows Command Prompt for installation and testing. If this is not something you are familiar with, I suggest choosing a different tool such as one that is available as a browser extension (and would likely suggest using a different tool regardless, stay tuned!). You can also follow this tutorial to learn how to get comfortable with the command line, which is a great skill and can be used to work with other tools such as Git.
There are a few different ways to use AATT, but all require the use of the command line. The main two options are:
Cloning the project from Github and running it locally on your computer. The AATT documentation provides a command to input in the command line that will generate a link to a webpage running locally on your computer. From there, you can copy and paste the URL or HTML you’d like to test.
I’ve tested both ways for this article and will be reviewing the results of both.
This review will rate the extension based on the following areas:
Watch a walkthrough of the tool and a short review or keep scrolling to read the full review. Jump to the conclusion if you are in a hurry.
In order to fully vet accessibility tools like this one, Sparkbox created a demo site with intentional errors. This site is not related to or connected with any live content or organization.
Want updates from Sparkbox?
Sign up to receive our emails right to your inbox—like Sparkbox’s monthly newsletters or updates about specific topics like more accessibility tool reviews!
Ease of Setup
Browser: Meets Expectations (2/4)
AATT requires the user to clone the Github repository and run a terminal command to use either the browser or code integration versions of the tool. While this may be fairly easy for a developer who is familiar with Git to do, it causes a barrier for anyone who is not a developer or who is unfamiliar with Git and/or the command line. That issue aside, the documentation for downloading and running this tool locally is short and to the point, describing all the needed commands and steps.
Integration: Areas for Improvement (1/4)
Installing AATT through integration proved to be a bit of a challenge. They provide brief information on a number of different ways to integrate it within your project, but each explanation is quite limited. It does require Node.js for installation which you can install using either Homebrew or nvm package installers.
Ease of Use
Browser: Exceeds Expectations: (3/4)
Once installed, this tool has a very simple interface. The webpage it loads for you contains a single input for a URL and then a button to run the tests. There are also options for pasting in HTML rather than providing a URL. The results of the test are then displayed in an easy-to-read table on the same webpage.
Integration: Meets Expectations: (2/4)
The main issue with the integration output comes from the lack of documentation that was mentioned above. Without a list of available options or really any other documentation besides the one example given, it’s difficult to be sure that you are even using the tool correctly. While the tool description claims that you can utilize different accessibility testing suites, without knowing what your options are you can’t tailor the tool to your liking.
Both: Exceeds Expectations (3/4)
This tool is based on Axe, HTML CodeSniffer, and the Google Chrome accessibility developer tools. All of these references are based on WCAG 2.1. This tool also explicitly states that it is based on WCAG 2.1, although it uses the three tools mentioned to actually run its tests. Throughout my use of the tool, I did not find any inconsistencies with this claim.
Accuracy of Feedback
Browser: Meets Expectations (2/4)
AATT missed three to four of the 23 accessibility failures that we expect to be caught by automated testing tools, but what exact things are missed depends on which tool (Axe, HTML CodeSniffer, or Google Chrome) you choose to test the site.
Integration: Fail (0/4)
This tool failed this category to a point that I thought maybe I was using it incorrectly. When using the code integration method of AATT, it only caught two errors out of the 23 errors on the test webpage. It gave a lot of additional notices and suggestions about checking different things manually, which was a good addition, but the actual errors it found were few and far between.
To see all the test site issues that the AATT found or missed, skip to Accessibility Issues Found.
Clarity of Feedback
Browser: Meets Expectations (2/4)
The AATT browser tool clearly lists the errors but does not give a reference to WCAG as to why this is an error. It only provides HTML tags as a clue to where the error is located. This made it virtually impossible to find where the error was occurring on the page. It also does not list how many occurrences of an error there are. For example, it will say there is a color contrast error, but where and how many instances remains a mystery.
Integration: Meets Expectations (2/4)
When you run your code with the AATT code integration it outputs JSON data which proves difficult for us human users to understand. I could not get the JSON to parse properly when trying to use this within my own project. I had to print out the raw response because what the tool was sending back to me was not correctly formatted JSON. This also has the same issues as the browser tool in that it does not give how many occurrences or a location other than a bunch of HTML soup. One redeeming quality of the integration is that it does provide links to the WCAG principles that the errors are based on.
Both: Outstanding (4/4)
This tool is 100% free
Strengths and Limitations
Both: Areas for Improvement (1/4)
The two ways to use this tool that I tested were both lacking in clarity of results and accuracy. They also both required the user to download the code and run it locally—steps that may prove too difficult for users that do not have development knowledge. There are a good deal of easy-to-use, more accurate tools out there that I would recommend before this one.
Conclusion: 👎 Not Recommended for General Use (17/28 and 13/28)
After testing PayPal’s Automated Accessibility Testing Tool, we’ve come to the conclusion that it is not the greatest choice for accessibility testing. This tool requires the user to download the code and use the command line, causing a barrier to ease of use and installation for anyone unfamiliar with the command line or Git. Additionally, AATT failed to capture a majority of errors on our testing website, and the errors that it did find did not include sufficient information on where they were located or how many occurrences there were.
As mentioned earlier, there are plenty of easier-to-use and more accurate tools that we would encourage you to use, learn more by checking out our accessibility tool review series.