A wide range of tools can help you find accessibility issues, from browser plugins and website scanners to JavaScript libraries that analyze your codebase. Each tool has its own strengths and weaknesses when it comes to catching accessibility issues. Sparkbox’s goal is to research a variety of accessibility tools and empower anyone to run accessibility audits. In this article, I will review Pa11y by using it on a testing site that has planned accessibility issues.
This review will rate the plugin based on the following areas:
Watch a walkthrough of the tool and a short review or keep scrolling to read the full review. Jump to the conclusion if you are in a hurry.
YouTube embeds track user data for advertising purposes. You can watch the video on YouTube if you prefer not to grant consent for YouTube embeds.
In order to fully vet accessibility tools like this one, Sparkbox created a demo site with intentional errors. This site is not related to or connected with any live content or organization.
Want updates from Sparkbox?
Sign up to receive our emails right to your inbox—like Sparkbox’s monthly newsletters or updates about specific topics like more accessibility tool reviews!
Ease of Setup: Exceeds Expectations (3/4)
Important information to note for this tool:Pa11y requires the use of a command line interface (CLI) such as Terminal or Windows Command Prompt for installation, testing, and reporting. If the command line is not something you are comfortable with, I would suggest choosing a different tool for your accessibility testing purposes, such as one that is available as a Google Chrome Extension. You can also follow this tutorial to learn how to get comfortable with the command line, which is a great skill and can be used to work with other tools such as Git.
Installation:Installing PA11y is relatively straightforward. It does require node.js for installation. You can install Node using either Homebrew or nvm package installers. I prefer Homebrew for this step myself, but you can use whatever you prefer for this step. Once Node is installed, only one command is required to install Pa11y and have it available for use npm install -g pa11y.
Availability:Although I only tested Pa11y on Mac OS, it is also available on Linux and Windows using a different Node package installation. Depending on whether you have Windows 10 or Windows 7, the complexity of installation varies.
Ease of Use: Outstanding (4/4)
Using Pa11y out of the box after installation is pretty simple. You can run a basic default audit with one command using only the website url you would like to audit, such as pa11y http://example.com. Testing against a local file can be accomplished using pa11y ./path/to/your/file.html. You can opt for the in-terminal report to print out to you on-screen, or you can use the --reporter flag to have it compile a .CSV file for you, for example pa11y --reporter csv http://example.com > report.csv.
Additional flags for different options are well documented and available for use. If you ever need a reminder of these, you can use the --help flag on the command line itself to get a list of the flags that are available and what they do in the command line. These flags allow you to make the reports more specific to a variety of factors. This flexibility makes Pa11y a more comprehensive toolset, but it also requires more knowledge and expertise for configuration.
Some things to note: Pa11y does also have a Pa11y Dashboard project that is actively maintained and gives Pa11y a user interface that someone who isn’t comfortable with command line could use. It would likely require someone with command line experience to set up. (I did not review this extension to Pa11y but wanted to mention it for those who may find it useful.)
There is also a desktop app called Koa11y that gives Pa11y a desktop application frontend. I did not test this application for this review, but it is free and open source just like Pa11y.
Reputable Sources: Outstanding (4/4)
Pa11y utilizes WCAG sources directly in its feedback. The report will list the WCAG guideline for the error in question directly in the report. Users can use that feedback and follow it directly to the guideline success criterion in question within the WCAG standard documentation.
Although it is important to note that the feedback can be a little difficult to understand at first, as it uses periods for spaces and underscores in place of periods in the text string. For example, if the report were to list this “WCAG2AA.Principle1.Guideline1_1.1_1_1.H37”, that would mean the guideline error is WCAG 2.0 AA Principle 1, Guideline 1.1, Success Criterion 1.1.1.
Accuracy of Feedback: Areas for Improvement (1/4)
With default “out of the box” configuration, this is Pa11y’s weakest category. Pa11y found a fair number of issues in our test site, and the feedback for those issues was accurate. But ultimately, it fell a bit short compared to other tools in terms of the number of issues it found, and it didn’t find anything we weren’t expecting it to find.
Pa11y also offers a wide range of configuration options that change what you are able to audit for, what you can exempt from your audit, and how it audits (such as a single page or the whole site). You can also use Pa11y as a runner with other tools such as axe. Configuring additional rules and runner resources allows you to create a more comprehensive setup that may give you a more informative audit feedback report than the default tool would.
Clarity of Feedback: Exceeds Expectations (3/4)
Pa11y offers very exact WCAG 2.1 criteria violation references and suggests changes for those issues. However, though it does make some suggestions, it does not do this for all issues. It also doesn’t always explain these fixes in a manner an accessibility novice would immediately understand. It does provide you with context of the element in violation as well as a text chain to map yourself to the element, which is okay, but not nearly as convenient as a clickable link directly to the element itself as some other tools offer.
Cost: Outstanding (4/4)
Pa11y is free, open source, and well-maintained. It doesn’t require any account creation, so it doesn’t collect any user information.
Strengths and Limitations: Exceeds Expectations (3/4)
Pa11y’s limitations lie pretty heavily in a user’s comfort level with the command line and their dedication to learning how to use Pa11y in a more complex manner. Configuring it in order to take advantage of all it has to offer can be very time-consuming.
Pa11y’s flexibility and scalability are its biggest strengths. It can be utilized straight out of the box with a default set of options as a one page url auditor. But it can also grow into a much more comprehensive tool that can audit multiple pages on demand while testing against specific criteria that you customize.
The documentation provided is a definite strength. Pa11y is a configuration-heavy toolset that can become somewhat complicated if you want to create your own audit specifications. The documentation anticipates this and provides great help. However, this can also be seen as a slight weakness as the amount of documentation and complexity can be intimidating until you get more comfortable with it. But since the documentation is pretty helpful and comprehensive, I would say the strengths here outweigh its intimidation factors.
Conclusion: 👍 Recommend (22/28)
While Pa11y may not be as convenient as some other tools that install via browser extension, it is not without its own merits. As a free option, it can fit any budget and does not come with user account access privacy concerns that may arise when considering browser extensions. The default configuration of Pa11y with single url auditing finds a decent amount of errors and is straightforward with its reporting. However, a prerequisite to Pa11y is command line use. This requirement does set this tool back when it comes to its ease of use and installation. If that is not a concern, Pa11y has a lot of additional configuration, functionality, and extensions available for customization. One such extension is Pa11y CI, which allows you to set up Pa11y to do automated testing for your site. Although Pa11y has some challenges with its prerequisites and configuration for more robust testing, its flexibility makes it a very strong tool in your accessibility toolbox.
Accessibility Issues Found
| Test Site Issue | Tools Should Find | People Should Find | Pa11y Found |
|---|---|---|---|
| Insufficient contrast in large text | ✅ | — | ✅ |
| Insufficient contrast in small text | ✅ | — | ✅ |
| Form labels not associated with inputs | ✅ | — | ✅ |
| Missing alt attribute | ✅ | — | ✅ |
| Missing lang attribute on html element | ✅ | — | ✅ |
| Missing title element | ✅ | — | ✅ |
| Landmark inside of a landmark | ✅ | — | ❌ |
| Heading hierarchy is incorrect | ✅ | — | ❌ |
| Unorder list missing containing ul tag | ✅ | — | ❌ |
| ID attributes with the same name | ✅ | — | ❌ |
| Target for click area is less than 44px by 44px | ✅ | — | ❌ |
| Duplicate link text (lots of “Click Here”s or “Learn More”s) | ✅ | — | ❌ |
| div soup | ✅ | — | ❌ |
| Funky tab order | — | ✅ | ❌ |
| Missing skip-to-content link | — | ✅ | ❌ |
| Using alt text on decorative images that don’t need it | — | ✅ | ❌ |
| Alt text with unhelpful text, not relevant, or no text when needed | — | ✅ | ❌ |
| Page title is not relative to the content on the page (missing) | — | ✅ | ❌ |
| Has technical jargon | — | ✅ | ❌ |
| Using only color to show error and success messages | — | ✅ | ❌ |
| Removed focus (either on certain objects or the entire site) | — | ✅ | ❌ |
| Form helper text not associated with inputs | — | ✅ | ❌ |
| Pop-up that doesn’t close when you tab through it | — | ✅ | ❌ |
