Comparing Visual Regression Testing Tools

I’ve previously written about visual regression testing for design systems, and I also discussed this topic in my UnConference lightning talk: Stay Cool & Never Change: What my visual regression tests wrote in my high school yearbook. In both, I mentioned a few visual regression testing tools that teams could use to help keep their design systems consistent and defect-free.

As more teams recognize the critical role that visual regression testing plays in their processes and design systems, the tools in this space have started to mature to meet the needs of the industry. Most notably, where most of the tools on the market previously only tested full webpages, we’re now seeing a rise in tools that can also test individual components—a boon for those of us working on design systems or component-based projects.

Let’s go over some of those changes and dig into a few of these tools with a little more detail.

Different Types of Tools

Let’s break these tools down into two categories: do it yourself (DIY) tools and software as a service (SaaS) tools. This breakdown allows us to differentiate based upon whether a tool is open source or proprietary software or if the tool is free, mostly free/freemium, or charges for any amount of usage.

I think it’s also worth mentioning that while I break tools down into those two categories, none of these tools are truly free—even if a piece of software doesn’t charge a fee for use, you or someone on your team will be spending time to set up, configure, maintain, and potentially host that tool. That’s not an endorsement of one category over the other, but it’s definitely worth considering as you evaluate which of these tools could be right for your team or your design system.

Also, while I’m limiting this article to a handful of tools in both categories, you should know that there are many, many tools in both the DIY and SaaS spaces. This article favors tools that are in the Node.js space, but almost all of the tools here can be used in non-Node.js projects.

Finally, you’ll notice that I continually mention a tool’s compatibility with Storybook. Not every design system or component-based project uses Storybook, but many of the design systems or component-based projects that I’ve worked on have used Storybook, and its popularity warranted my mentioning it below.

DIY Pros and Cons

If you’re just getting started with visual regression testing, or if you’re trying to evaluate how it might fit into your process, a great place to start is by trying out a free, open source tool. You get the benefit of finding out if that tool will fit your needs without having to sign up for a service or worry about pricing. Plus, if you truly need to configure one of these tools in a way that it hasn’t yet been configured, you have the power to fork the code and modify it to your liking—something that you can’t do with SaaS tools.

That said, as I mentioned above, relying on open source software can still come at a cost to you or your team in the form of support time, configuration time, and hosting.

SaaS Pros and Cons

For me, the best feature of the SaaS tools I discuss here is the user experience around reviewing, approving, and rejecting detected differences in screenshots. You’re not left to wonder or cobble together your own solution for what is, in my opinion, the most critical part of visual regression testing: getting people to actually use the tests in a meaningful way. Each of the tools listed below has a user friendly reviews process right out-of-the-box.

Cost is certainly a consideration, but each of these tools offers a free tier, which should be enough to let you evaluate its capabilities. Each tool’s pricing is based upon the number of screenshots taken per month, with a mix of other additional features as you escalate through their higher tiers. Depending on the size of your project, you might find that the free tier fits your needs!

Tools At-A-Glance

I’ve provided an overview of each tool in the following table for quick comparisons. Keep reading for a pricing table and more detailed explanations of the tools.

I’ve put together a chart to compare pricing, user limits, and the number of screenshots allowed for each SaaS tool. The tiers below represent the points at which each tool requires a larger plan.

Tools Details

Let’s take a deeper look at the visual regression testing tools (or jump to the one you’re interested in).

• BackstopJS

• Wraith

• WebdriverIO

• Percy

• Applitools

• Chromatic

BackstopJS

BackstopJS automates visual regression testing by rendering and capturing screenshots of your UI with Headless Chromium, and it also allows developers to simulate interactions in their tests with Puppeteer.

What’s Great About BackstopJS?

BackstopJS is a great way for developers and teams to get started with visual regression testing without having to lock into a paid contract. It’s actively maintained and, because it’s an open-source project, teams can modify it to fit their workflows.

BackstopJS also reports the results of your tests in two ways: via the command line and via an in-browser reporting dashboard that allows you to see any visual differences that might have been caught during the test runs.

What’s Not so Great about BackstopJS?

The first major drawback of BackstopJS is that you’re only able to test against Chrome. There is an open pull request (albeit, an older one) on the BackstopJS repository which would allow for testing in other browsers via the Webdriver protocol, but there doesn’t seem to be much movement toward a resolution for this.

Second, if you’re using Storybook to develop and showcase your components and want to test them in there, you’ll need to do some extra configuration to have BackstopJS load and test your Storybook components. This will take extra time because you’ll need to specify the paths to the stories that you want to test and possibly do other configurations to get your Storybook to play nicely with Backstop.

Cost

Free

Wraith

Wraith was created by a team at the BBC and uses PhantomJS, SlimerJS, or Chrome to create screenshots of webpages and their diffs. Wraith has been around for several years and is one of the first visual regression testing tools I remember hearing about. We wrote about our experiences with Wraith back in 2015, about a year after its initial release.

What’s great about Wraith?

Developers have praised Wraith’s ease-of-use and, once you get it up and running, Wraith will generate a gallery for the screenshots that it takes, allowing your team to review diffs in the browser. Based upon how you configure it, Wraith can also generate screenshots from different browsers.

What’s not so great about Wraith?

Unfortunately, it looks like the momentum behind Wraith has slowed considerably. I know that a project’s GitHub page doesn’t always tell the full story behind whether a project is actively being maintained (maybe there haven’t been any updates because the software is just that good!), but it does look like there are a number of unaddressed issues and pull requests waiting for someone’s attention.

Also, if you’re using Storybook, you’d likely have to come up with a custom solution to get Wraith and Storybook to play nicely together. So it’s on your team to get over the learning curve, and you’ll be investing time (aka money!) into getting Wraith set up, integrating with Storybook, and supporting it going forward.

Cost

Free

WebdriverIO

WebdriverIO bills itself as a “next-gen browser and mobile automation test framework for Node.js,” and it can be run on the Webdriver Protocol, which the W3C has taken steps to formalize as a cross-browser automation standard that drives and ensures interoperability on the web platform.

What’s great about WebdriverIO?

WebdriverIO is a feature-rich testing framework that allows its users to customize the types of tests they want to run and how they want to receive reports from those tests. The Webdriver Visual Regression Testing Service is just one aspect of the framework that provides similar functionality to the other tools I’m discussing here, and it makes WebdriverIO a great choice if you’re looking for a tool that will handle more than just visual regression testing.

Because it’s so feature-rich and the protocol is based upon a W3C recommendation, WebdriverIO is the most attractive DIY tool out of the three I’ve mentioned here. Having the backing of the OpenJS Foundation doesn’t hurt either and gives me a bit more confidence that it won’t become a stale open source project.

What’s not so great about Webdriver?

WebdriverIO might be too feature-rich for those who just want a visual regression testing tool. Additionally, if you’re wanting to test against Storybook, you’ll need to investigate a community boilerplate project to see if you can get it up and running.

Another potential downside is that after much searching, it’s unclear to me just exactly how the Webdriver Visual Regression Testing Service allows users to review detected diffs and what the approve/reject process is like once it detects diffs. The WebdriverIO documentation lists an alternative service called Novus Visual Regression Service, which allows users to upload screenshots to an open-source UI called Spectre, but this combination of tools is something that I have yet to try myself. If you’ve experimented with any of the WebdriverIO service combinations I mentioned above, we’d love to hear from you!

Cost

Free

Details

Percy

Percy touts itself as an “all-in-one visual review platform,” meaning that they want their product to be the place where your team manages everything around visual regression testing. With Percy, your team is going to have a dashboard to review screenshots, approve or reject detected changes, and keep your team up-to-date throughout the visual reviews process with comments, notifications, and commit status updates.

Percy was acquired by BrowserStack in July 2020—a natural fit for a visual regression testing tool. I’m looking forward to seeing what comes out of this new partnership, especially if it means that users can run visual regression tests against even more browsers. I’m also curious to see if there will be pricing tiers that allow current BrowserStack customers to bundle their existing browser testing subscriptions with the visual regression testing that Percy provides.

What’s great about Percy?

Not only does Percy offer SDKs for a variety of frameworks (Rails, React, Vue, Angular, and more), Percy also integrates with all of the major continuous integration and continuous deployment services. And if your project is using an end-to-end testing framework, Percy has you covered with SDKs for Cypress, TestCafe, WebdriverIO, and more. So if your team is already leaning on the aforementioned WebdriverIO for other automated browser tests, you can lean on Percy to handle your visual regression tests.

Just like the other two SaaS tools that I mention in this article, Percy also has an excellent dashboard where users can review the results of their visual regression tests, approve or reject the results, and request changes if necessary.

If your project is using Storybook, Percy offers component-level testing via Storybook for Ember, Angular, Vue, and React-based projects.

What’s not so great about Percy?

Right now, Percy only supports visual regression testing against Chrome and Firefox. However, Percy’s documentation notes that support for more browsers is in the works. No timeframe for that support is noted, but the BrowserStack acquisition gives me hope that Percy will allow for testing against many, many more browsers in the future.

Cost

Percy’s pricing is split into tiers based on usage and features.

The free tier gives users up to 5,000 screenshots per month and includes unlimited users, cross-browser support, responsive visual regression testing, and a 30-day build history.

The “professional” tier is $99 per month, gives users up to 25,000 screenshots per month, and includes all of the features that the free tier does, plus a 1-year build history.

Percy also has an enterprise level that includes things like single sign-on and a customer service manager. For pricing information, users will have to contact the BrowserStack sales team.

Applitools

Applitools offers a wide-range of visual regression testing integrations and isn’t limited to just testing full web pages or testing components: it can handle both via its large number of language and test runner SDKs or through Storybook with its Eyes Storybook package.

What’s great about Applitools?

The project I’m currently working on uses Applitools to test our components in Storybook, and the experience has been really great. We’ve integrated our visual regression tests with our continuous integration service, and Applitools runs against each pull request that is issued against our repository.

Similar to Percy, Applitools has a reviews dashboard where users review, approve, and reject screenshots, and the workflow around handling these reviews with each pull request has been efficient and valuable for our team.

In my experience, customer support has been great. I’m not a person who loves contacting companies to find out more about their products or to get a demo, but the limited interactions that I have had with the folks at Applitools have been fantastic.

What’s not so great about Applitools?

The pricing for Applitools is a mystery. Whereas the other tools I’ve discussed here have their pricing tiers published on their websites, you have to contact Applitools to find out what their service could cost your team.

The reviews dashboard isn’t as streamlined as Percy’s, and the amount of information shown can be a little overwhelming. I’ve enjoyed using the product, but when I go into the dashboard to review diffs, I find myself clicking around a lot and wondering if things couldn’t be just a bit more smooth.

Cost

As I mentioned above, you have to contact Applitools to get information about their pricing. Their pricing page lists 4 tiers: free, starter, eyes, and ultrafast test cloud. The tiers have varying degrees of features, most notably the amount of users allowed per tier.

Unless you’re working solo and only need a very limited number of screenshots per month (100), you’re going to need to subscribe to the “starter” tier at the very least. Buying into the two upper tiers gets you things like enterprise support, unlimited users, 20 concurrent sessions, and your choice of where the tool lives: on a public cloud, on a dedicated cloud, or on premises at your company.

Chromatic

What was previously just a visual regression testing service for projects using Storybook is now a full-fledged UI review system. Chromatic stands apart from the other SaaS tools mentioned in that it is now much more than just a visual regression testing tool.

What’s great about Chromatic?

Because Chromatic is maintained by the Storybook team, it’s a fantastic choice if your team is using Storybook to develop its components. With the release of Chromatic 2.0 early in 2020, a free tier is now offered for teams who want to try out the product. This is great because previously there was only a limited trial period before you had to decide whether to pay or drop the product.

What’s not so great about Chromatic?

First, if your team isn’t using Storybook, you can’t use Chromatic. And if you are using Storybook but only looking for visual regression testing, you might not want all of Chromatic’s full-fledged UI review tools.

I love that Chromatic now has a free tier, but you’ll want to keep an eye on how quickly your team burns through its 5,000 screenshots limit, especially in a new project that’s just getting set up or in a project with a lot of churn. The step between free and the first paid tier is quite large.

Another thing to note is that once you’ve chosen Chromatic, you’re kind of buying into their system. Your Storybook is going to be published to their CDN, you’ll be using their visual regression testing tool, and you’ll be using their review system. This isn’t necessarily a bad thing, but if you want control over each of these parts, it looks like it may be difficult to separate them.

Cost

Chromatic has five pricing tiers, starting with a free tier. The free tier gets you unlimited Storybooks, unlimited collaborators, component version tracking, secure CDN access, 5,000 screenshots per month, and integration with Git and continuous integration tools.

Upgrading to any of the next three tiers (at $149 per month, $349 per month, and $649 per month) gets you more screenshots, with the standard and pro tiers also giving you access to testing against additional browsers like Firefox and IE 11.

At the highest end, Chromatic also offers a custom enterprise tier that allows for single sign-on, custom Git integrations, priority support, invoicing and more.

Pick a Tool and Get Started

I hope that I’ve been able to provide a few choices for your team to evaluate. If I was setting up a new project today—especially a design system—I’d choose one of the SaaS tools I’ve mentioned. But, whether you decide on a DIY or a SaaS visual regression testing tool, you really can’t go wrong by having some kind of visual regression testing in your testing suite.

If you evaluate any of the tools I’ve talked about here, or if your team loves using a tool that I didn’t mention, we’d love to hear about it! Just reach out to Sparkbox on Twitter.