Design systems are great for developer efficiency, visual consistency, and accessible experiences. But measurement can be challenging, and even here at Sparkbox we’re often relying on anecdotes from our own projects or from self-reported metrics that we’ve gathered from our annual design systems survey. We wanted more proof, so we decided to test our hypothesis, “design systems help developers produce better code faster.” Our test subjects? A handful of willing Sparkbox developers.
Here’s how we structured our study. We picked a well-known design system that was publicly available, had excellent documentation, and that none our developers had ever used. Enter IBM’s Carbon design system. Then a Sparkbox designer used Carbon to design a contact form in Figma. We recruited eight Sparkbox developers to code the form from scratch while timing themselves. Then they coded the form using the Carbon design system, again while timing themselves. Then we compared the median and average times to determine efficiency
Next, another eight Sparkbox team members reviewed the hand-built forms and Carbon forms for accessibility and for visual consistency, ranking the submissions from best to worst. The individual rankings were combined for an aggregate score. Learn more about the methodology.
The Test
1. Create Form Design
The Figma design was built according to the Carbon design system and used as the source of truth for the developers and the reviewers.
2. Develop From Scratch
We recruited eight Sparkbox developers to code the form from scratch while timing themselves.
3. Develop with Carbon Design System
Our developers coded the form using the Carbon design system, again while timing themselves.
4. Compare Techniques
Finally, we compared the median and average times to determine efficiency.
The Findings
Based on our sample of eight Sparkbox developers, we uncovered the following:
Faster Development
Using a design system made a simple form page 47% faster to develop versus coding it from scratch. The median time for the scratch submissions was 4.2 hours compared to the 2 hour median time for Carbon submissions. The Carbon timing included the time the developers spent familiarizing themselves with the design system.
| Developer | From Scratch | Carbon |
|---|---|---|
| Developer One | 4.2 hours from scratch | 1.1 hours using Carbon |
| Developer Two | 5.1 hours from scratch | 2.5 hours using Carbon |
| Developer Three | 5.1 hours from scratch | 3.7 hours using Carbon |
| Developer Four | 4.5 hours from scratch | 2.7 hours using Carbon |
| Developer Five | 4.1 hours from scratch | 2.5 hours using carbon |
| Developer Six | 2.8 hours from scratch | 1.3 hours using Carbon |
| Developer Seven | 2.7 hours from scratch | 1.5 hours using Carbon |
| Developer Eight | 2.2 hours from scratch | 1 hour using Carbon |
Visual Consistency
Using a design system helped our developers produce code that was more visually consistent with the design. In fact, only one of the eight developers built a form from scratch that ranked higher in visual consistency than the form they built using Carbon.
- The top two submissions for visual consistency used the design system.
- Visual consistency for five of the eight developers improved when using the design system. In fact, one developer went from 14th place out of 16 for their scratch submission to first place for their Carbon submission.
- The visual consistency score for two of the developers was roughly the same for their from-scratch submission as their Carbon submission.
- Only one developer’s scratch coding soundly beat their design system submission.
More Accessible Code
- One developer went from last place to the middle of the pack for accessibility when they used the design system.
- Two developers saw their accessibility improve slightly when using the design system.
- Three developers’ hand-coded submissions and Carbon submissions ranked roughly the same for accessibility.
- Two developers’ scratch submissions bested the design system for accessibility.
Let’s be honest, Sparkbox is great at building for accessibility—it’s just how we work. Two of the developers who coded submissions were IAAP certified Web Accessibility Specialists. Since many of our developers are deeply knowledgeable and focused on making the web accessible, we are a little different from many teams. If accessibility is not typically a priority for your organization, a design system may help you more than it helped us.
Let’s make these stats more exact
This was a small study of a handful of excellent developers. We need your excellence to keep exploring these findings! If you’d like to participate, email info@sparkbox.com with a subject line of “Participate in Design System ROI” and we’ll be in touch!
More about Our Developers and the Methodology
The eight developers who coded the forms came with a variety of experience:
- Professional development experience ranged from junior developer to senior developer.
- All the developers were full-stack, but some were frontend focused while others were more backend focused.
- Every Sparkbox developer has had significant accessibility training, and two of Sparkbox’s IAAP certified Web Accessibility Specialists participated.
- Some of our developers had previous experience building and using design systems before, while others did not.
We gave the developers the following instructions for the code exercise. You might be wondering why we had the developers code the form from scratch first. We didn’t want to influence their handbuilt code by showing them how Carbon would’ve coded it first. And, we figured any advantage they gained by scratch coding first would be mitigated by having to familiarize themselves with the Carbon design system. In the end, everyone participated in their free time, so while some people were able to get it done in a day or two, other developers had weeks or months between their scratch-coded and Carbon submissions.
For time scoring, each developer was asked to use their timesheet tool’s timer (we use Harvest) to record how long it took them to complete the scratch submission and the Carbon submission.
For visual consistency and accessibility reviews, each of the eight reviewers was given a randomized list of submissions. For the visual consistency review, they were instructed to compare each submission to the original design file and note any deviations. For the accessibility review, the reviewers tested the form for keyboard-only use and used Axe Dev Tools or WAVE to find errors. All rankings were ultimately subjective and were combined into an overall score.
Credits
Erin Blad, Rise Erpelding, Patrick Fulton, Flash Gooden, Luis Hernandez, Nate Jacobs, Hunter Keca, Mandy Kendall, Ryan Kelbel, Sheridan McKisick, Catherine Meade, Travis Sanon, Andrew Spencer, Melissa Thompson, Dustin Whisman, Julie Young, and Philip Zastrow