A mainstay of the modern-day web is the web application, a feature of Web 2.0 that brings experiences previously reserved for native applications to web browsers. Web applications like Twitter, Facebook, and Netflix have fundamentally shaped consumer expectations of the web by providing desktop-class experiences on their websites. At the core of these types of sites is the application store: a metaphorical accountant whose purpose is to keep a record of the current state of data. We know these accountants as Redux in React and Vuex in Vue. These specialized frameworks hydrate the user interface with data by orchestrating our create, read, update, and delete (CRUD) data operations. Inevitably, our web application grows. And as we add new features and perform necessary refactors, the complexity of data operations also grows. To maintain confidence that our application store performs as expected, we do the responsible thing and add unit tests that independently cover each individual operation. However, this situation provides a unique opportunity to change our perspective on how we test. This new way is much more time-efficient without sacrificing code coverage—meet the black-box testing approach.
Black-box Testing Versus White-box Testing
So what is black-box testing? Well, let’s start with a real-world example. If I have a flashlight and I want to check if it’s working, I can test it a few ways. The first way is to simply turn it on. If the lightbulb illuminates, then I know it’s working. This is similar to a black-box test because I’m only checking the high-level output (the lightbulb turning on). The second way I could test the flashlight is to disassemble its parts and check the internal electronics. This could involve checking that the batteries are providing current, the switch is directing current to the lightbulb when it’s in the “on” position, and making sure that the lightbulb illuminates when providing current. This approach is similar to white-box testing because I’m testing the flashlight’s internal components.
In short, this is the difference between black-box and white-box testing. The black-box approach tests code at a high level while remaining indifferent to its inner workings. It adopts the position of “I don’t care how it’s done; I just care that it is done.”
Anatomy of an Application Store
Before we delve into why a store is perfectly suited for black-box testing, we need to cover the basic mechanics of state management. In both Redux and Vuex, retrieving data from the store follows similar patterns: instead of reading from state directly, we use getter functions to get the last copy of state. This is done via getState in Redux and your own functions in your getters module in Vuex. Commiting changes to state involves a slightly more complicated pattern. For example, in Redux, we make changes to state by dispatching actions that invoke reducers (functions to handle the action payload) and return the newly-updated state. Using a similar pattern, Vuex also handles commits to state by dispatching actions. Except in Vuex, these actions invoke mutators that modify state directly instead of returning an updated value. Although the approaches used by both Redux and Vuex are different, a core idea is shared: communicating updates to the store should be streamlined through a central channel.
By designating that the application uses actions as the delegate for updating the store, we reduce coupling between the application and the store’s underlying mechanics. This gives us more flexibility when making changes to the store since the surface area exposed to the rest of the application is minimized. In fact, with updates to state and reads from state being centralized through actions and getters respectively, we’re already treating the store itself as a black box. We’re effectively telling the store, “I don’t care how you make this change to state. I just care that you do.”
Black-box Testing an Application Store
Now we’ve established that the store itself is treated as a black box by its application, how can we apply this approach to our testing? Well, let’s first cover how tests for a store are traditionally set up. Up until now, we’ve covered examples in both Redux and Vuex, but since that will lead to some unnecessary repetition, I’m going to focus on Vuex for the remainder of this article. (If you’re not familiar with Vuex, don’t worry—we’ll only be covering it from a high level.)
Traditional White-box Testing Strategy
In Vuex, we have four basic concepts that are combined into a single store: state, actions, mutations, and getters. It’s common to break out these modules into individual files like this:
store.js
store
|
|- state.js
|- actions.js
|- mutations.js
|- getters.js
The store.js file will import state.js, actions.js, getters.js, and mutations.js and compose those separate modules into a single store. If we wanted to build a store that was responsible for keeping track of a counter, it might look something like this:
// state.js
export const state = {
count: 0,
}
// actions.js
export const actions = {
updateCount: ({ commit }) => {
commit('INCREMENT_COUNT');
},
}
// mutations.js
export const mutations = {
INCREMENT_COUNT: state => {
state.count++;
},
}
// getters.js
export const getters = {
getCount: ({ count }) => count,
}
Here, you can see that we are able to update the count in state by dispatching the updateCount action. This action will then commit the INCREMENT_COUNT mutation that will actually update state with a new count. Lastly, the getter getCount retrieves the current count. To test this setup, we’d traditionally add a test for each module:
store.js
store
|
|- state.js
|- state.spec.js
|- actions.js
|- actions.spec.js
|- mutations.js
|- mutations.spec.js
|- getters.js
|- getters.spec.js
In this approach, each module is independently tested by its corresponding test file (the state.spec.js tests the state.js module to make sure count is in state, the actions.spec.js tests the actions.js module to make sure that dispatching the updateCount action commits the INCREMENT_COUNT mutation, etc.). This style of testing treats each module inside of the store as its own containerized component by independently testing each one. This approach allows granular testing of each part of the store and is the same one outlined in the official Vuex testing documentation. However, why should we employ different perspectives when we view the store in relation to our application and when we view it in relation to testing? What if we carried over the black-box mentality from implementation to our testing strategy?
Applying a Black-box Testing Strategy
This leads us to implementing a black-box testing approach to our store. We’ve identified that the store has a relatively small surface area to the application, meaning that the application is built as a module that interfaces with the application using a limited number of entry points (actions and getters). This allows us to easily adopt a black-box testing approach. We can use the store’s actions to dispatch changes and subsequently use the store’s getters to make assertions that changes were made. In our example store, this would mean that instead of independently testing each module of the store, we’d simply have a single test that would make assertions against the entire thing, like this:
store.js
store.spec.js
store
|
|- state.js
|- actions.js
|- mutations.js
|- getters.js
A single test sibling to store.js would test the application store from a high level. Consolidating all tests to this single test means that we can now only see into the application store as much as our application does. In other words, we lose sight of all the internal inner workings of the store. We only see the store through the lens of actions and getters. So what happens to our tests for both state.js and mutations.js? This is where our tests require adjustment to fit our new approach.
Previously, we were independently testing both state.js and mutations.js. However, with a black-box approach, we pivot to comprehensively testing the store as a singular entity. This means that assertions we were making against both state and our mutations need to be incorporated into our new testing approach. We can do this by changing our tests to make single assertions that traverse all layers of the store instead of just individual components. For example, in our previous traditional testing approach, we made independent assertions for the count state, the updateCount action, the INCREMENT_COUNT, and the getCount getter. However, when composed, all four come together to make up a singular idea about reading and writing count. This allows us to make assertions against all four in a single test by simply dispatching the updateCount action, confirming that the retrieved value from getCount is incremented by one.
describe('Count', () => {
it('increments the count by one', () => {
// Check the initial count
const countExisting = store.getters.getCount();
// Dispatch the action to increment the count
store.dispatch('updateCount');
// Get the new count
const countUpdated = store.getters.getCount();
expect(countUpdated).toBe(countExisting + 1);
});
});
As you can see in the above example, we’re able to make a single, comprehensive assertion that tests across all four modules of the store using only the actions and getters made available to the application. Although not explicitly testing our state or mutation, the above test would fail if state didn’t exist or if the mutation didn’t properly increment the value of count.
Benefits and Drawbacks of Black-box Testing
In our example test, you clearly see some benefits of adopting a black-box approach. To start, the number of tests is dramatically reduced. In our small example store, we’ve condensed four tests into one, reducing the total number by 75%! The benefits of this approach also extend to the refactoring process. As you refactor the underlying mechanics of the store, you won’t find yourself having to update your tests. Thanks to the black-box testing mantra of, “I don’t care how it’s done, I just care that it is,” you’ll see that as long as externally-used APIs (read, “actions and getters”) aren’t modified, your tests won’t fail. And since these tests are making assertions against the store as a singular entity, you may find flaws in your code you wouldn’t normally find when individually testing each component of the store with mocks.
Disadvantages of Black-box Testing
Of course, there are tradeoffs; the most obvious of them being a loss in test granularity. In the same way that your application is unaware of exactly where the error is when the store experiences a failure, your tests will be similarly unaware of why the store fails a test. In our example test, if count is a string instead of an integer in state.js or if INCREMENT_COUNT actually applies decrements instead of increments in mutations.js, the test failure will be the same. In both cases, you have to look through the code to determine where the failure is occurring and why the returned value doesn’t match the expected result. In the more traditional approach of individually testing each module separately, we would get a much more descriptive failure and precise indication of where the error is.
Additionally, utilizing a black-box testing approach to your store may give you a false sense of adequate testing coverage. This is because a black-box testing approach generally requires a fewer number of total tests, but each individual test covers a larger area of code. So there’s no lapse in code coverage, but your individual test might not actually be making adequate assertions that would result in failing tests when breaking changes are made. For example, imagine a store where a single action commits two different mutations to two different items of state:
// state.js
export const state = {
count: 0,
countDidRun: false,
}
// actions.js
export const actions = {
updateCount: ({ commit }) => {
commit('INCREMENT_COUNT');
commit('SET_COUNT_DID_RUN', true);
}
}
// mutations.js
export const mutations = {
INCREMENT_COUNT: state => {
state.count++;
},
SET_COUNT_DID_RUN: (state, countRan) => {
state.countDidRun = countRan;
},
}
// getters.js
export const getters = {
getCount: ({ count }) => count,
getCountDidRun: ({ countDidRun }) => countDidRun,
}
If we use the same black-box test we wrote earlier, we’ll get 100% code coverage on our state, actions, and mutations—even though we only ran assertions that converted the incrementor. The only clue to remind us that we didn’t run assertions against the code responsible for incrementing the count is our lack of coverage on getCountDidRun. However, this can be easily overlooked, especially in larger stores or in cases where the resulting mutation doesn’t have an obvious complemented getter. In these cases, you’d have to rely on eagle-eyed developers or something like mutation testing to ensure that tests fail when breaking changes are introduced.
Final thoughts
Is this approach appropriate for your project using state management? It’s difficult to make an across-the-board recommendation for projects of various sizes. However, I’ve anecdotally found that black-box testing is particularly useful when the store is rapidly developing (either during the initial store construction or a large refactor). In these cases, the benefit to using this approach simply comes down to time. If your store is in a state of rapid development and consistently undergoing major changes, using this approach saves time since you’re writing a smaller number of tests. During refactors, this approach also saves time because it does not require changing your tests at all since refactors would not involve changes to the action’s input or in the getters output.
Considering we treat the store as a singular entity when interfacing with the application, the argument to carry that relationship over to tests is a compelling one. In my own experience, I’ve found this approach strikes a sensible balance between adequately covering the store with tests and providing the flexibility to refactor the store’s underlying mechanics without having to unnecessarily update unit tests. This is a powerful benefit since it grants more autonomy to the collective store by taking a hands-off approach to checking precisely how changes to state are made within the store and, instead, simply checking that changes are made. By adopting the same black-box approach for testing that we’re already using on the implementation side, we maintain our testing coverage and confidence in our tests while dedicating less time to writing (and rewriting) them.
