At SC5, we have been using a lot of automation in process development. The idea is to provide bulletproof products with shortened timeframe and less human resources, in other words, cheaper.
In front-end engineering, one of the last automation trends is visual regression testing. At the first glance, it seems very complicated to introduce this to a process. Even knowing how much benefits it would bring to a project, it is not always possible to dedicate time and resources to it.
Still, following the mantra of automating everything, I was looking for a way to introduce visual tests to any project skipping the time-consuming implementation. Once I got the job started, I found it. Currently, it takes me minutes to bring this practice into a new development project.
The key idea
The continuous development process requires an application to be bug-free at every iteration. For frontend engineers this can be a more crucial point of concern. Why? Because customers evaluate a product through its appearance and functions, any inaccuracy can wipe their trust. Serious bugs often cause a failure of projects.
Also, it is not a fun experience to find that some feature, into which developers invested so much time and creativity, simply does not work. This is why we, developers, hunt bugs. It is always better to catch instead of being caught.
Nowadays, many proven solutions help with this issue. The industry standard is to provide different kinds of tests, including unit tests, integration tests, compatibility tests, etc. It makes our lives much easier but even this broad range of automated testing can not ensure 100% workability.
Every change raises a question ‘Does my website make sense for the customer right now?’. Most developers try to answer this themselves. It is customary to open two versions of a web app and compare them visually with that same skill that we all mastered in kids’ magazines.
Luckily, it’s the 21st century, and we have software to automate this process called visual regression testing. Key is to make reference screenshots of required pages and store them. After code commitment, which can screw interface, we make the screenshots again and compare them to the original. This comparison is done with an image library and provides us with perceptual differences to analyze.
To understand how this works let us practice a little bit with a real case. Here are two versions of the same page before and after some changes:
Try to find out what is wrong. You may write down all your findings. Then, check the right answer :
The perceptual (or visual) diffs are the images marked with spots in differentiating areas. When analyzing such diffs a developer keeps focus on these spots and can easily figure out what are the underlying reasons for them.
The example image above shows the discrepancies between font sizes, background colors, texts, and icons. You can also notice the inaccuracy in the price block content.
Right way to use
Unfortunately, being familiar with a method will not take you far. It is more important to know how to apply a method for the results you need. This approach is no exception. The results of such comparison depend on the volume of investigated changes. For example, these pages do not differ that much.
Only the header block has been shifted 5 pixels, but the visual diff contains so many marked areas that it does not help overall.
In this regard, the better approach is to compare smaller pieces, such as components presented on the pages. It allows focusing precisely.
Component grew or diminishes
Unlinked icon font
We, at SC5, are passionate about a component approach when developing web interfaces. This is why we created a helping tool named SC5 Style Guide that we use on a daily basis. This practice turned out to be fruitful in many cases.
When it comes to visual regression testing, SC5 Style Guide proves to be useful. It renders all UI components with the same styles that built the product pages. However, in SC5 Style Guide, each component is represented by a separate page that suits the visual regression testing wonderfully.
Following the modular approach, we provided an extension package for the SC5 Style Guide, namely sc5-styleguide-visualtest. To use it, you only need to install it along with sc5-styleguide and declare two gulp tasks.
npm install sc5-styleguide-visualtest
One gulp task (
test:visual:update) is necessary for producing the original screenshots. When you run it along with generated style guide, it takes a screenshot of every documented component. Every taken image is stored in a folder that you include into the project repository for future use as a reference.
The other gulp task (
test:visual) runs visual tests of the components. In fact, it also takes a screenshot of every component and compares it to the reference. The console shows which component looks correct and which failed the test.
The provided numbers are the section numbers from the generated style guide.
When errors appear, you can open generated
gemini-report/index.html file to see the image diffs and analyze them.
The development flow
With changes applied, a developer runs tests to check the impact on a particular section or a bunch of components:
# Only for named sections
> gulp test:visual --section 7.3.4 --section 8.5
# For everything
> gulp test:visual
Keeping all the reference screenshots up-to-date is important. That is why the developer updates the references after a final approvement of the visual changes.
> gulp test:visual:update --section 8.5
In most cases, this update is required for the components under development. However, in the case of global changes that may affect the whole UI (e.g. font size), a developer updates all the original screenshots.
> gulp test:visual:update
Another pleasant thing we discovered is an opportunity to include the changes of the reference screenshot in the same pull request. Thus, a reviewer sees the code diff and the browser rendered diff.
Real life stories
This is my second time to use the visual regression testing with SC5 Style Guide in product development and I can recount some stories about the cases when it helped.
It started with a missing icon font. I was developing a UI library containing hundreds of components. In this case, it is not possible to test each of them manually. However, sometimes the global changes cause a problem in an unexpected place.
The task was to change the SASS processing in the whole library. I have done that, checked some components which looked OK, and released the library. The project development team was waiting for this release to update their website. They checked the changed pages and found them as expected. Then they launched the website with the new version of the library. An hour later, however, we figured out that some elements on the other pages (sadly, sales pages) had lost the icons. It was a side-effect of implementing the new SASS processing. In a hurry, we rolled everything back.
The underlying reason for this bug was a human mistake, but I realized that no human can prevent such thing in the future. So, we needed an automated solution.
With an introduction of the testing process, the development turned out to be fruitful. The implementation of global changes with a controlled impact on the fonts and typography became easy. We could re-factor the atomic components that constitute bigger parts. After re-factoring, every component was tested again, and I was sure that everything looked as expected. We’ve got an opportunity to remove the unused code and check if it was not, in fact, used anywhere and its removal would not break anything.
In general, I see a huge positive impact of visual regression testing to the process of development.
Previously, I found it unfair that we, developers, must be accurate while there was no tool to help us. The code reviewers are still people, and they make mistakes. Everything gets better when we have access to an automated, robot-controlled process that we can rely on.
Personally, I feel safer and braver. I noticed that the library contributors now feel the same. The amount of re-factoring grew, and more experiments and bold changes are taking place. In fact, even a junior developer can now experiment easily and safely with automated visual tests.