Added Visual Regression Testing via percy.io

[jameslnewell]

Description

I've added automated Visual Regression Testing for each Storybook using percy.io.

Why

Components in a component library like @wordpress/components are black boxes. The consuming developer and the end-user aren't concerned what the HTML, CSS or JavaScript looks like as long as the visuals and the behaviour (including a11y) work as expected.

When the markup or styles of a component change, we, consuming developers and end-users aren't concerned with what the internal change was as long as the external outcome is correct and doesn't introduce a regression. It can be very difficult from the diff of a markup or style change to assess whether a change will introduce a regression - particularly when a subtle style change can lead to significant visual changes.

How

Visual Regression Testing can be difficult to setup and troublesome to maintain (e.g. antialiasing). Therefore I've used the percy.io service which saves a lot of effort and learnings.

On CI the Percy CLI runs and uploads a screenshot of each story in the Storybook to the Percy service.

They keep a history of these images for comparison and use some secret sauce in an attempt to limit false negatives. If the visuals have significantly changed it will block the PR and require someone to review the screenshots highlighting any changes to the visuals.

Alternatives to Percy could include jest-image-snapshot.

Remaining setup

1. Create an official WordPress percy.io account.
2. Setup the Percy integration with Github
3. Add the PERCY_TOKEN to Travis as a secure environment variable

How has this been tested?

Ran storybook:snapshot locally, made a visual change and ran storybook:snapshot locally again.

Screenshots

Types of changes

No external changes.

Checklist:

• My code is tested.
• My code follows the WordPress code style.
• My code follows the accessibility standards.
• My code has proper inline documentation.
• I've included developer documentation if appropriate.
• I've updated all React Native files affected by any refactorings/renamings in this PR. .

[ItsJonQ]

@jameslnewell This is exciting! I've experimented with Percy in the past, but I haven't used it for a large / open project before. If something does get flagged. How would someone go into the app and give their 👍 / 👎 ?

Thank you!

[jameslnewell]

The percy project would be made public so contributors can see the screenshots and make any necessary changes when there is a regression.

Then there'd be an authorised set of reviewers who have the ability to approve the changes. These reviewers could be the same set of reviewers as is selected for PRs today.

If we want to extend the trust beyond a small set of reviewers, we'd might need to consider building a Github integration to make it scale.

[gziolo]

Thank you for exploration. We still didn’t introduce visual regression testing. There is a similar exploration for Chromatic in #22155. Let’s close this PR but keep it as a reference.