Add testing.check_figures_equal to avoid storing baseline images

[seisman]

Description of proposed changes

This PR adds a new function check_figures_equal to check if two pygmt.Figure() objects are the same, mostly inspired by comment #522 (review) and the matplotlib decorator check_figures_equal.

What the function check_figures_equal does is very simple:

1. It accepts two pygmt.Figure() objects (fig_ref and fig_test) as arguments
2. save them into two PNG images
3. compare these two images using the compare_images function from matplotlib and calculate the RMS value
4. if the two images are the same, then delete them; if the rms is greater than tol, generate the diff image and raise an exception.

I add three new tests:

• test_check_figures_equal checks the case when two images are equal
• test_check_figures_unequal checks the case when two images are unequal. The exception is correctly caught by pytest.raises.
• test_grdimage_central_longitude is a real test to check the images generated by passing a matrix or a grid to grdimage (we always assume that passing a netCDF grid to GMT gives the correct/reference image). The test helps me find a GMT bug, which is usually very difficult to detect (see Matrix as grid with changing central meridian doesn't work well for gridline grids gmt#3844 and Add special check for non-rotated global grids gmt#3849).

Some known issues/limitations:

1. some paths are hard-coded, but they may be improved to be more flexible
2. The test test_check_figures_unequal() checks if the function check_figures_equal correctly raises the GMTImageComparisonFailure exception when two images are unequal, and I use pytest.raises to catch the exception so that the test passes. However, the two images are not deleted after the test. They are now.

Fixes #

Reminders

• Run make format and make check to make sure the code follows the style guide.
• Add tests for new features or tests that would have caught the bug that you're fixing.
• Add new public functions/methods/classes to doc/api/index.rst.
• Write detailed docstrings for all functions/methods.
• If adding new functionality, add an example to docstrings or tutorials.

[weiji14]

@seisman, hope you don't mind if I continue some work on this.

1. some paths are hard-coded, but they may be improved to be more flexible

Will try to see if it's possible to mimic the pytest-mpl paths, or look to matplotlib to see how they do it.

2. The test test_check_figures_unequal() checks if the function check_figures_equal correctly raises the GMTImageComparisonFailure exception when two images are unequal, and I use pytest.raises to catch the exception so that the test passes. However, the two images are not deleted after the test.

It should be easy-enough to use GMTTempFile so that the images are cleaned up after the test, but we'll need to find a way to keep the images if the test fails so that they can be examined.

[weiji14]

Ok, this is pretty much ready for review. After stress testing this with pytest, I think the check_figures_equal function will have to end up pretty much exactly like matplotlib's https://matplotlib.org/3.3.1/_modules/matplotlib/testing/decorators.html#check_figures_equal, except that we're using fig=pygmt.Figure() instead of matplotlib's plt.subplot. If only we can monkeypatch it!!

It might be worth raising an issue/PR on matplotlib to see if there's a way to reduce code duplication (for the sake of long term maintainability), as they've clearly spent months/years of thought and effort into this one check_figures_equal function. This would involve some sort of subclassing (or I don't know, pluggability?), so that we can swap in pygmt.Figure into fig_ref and fig_test, while getting all of the image_compare goodness.

[weiji14]

Figured out how to make our PyGMT check_figures_equal decorator work with pytest fixtures (e.g. grid=xr.DataArray(...)) in 3e0d3fb. This is basically just copying what was done in matplotlib at matplotlib/matplotlib#16800.

[weiji14]

Suggested change

	@check_figures_equal()
	def test_grdimage_central_longitude(grid, fig_ref, fig_test):
	"""
	Test that plotting a grid centred at different longitudes/meridians work.
	"""
	fig_ref.grdimage("@earth_relief_01d_g", projection="W120/15c", cmap="geo")
	fig_test.grdimage(grid, projection="W120/15c", cmap="geo")
	@pytest.mark.parametrize("meridian", [0, 33, 120, 180])
	@check_figures_equal()
	@pytest.mark.parametrize("proj_type", ["H", "Q", "W"])
	def test_grdimage_different_central_meridians_and_projections(
	grid, proj_type, meridian, fig_ref, fig_test
	):
	"""
	Test that plotting a grid centred on different meridians using different
	projection systems work.
	"""
	fig_ref.grdimage(
	"@earth_relief_01d_g", projection=f"{proj_type}{meridian}/15c", cmap="geo"
	)
	fig_test.grdimage(grid, projection=f"{proj_type}{meridian}/15c", cmap="geo")

I'll update this test in #560 later 😄. Problem with using this fancy pytest.mark.parametrize is that it would complicate the check_figures_equal code (see matplotlib/matplotlib#15199 and matplotlib/matplotlib#16693), and make this PR even harder to review.

[seisman]

from matplotlib.testing.compare import compare_images

As I understand it, the code means that now matplotlib becomes a required dependency, even for users who never run the tests, right?

Although PyGMT already requires matplotlib for testings and most users usually have matplotlib installed. I still don't want to add one dependency to PyGMT.

When I wrote the first commit (8b78614), I put the codes in pygmt/helpers/testing.py. By doing that way, I think matplotlib is still optional, although I haven't tested it.

[weiji14]

Good point, I think you're right here. I also encountered issues with circular imports when moving the code to decorators.py, hence this line:

pygmt/pygmt/helpers/decorators.py

Line 459 in 04b3f41

from ..figure import Figure # pylint: disable=import-outside-toplevel

Probably should move it back under pygmt/helpers/testing.py then. As an aside, I've opened up a feature request at matplotlib/pytest-mpl#94, and we might be able to do all this from pytest-mpl in the future.

[seisman]

One more thing about the result_dir is that, the check_figures_equal decorator generates images in result_images directory, while pytest.mark.mpl_image_compare generates images in directories like results/tmpjtnnwqt4.

[weiji14]

Yes, which was partly why I opened up the issue at matplotlib/pytest-mpl#94, to get all of that pytest-mpl goodness (e.g. not having a hardcoded result_dir). I'll try to make a Pull Request to pytest-mpl for that, so we can just use a proper @pytest.mark.mpl_check_equal decorator in the future (will open a new issue after this one is merged). For now though, since we don't have many tests using check_figures_equal yet, we can probably just leave it like so.

[seisman]

Yes, looks good to me.

[weiji14]

Added some notes here to CONTRIBUTING.md, adapted from https://matplotlib.org/3.3.1/devel/testing.html#writing-an-image-comparison-test.

[seisman]

Looks good to me, but I can't approve it because I opened this PR.

[weiji14]

Ah yes, I'll approve it then 😆