Fix heasarc tests #2842

zoghbi-a · 2023-09-25T14:16:35Z

This a fix for the heasarc tests. Specifically:

Fix column name in test_heasarc_remote_isdc.test_mission_cols, so the test now passes.
Move patch_get to class level to allow for other classes (in prep) that do not need it.

pep8speaks · 2023-09-25T14:16:43Z

Hello @zoghbi-a! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2023-10-23 10:49:10 UTC

codecov · 2023-09-25T14:32:09Z

Codecov Report

Merging #2842 (9c0350e) into main (37758c1) will decrease coverage by 0.03%.
Report is 3 commits behind head on main.
The diff coverage is n/a.

❗ Current head 9c0350e differs from pull request most recent head c4f6ff4. Consider uploading reports for the commit c4f6ff4 to get more accurate results

@@            Coverage Diff             @@
##             main    #2842      +/-   ##
==========================================
- Coverage   66.51%   66.49%   -0.03%     
==========================================
  Files         235      235              
  Lines       18096    18081      -15     
==========================================
- Hits        12037    12023      -14     
+ Misses       6059     6058       -1

see 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

bsipocz · 2023-09-25T17:17:45Z

astroquery/heasarc/tests/test_heasarc_remote.py

@@ -15,6 +15,11 @@

 @parametrization_local_save_remote
 class TestHeasarc:
+
+    @pytest.fixture(autouse=True)
+    def _patch_get(self, patch_get):


Hmm, I think the original issue is that the mock should not have a remote conditional in it at the first place, the idea with all the remote tests is that they are used the same way a user would (aka without the mock patch)

Hmm, I think the original issue is that the mock should not have a remote conditional in it at the first place, the idea with all the remote tests is that they are used the same way a user would (aka without the mock patch)

I am working on extending the heasarc services, and when I added a new test class, it was using patch_get because it had autouse=True, so I moved the autouse=True part to the classes that need it. The class I am creating does not need it. Is there a better way to do it?

I changed this to WIP because I see errors in my local testing and I thought I am fixing them, but they don't show up in the CI.

re autouse: yes, removing it sounds good, most modules don't have autouse.

CI: we don't run the remote tests in CI on PRs, only once a week from cron, so that's why you may not see them failing. E.g. here I see the same heasarc failure as I see locally: https://github.com/astropy/astroquery/actions/runs/6279884195/job/17056292621#step:5:4890

That being said, any improvements to the module is more than welcome, even to the point of a total gut out and starting from new (e.g. that's what I'm doing with the IRSA one, switching it altogether to VO backends). For heasarc, there are a few stalled PRs, too, but we can close them off if needed.

So, feel free to ping me on slack if there are any questions or if I can help with any of this.

re autouse: yes, removing it sounds good, most modules don't have autouse.

CI: we don't run the remote tests in CI on PRs, only once a week from cron, so that's why you may not see them failing. E.g. here I see the same heasarc failure as I see locally: https://github.com/astropy/astroquery/actions/runs/6279884195/job/17056292621#step:5:4890

I see. That makes some sense. From that link I see both local and remote tests fail. Does test_mission_cols[local] mean it is not using remote service?

../../.tox/py311-test-alldeps-devdeps-online/lib/python3.11/site-packages/astroquery/heasarc/tests/test_heasarc_remote_isdc.py::TestHeasarcISDC::test_mission_cols[local] FAILED [ 23%] ../../.tox/py311-test-alldeps-devdeps-online/lib/python3.11/site-packages/astroquery/heasarc/tests/test_heasarc_remote_isdc.py::TestHeasarcISDC::test_mission_cols[save] SKIPPED [ 23%] ../../.tox/py311-test-alldeps-devdeps-online/lib/python3.11/site-packages/astroquery/heasarc/tests/test_heasarc_remote_isdc.py::TestHeasarcISDC::test_mission_cols[remote] RERUN [ 23%] ../../.tox/py311-test-alldeps-devdeps-online/lib/python3.11/site-packages/astroquery/heasarc/tests/test_heasarc_remote_isdc.py::TestHeasarcISDC::test_mission_cols[remote] FAILED [ 23%]

yes, I think I overgeneralized above, in all the other modules, mocked and other local tests are separated from the remote ones, here it seems that everything has a local and a remote version. According to that, test_mission_cols[local] should not use remote service.

Is it obvious to you why test_mission_cols[local] is failing in that link, but not in the CI test for the this PR?

guesswork: I would think the canned version of that test integral query is outdated. And there is at least one bug (I guess more), with how the tests are run. E.g. a previous test method downloads the current version into the temp data dir (at least it doesn't go into the central cache location, that has been fixed already), and later tests pick that up instead of the canned version, even for the mocked version of the test.
Historically, I think this is one reason why we kept local and remote tests separate, less headache with what the given instances see, etc. (and we also don't need to fully duplicate everything local to remote).

E.g., if I remove all that uses integral_rev3_scw, it's not failing any more:

astroquery/heasarc/tests/test_heasarc_remote_isdc.py .s..s..s..s..sF.s..s..s. [100%] ================================================================= FAILURES ================================================================== _________________________________________________ TestHeasarcISDC.test_mission_cols[remote] _________________________________________________ self = <astroquery.heasarc.tests.test_heasarc_remote_isdc.TestHeasarcISDC object at 0x129c28670> def test_mission_cols(self): heasarc = Heasarc() mission = 'integral_rev3_scw' with self.isdc_context: cols = heasarc.query_mission_cols(mission=mission) assert len(cols) == 35 # Test that the cols list contains known names assert 'SCW_ID' in cols assert 'GOOD_ISGRI' in cols assert 'RA_X' in cols assert 'DEC_X' in cols > assert '_SEARCH_OFFSET' in cols E AssertionError: assert '_SEARCH_OFFSET' in ['SCW_ID', 'SCW_VER', 'SCW_TYPE', 'RA_X', 'DEC_X', 'OBS_ID', ...] astroquery/heasarc/tests/test_heasarc_remote_isdc.py:133: AssertionError ========================================================== short test summary info ========================================================== FAILED astroquery/heasarc/tests/test_heasarc_remote_isdc.py::TestHeasarcISDC::test_mission_cols[remote] - AssertionError: assert '_SEARCH_OFFSET' in ['SCW_ID', 'SCW_VER', 'SCW_TYPE', 'RA_X', 'DEC_X', 'OBS_ID', ...] ============================================ 1 failed, 33 passed, 17 skipped in 61.12s (0:01:01) ============================================

This is something I've been trying to work around also when experimenting with trying to switch to VOTable like in #2353. I found a solution that works 90% of the way to converting to VOTable but I ran into similar issues with the changes I made particularly for integral_rev3_scw testing. Maybe I'll just submit a PR for it to get some potential input, but have been at it in some free time for a few months now and haven't gotten anywhere.

Not sure if this is helpful or not, but just thought that since there was a mention on extending heasarc I would share a quick note on my experiences.

zoghbi-a · 2023-10-18T21:38:39Z

@bsipocz,
Do you think we can merge this so the tests work? I will submit a separate PR for the class changes.

bsipocz

The column name back-and-forth needs to be fixed by updating the name to pass the remote queries and thus the canned data file in the mock tests also needs the column name fixed.

fixture fixes look good, and I agree that reorganizing tests can be done in a follow-up.

bsipocz · 2023-10-18T21:58:47Z

astroquery/heasarc/tests/test_heasarc_remote.py

-        assert 'SEARCH_OFFSET_' in cols
+        # assert 'SEARCH_OFFSET_' in cols


There has been a bit of a back-and-forth with this column name. Could you consolidate those changes to actually fix rather than skip the test? E.g. have a pass with the remote-data tests, and update the canned data file accordingly to get the mocked tests pass.

I don't know how to update the data files. I tried deleting them and running remote tests, but they didn't update

you need to either manually edit, or manually overwrite them with a freshly downloaded version (as ideally the datafiles we have canned up in the test suite are actually relevant versions of the remote responses).

…sses that do not need it

zoghbi-a · 2023-10-23T10:59:02Z

The remaining failure is from the cadc tests.

bsipocz · 2023-10-24T15:23:03Z

Thank you so much @zoghbi-a!

zoghbi-a force-pushed the fix-heasarc-tests branch from c1b7851 to df4e08a Compare September 25, 2023 14:28

zoghbi-a changed the title ~~Fix heasarc tests~~ WIP: Fix heasarc tests Sep 25, 2023

bsipocz reviewed Sep 25, 2023

View reviewed changes

bsipocz added heasarc no-changelog-entry-needed labels Sep 25, 2023

zoghbi-a changed the title ~~WIP: Fix heasarc tests~~ Fix heasarc tests Oct 2, 2023

bsipocz requested changes Oct 18, 2023

View reviewed changes

zoghbi-a force-pushed the fix-heasarc-tests branch 3 times, most recently from df54874 to 8533c7d Compare October 23, 2023 01:58

zoghbi-a added 4 commits October 23, 2023 06:48

fix column name in test_heasarc_remote_isdc.test_mission_cols

f0d2b86

in heasarc.test, move patch_get to class level to allow for other cla…

dd3a1b0

…sses that do not need it

fix flake8 E303 error

f44a0c3

attempting to fix the tables

c4f6ff4

zoghbi-a force-pushed the fix-heasarc-tests branch from 3e98967 to c4f6ff4 Compare October 23, 2023 10:49

bsipocz approved these changes Oct 24, 2023

View reviewed changes

bsipocz added this to the v0.4.7 milestone Oct 24, 2023

bsipocz merged commit ff01bc1 into astropy:main Oct 24, 2023
6 of 7 checks passed

zoghbi-a deleted the fix-heasarc-tests branch November 21, 2023 21:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix heasarc tests #2842

Fix heasarc tests #2842

zoghbi-a commented Sep 25, 2023

pep8speaks commented Sep 25, 2023 •

edited

Loading

codecov bot commented Sep 25, 2023 •

edited

Loading

bsipocz Sep 25, 2023

zoghbi-a Sep 25, 2023

zoghbi-a Sep 25, 2023

bsipocz Sep 25, 2023

bsipocz Sep 25, 2023

zoghbi-a Sep 25, 2023

bsipocz Sep 25, 2023

zoghbi-a Sep 25, 2023

bsipocz Sep 25, 2023

nkphysics Oct 16, 2023

zoghbi-a commented Oct 18, 2023

bsipocz left a comment

bsipocz Oct 18, 2023

zoghbi-a Oct 19, 2023

bsipocz Oct 19, 2023 •

edited

Loading

zoghbi-a commented Oct 23, 2023

bsipocz commented Oct 24, 2023

		assert 'SEARCH_OFFSET_' in cols
		# assert 'SEARCH_OFFSET_' in cols

Fix heasarc tests #2842

Fix heasarc tests #2842

Conversation

zoghbi-a commented Sep 25, 2023

pep8speaks commented Sep 25, 2023 • edited Loading

Comment last updated at 2023-10-23 10:49:10 UTC

codecov bot commented Sep 25, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zoghbi-a commented Oct 18, 2023

bsipocz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bsipocz Oct 19, 2023 • edited Loading

Choose a reason for hiding this comment

zoghbi-a commented Oct 23, 2023

bsipocz commented Oct 24, 2023

pep8speaks commented Sep 25, 2023 •

edited

Loading

codecov bot commented Sep 25, 2023 •

edited

Loading

bsipocz Oct 19, 2023 •

edited

Loading