Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Address python 3.12 deprecations #8046

Closed
braingram opened this issue Nov 1, 2023 · 7 comments
Closed

Address python 3.12 deprecations #8046

braingram opened this issue Nov 1, 2023 · 7 comments

Comments

@braingram
Copy link
Collaborator

As seen in python 3.12 tests for: #8033

https://github.com/spacetelescope/jwst/actions/runs/6723984874/job/18275278424?pr=8033

There are numerous deprecation warnings...
Use of os.fork

  jwst/jump/tests/test_detect_jumps.py: 1 warning
  jwst/jump/tests/test_jump_step.py: 12 warnings
    /opt/hostedtoolcache/Python/3.12.0/x64/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=2929) is multi-threaded, use of fork() may lead to deadlocks in the child.
      self.pid = os.fork()

Use of utcnow

jwst/associations/tests/test_asn_names.py::test_level3_asn_names_with_version[data/pool_020_00009_image_miri.csv]
jwst/associations/tests/test_asn_names.py::test_level2_asn_names_with_version[data/pool_020_00009_image_miri.csv]
jwst/associations/tests/test_main.py::test_generate_version_id[-\\d{3}t\\d{6}]
  /home/runner/work/jwst/jwst/jwst/associations/association.py:567: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    timestamp = datetime.utcnow().strftime(

   jwst/stpipe/tests/test_config.py::test_step_config_to_asdf
    /home/runner/work/jwst/jwst/jwst/stpipe/tests/test_config.py:59: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
      assert (datetime.utcnow() - datetime.fromisoformat(asdf_file["meta"]["date"])) < timedelta(seconds=10)
@WilliamJamieson
Copy link
Collaborator

Did #8033 solve this issue?

@braingram
Copy link
Collaborator Author

Only partially. The os.fork warnings were not addressed in #8033

@WilliamJamieson
Copy link
Collaborator

Only partially. The os.fork warnings were not addressed in #8033

This should be fixed by: 689fd5b. Do you want this separated into its own follow on PR or just keep in in #8093?

After some investigation I found that this error is not because jwst or stcal is calling os.fork, rather it is due to the multiprocessing using the fork start method for creating its multiple processes. Currently, the fork method is used by default multiprocessing; however, this behavior will be changed to the forkserver in Python 3.14, see https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods for details. This is because there are some runtime dangers associated with the pure forking behavior that are address this behavior by starting a server process to track the forking performed by multiprocessing.

@WilliamJamieson
Copy link
Collaborator

Note that the multiprocessing actually occurs in stcal, which is where this fix should ultimately be enacted.

@braingram
Copy link
Collaborator Author

Perhaps @hbushouse can weigh in on this. He commented on #8033:

I think I'd rather keep this pinned to <3.12 for now, until we put together our DMS B10.1 release in a few weeks and do thorough testing of that whole release under 3.12.

@hbushouse
Copy link
Collaborator

I'm not opposed to including python 3.12 in our matrix of CI tests, but I'd rather not make it the default for anything, because as of right now there are no plans to deliver the upcoming jwst B10.1 for use with 3.12. It'll still be delivered with 3.11 as the max allowable, at least until we're satisfied that everything is OK under 3.12. Operations will still be running it under 3.11.

@WilliamJamieson
Copy link
Collaborator

If we merge #8095 followed by #8093, then all the known issues with Python 3.12 will be addressed. Note that we won't be to run it in the Jenkins jobs at this time because lxml which is reqired by pysiaf seems to want to build from source even though it looks like there are Python 3.12 wheels.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants