Deadlock when deloying flow with dependencies that reconfigure the python logger

[j-tr]

Bug summary

Flows that reconfigure the python logger can cause deadlocks during deployment very similar to #9345

prefect3-deadlock.py:

from prefect import flow
import logging
from logging.config import dictConfig

# this happens for example when running "import anyscale" (see anyscale/__init__.py  https://pypi.org/project/anyscale/)
logger = logging.getLogger(__name__)
logger.setLevel("WARN")

print("This line is reached")

# this is a very simplified version of what happens in ray/_private/log.py (generate_logging_config)
# it is called from ray/__init__.py
dictConfig({"version": 1})

print("This line is never reached")

# when importing anyscale, the logger is configured,
# then ray is imported by anyscale, which reconfigures the logger again.

@flow
def myflow():
    pass

prefect.yaml:

# Welcome to your prefect.yaml file! You can use this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.

# Generic metadata about this project
name: my-project
prefect-version: 3.1.4

# build section allows you to manage and build docker images
build: null

# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.set_working_directory:
    directory: /my-project

# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: test
  version: null
  tags: []
  description: null
  schedule: {}
  flow_name: null
  entrypoint: prefect3-deadlock.py:myflow
  parameters: {}
  work_pool:
    name: test
    work_queue_name: null
    job_variables: {}

How to reproduce:

• create folder my-project, copy prefect3-deadlock.py and prefect.yaml
• prefect worker start -p test to setup the worker
• prefect deploy --all

Output:
This line is reached Then hanging indefinitely. This line is never reached is never printed.

Version info

Version:             3.1.4
API version:         0.8.4
Python version:      3.11.9
Git commit:          78ee41cb
Built:               Wed, Nov 20, 2024 7:37 PM
OS/Arch:             linux/x86_64
Profile:             local
Server type:         server
Pydantic version:    2.9.2
Integrations:
  prefect-aws:       0.5.1
  prefect-ray:       0.4.2

Additional context

The same flow works fine in Prefect 2.20.14.
Since we are making heavy use of Ray and Anyscale, this is currently blocking us from adopting Prefect 3.
While this issue popped up using Anyscale, it is a general problem of Prefect 3 handling reconfiguration of the python logger.

[lennartvandeguchte]

I'm running into the same issue when importing MLFlow. This is also blocking us from adopting Prefect 3.

[jeanluciano]

We are currently looking into this. So far we have we understand that this happens when logging.congig.dictConfig gets called with a config that has incremental=True. This then calls, logging.shutdown which in turn calls the flush method in our logs handler. We are looking for a fix to how we call drain_all inside of the flush method.