Reloader doesn't add ".exe" to file on Windows

[peppobon]

if I run a flask app under 0.15.5 with
FLASK_ENV=development
the dev server throws an error.

With FLASK_ENV=production the server starts ok

After reinstalling 0.15.4 the same app works ok (both in dev and prod mode).

Windows/py3.6.7/pipenv

Kind regards

[ad-65]

Same issue using Flask 1.1.1/Werkzeug 0.15.5 on Windows 10, python 2.7.16, in a virtual environment. Looks like this: pallets/flask#1829 but it was fixed a long time ago.

[Foknetics]

Noticed this with the release of 0.15.5 as well with an application factory pattern, executing from Windows 10, Python 3.6.2, Flask 1.0.3, via Powershell. Created an example repo here. The change seems to be caused by the function _get_args_for_reloading() in _reloader.py

in 0.15.4 pymodule (line 108) gets set to .flask from the += "." + name which strips down to just flask in rv.

in 0.15.5 pymodule (line 109) gets set to C:\Users\conno\Desktop\werk_ex\venv\Scripts\flask, which strips to the same thing in rv
if os.path.isfile(py_script): (line 100) evaluating to false is really what causes the change from an execution perspective between the two versions.

[davidism]

I can't reproduce this. I have Flask 1.1.1 and Werkzeug 0.15.5, and tried flask run with the reloader in cmd and powershell, and both worked as expected.

[ad-65]

I tried this again on another Windows 10 machine with a system install of Python 2.7.15 (32-bit) and Python 3.7.2 (32-bit). The following steps (in powershell) reproduce the behavior for me. FLASK_ENV=production works fine, FLASK_ENV=development causes the error.

mkdir C:\Temp\flask_error
cd C:\Temp\flask_error
py -2 -m virtualenv env
.\env\Scripts\activate
pip install flask
echo $null >> hello.py

# Add content to hello.py, see below.

$env:FLASK_APP = "hello.py"
$env:FLASK_ENV = "development"
flask run

 * Serving Flask app "hello.py" (lazy loading)
 * Environment: development
 * Debug mode: on
 * Restarting with stat
c:\temp\flask_error\env\scripts\python.exe: No module named C:\Temp\flask_error\env\Scripts\flask

$env:FLASK_ENV = "production"
flask run

 * Serving Flask app "hello.py"
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

The contents of hello.py are from the flask tutorial:

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

[davidism]

I've been discussing this on Discord too, and no matter what I try, everything works for me. I'm doing the same thing everyone else is to set up an env and run it. I'm going to have a hard time fixing this if I can't observe the behavior to debug it. Someone who has the issue is going to need to debug _get_args_for_reloader() and see what values it's getting along the way and figure out where they're coming from.

[davidism]

If someone can do git bisect to find the responsible commit that would help too.

[asottile]

here's the argv / executable for that:

$ FLASK_ENV=development flask run
 * Environment: development
 * Debug mode: on
 * Restarting with stat
> c:\users\anthony\appdata\local\temp\x\venv\lib\site-packages\werkzeug\_reloader.py(68)_get_args_for_reloading()
-> rv = [sys.executable]
(Pdb) p sys.executable
'c:\\users\\anthony\\appdata\\local\\temp\\x\\venv\\scripts\\python.exe'
(Pdb) p sys.argv
['C:\\Users\\Anthony\\AppData\\Local\\Temp\\x\\venv\\Scripts\\flask', 'run']

if I use -m flask run I get this:

$ FLASK_ENV=development python -m flask run
 * Environment: development
 * Debug mode: on
 * Restarting with stat
> c:\users\anthony\appdata\local\temp\x\venv\lib\site-packages\werkzeug\_reloade
r.py(68)_get_args_for_reloading()
-> rv = [sys.executable]
(Pdb) p sys.executable
'C:\\Users\\Anthony\\AppData\\Local\\Temp\\x\\venv\\Scripts\\python.exe'
(Pdb) p sys.argv
['C:\\Users\\Anthony\\AppData\\Local\\Temp\\x\\venv\\lib\\site-packages\\flask\\
__main__.py', 'run']
(Pdb) c
 * Debugger is active!
 * Debugger PIN: 132-133-383
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

[asottile]

oops pressed enter too quickly, I have more info to post

[davidism]

It's expected that flask and -m flask will produce different args. The problem is that for flask you should be getting flask.exe, and the code around that shouldn't have changed.

[asottile]

reverting e686ae7 fixes the issue

here's where things appear to go wrong:

$ FLASK_ENV=development flask run
 * Environment: development
 * Debug mode: on
 * Restarting with stat
> c:\users\anthony\appdata\local\temp\x\werkzeug\src\werkzeug\_reloader.py(95)_get_args_for_reloading()
-> if sys.argv[0] == "-m":
(Pdb) n
> c:\users\anthony\appdata\local\temp\x\werkzeug\src\werkzeug\_reloader.py(101)_get_args_for_reloading()
-> if os.path.isfile(py_script):
(Pdb) py_script
'C:\\Users\\Anthony\\AppData\\Local\\Temp\\x\\venv\\Scripts\\flask'
(Pdb) os.path.isfile(py_script)
False
(Pdb) list
 96                 # Flask works around previous behavior by putting
 97                 # "-m flask" in sys.argv.
 98                 # TODO remove this once Flask no longer misbehaves
 99                 args = sys.argv
100             else:
101  ->             if os.path.isfile(py_script):
102                     # Rewritten by Python from "-m script" to "/path/to/script.py".
103                     py_module = __main__.__package__
104                     name = os.path.splitext(os.path.basename(py_script))[0]
105
106                     if name != "__main__":

[asottile]

that doesn't solve the full mystery though -- why are you unable to reproduce this?

I have a hunch, or at least, I've found a way to not reproduce it:

$ pip install flask --no-binary :all:  # previously I uninstalled flask
Collecting flask
  Downloading https://files.pythonhosted.org/packages/2e/80/3726a729de758513fd3dbc64e93098eb009c49305a97c6751de55b20b694/Flask-1.1.1.tar.gz (625kB)
Requirement already satisfied: Werkzeug>=0.15 in c:\users\anthony\appdata\local\temp\x\werkzeug\src (from flask) (1.0.0.dev0)
Requirement already satisfied: Jinja2>=2.10.1 in c:\users\anthony\appdata\local\temp\x\venv\lib\site-packages (from flask) (2.10.1)
Requirement already satisfied: itsdangerous>=0.24 in c:\users\anthony\appdata\local\temp\x\venv\lib\site-packages (from flask) (1.1.0)
Requirement already satisfied: click>=5.1 in c:\users\anthony\appdata\local\temp\x\venv\lib\site-packages (fromflask) (7.0)
Requirement already satisfied: MarkupSafe>=0.23 in c:\users\anthony\appdata\local\temp\x\venv\lib\site-packages(from Jinja2>=2.10.1->flask) (1.1.1)
Installing collected packages: flask
  Running setup.py install for flask: started
    Running setup.py install for flask: finished with status 'done'
Successfully installed flask-1.1.1
(venv)
Anthony@AnthonysDesktop MINGW64 /tmp/x/werkzeug (master)
$ FLASK_ENV=development flask run
 * Environment: development
 * Debug mode: on
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 158-953-008
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

[frostming]

Same issue here, on windows 10 Python 3.7.4, Flask 1.1.1/werkzeug 0.15.4

Git bisect shows e686ae7 is the first bad commit.

On line 106:

if os.path.isfile(py_script):
    # Rewritten by Python from "-m script" to "/path/to/script.py".

This condition bypasses on windows as the script path is without .exe, so that py_module remains to be an empty string.

[davidism]

Ah, I was installing Flask by doing git checkout 1.1.1 then pip install -e ., which I guess produces different entry point behavior. I wonder if that should be reported to setuptools or pip.

It's strange that the Windows installed entry point makes it look like a python -m flask invocation (from the perspective of __main__.__package__) but that it doesn't get processed correctly back into python -m flask.

I can't just revert that commit, because it fixes another problem. It seems like we need to figure out some better heuristics for the different run cases.

[Nelson-972]

Hello,

I've had a similar issue where I launch a Flask application from an exe in the Scripts folder of the python environment (built using the console_scripts in setuptools).

Basically the reloader thinks this is a regular python package and not an exe because the __package__ in sys.module['__main__'] is an empty string and not None, so I don't enter the condition in _get_args_for_reloading where it adds the .exe in the reloader: if getattr(__main__, "__package__", None) is None:

The workaround to get my application working was to force __package__ to None for now, but this is far from ideal.

Let me know if I'm missing something or if I'm off base in what I just described.

[samayo]

How do we remove werkzeug library from flask? Flask is tagged v1 and we can't even run hello world with it without fatal error

[ThiefMaster]

How do we remove werkzeug library from flask? Flask is tagged v1 and we can't even run hello world with it without fatal error

you can't and you don't. why would you want to do that? flask requires werkzeug. Use the latest werkzeug release together with the latest flask release.

[carc1n0gen]

@samayo while we're at it, let's remove python from flask too. </sarcasm>

More serious: the workaround is python -m flask run until the next Werkzeug release. Calm down.

[davidism]

If anyone wants to look at this, I'd be very happy. I haven't had much free time lately.

[frostming]

@davidism Working on it, it turned out to be an easy fix after some research.

[davidism]

#1627 does work, but goes down the wrong code path when running an example.py module as python -m example. Given that -m example is probably a very uncommon way to run a single module, and given that the reloader still works, I'm willing to go with this change if we can't quickly find a better solution.

[davidism]

I added some debug statements to werkzeug._reloader._get_args_for_reloading to demonstrate the ways the code paths are triggered on Windows. The first line is sys.executable and sys.argv, the last line is the command passed to subprocess.call.

Show debug output

• Installed in editable mode (pip install -e .)

  • flask run

    *  c:\users\david\.virtualenvs\flask\scripts\python.exe ['C:\\Users\\David\\.virtualenvs\\flask\\Scripts\\flask', 'run']
    * __package__ is None, a file
    * adding .exe: 'C:\\Users\\David\\.virtualenvs\\flask\\Scripts\\flask.exe'
    * dropping sys.executable
    *  ['C:\\Users\\David\\.virtualenvs\\flask\\Scripts\\flask.exe', 'run']
    

  • python -m flask run

    *  C:\Users\David\.virtualenvs\flask\Scripts\python.exe ['c:\\users\\david\\projects\\flask\\src\\flask\\__main__.py', 'run']
    * __package__ is not None, a module: 'flask'
    * script is a file, rewritten -m: 'C:\\Users\\David\\.virtualenvs\\flask\\lib\\site-packages\\flask\\__main__.py'
    *  ['C:\\Users\\David\\.virtualenvs\\flask\\Scripts\\python.exe', '-m', 'flask', 'run']
    

  • python example.py

    *  C:\Users\David\.virtualenvs\flask\Scripts\python.exe ['example.py']
    * __package__ is None, a file
    *  ['C:\\Users\\David\\.virtualenvs\\flask\\Scripts\\python.exe', 'C:\\Users\\David\\Projects\\flask\\example.py']
    

  • python -m example

    *  C:\Users\David\.virtualenvs\flask\Scripts\python.exe ['C:\\Users\\David\\Projects\\flask\\example.py']
    * __package__ is not None, a module: ''
    * script is a file, rewritten -m: 'C:\\Users\\David\\Projects\\flask\\example.py'
    * not __main__: '.example'
    *  ['C:\\Users\\David\\.virtualenvs\\flask\\Scripts\\python.exe', '-m', 'example']
    

• Installed from dist (pip install ., dist/flask.tar.gz, dist/flask.whl)

  • flask run (fail)

    *  c:\users\david\.virtualenvs\flask\scripts\python.exe ['C:\\Users\\David\\.virtualenvs\\flask\\Scripts\\flask', 'run']
    * __package__ is not None, a module: ''
    * module is not a file: 'C:\\Users\\David\\.virtualenvs\\flask\\Scripts\\flask'
    *  ['c:\\users\\david\\.virtualenvs\\flask\\scripts\\python.exe', '-m', 'C:\\Users\\David\\.virtualenvs\\flask\\Scripts\\flask', 'run']
    

    c:\users\david\.virtualenvs\flask\scripts\python.exe: Error while finding module specification for 'C:\\Users\\David\\.virtualenvs\\flask\\Scripts\\flask' (ModuleNotFoundError: No module named 'C:\\Users\\David\\')
    

  • Other methods worked the same as -e

Ultimately, this is probably a bug with setuptools, since the entry point it creates on Windows behaves differently for editable vs dist installs.

[davidism]

Closed in #1627