Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

M1 mac with Docker WatchfilesRustInternalError: Error creating watcher: Function not implemented (os error 38) #167

Closed
daniel-butler opened this issue Jul 10, 2022 · 18 comments · Fixed by #172
Labels

Comments

@daniel-butler
Copy link

Description

When starting django with the watcher (after installing from pypi from the wheel)

PostgreSQL is available
INFO:     Will watch for changes in these directories: ['/app']
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [18] using WatchFiles
Traceback (most recent call last):
  File "/usr/local/bin/uvicorn", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
=== Starting migration
Operations to perform:
  Apply all migrations: account, address, admin, auth, contenttypes, customers, orders_tenant, permissions, sessions, socialaccount, users
Running migrations:
  No migrations to apply.
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/main.py", line 407, in main
    run(
  File "/usr/local/lib/python3.9/site-packages/uvicorn/main.py", line 570, in run
    ChangeReload(config, target=server.run, sockets=[sock]).run()
  File "/usr/local/lib/python3.9/site-packages/uvicorn/supervisors/basereload.py", line 45, in run
    for changes in self:
  File "/usr/local/lib/python3.9/site-packages/uvicorn/supervisors/basereload.py", line 64, in __next__
    return self.should_restart()
  File "/usr/local/lib/python3.9/site-packages/uvicorn/supervisors/watchfilesreload.py", line 85, in should_restart
    changes = next(self.watcher)
  File "/usr/local/lib/python3.9/site-packages/watchfiles/main.py", line 103, in watch
    watcher = RustNotify([str(p) for p in paths], debug, force_polling, poll_delay_ms)
_rust_notify.WatchfilesRustInternalError: Error creating watcher: Function not implemented (os error 38)

Example Code

No response

Example Code Output

No response

Operating System

Mac M1 12.3.1 (21E258)

Environment

Docker

Watchfiles Version

0.15.0

Python Version

3.9

Rust & Cargo Version

cargo: not found; rustc: not found

@samuelcolvin
Copy link
Owner

Which docker image?

@daniel-butler
Copy link
Author

Sorry I forgot to include that it is python:3.9-slim-bullseye

I'm running docker in intel chip emulation mode, too. I don't remember the correct technical name for it.

@samuelcolvin
Copy link
Owner

Maybe worth checking if the correct wheel was installed.

I've run watchfiles inside docket on my M1 Mac and it worked fine.

@br3ndonland
Copy link

I can reproduce this without Django or Uvicorn. The error occurs on an Apple Silicon M1 Mac, but not on an Intel Mac, and occurs even if specifying the platform.

watch.py

from watchfiles import watch

for changes in watch('/hello.py'):
    print(changes)

hello.py

print('hello world')

Dockerfile

FROM python:3.10
RUN pip install watchfiles==0.15.0
COPY hello.py watch.py /
CMD watchfiles watch /hello.py
docker build . --platform linux/amd64 -t watchfiles-example
docker run -p 8080:80  --platform linux/amd64 -v "$(pwd)/hello.py":"/hello.py" watchfiles-example
[20:55:51] watchfiles 👀  path="/hello.py" target="watch" (command) filter=DefaultFilter...
Traceback (most recent call last):
  File "/usr/local/bin/watchfiles", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.10/site-packages/watchfiles/cli.py", line 150, in cli
    run_process(
  File "/usr/local/lib/python3.10/site-packages/watchfiles/run.py", line 130, in run_process
    for changes in watch(
  File "/usr/local/lib/python3.10/site-packages/watchfiles/main.py", line 103, in watch
    watcher = RustNotify([str(p) for p in paths], debug, force_polling, poll_delay_ms)
_rust_notify.WatchfilesRustInternalError: Error creating watcher: Function not implemented (os error 38)

Consider reporting to Docker upstream (docker/roadmap#7).

@gravy-jones-locker
Copy link

+1 on this

@samuelcolvin
Copy link
Owner

Okay, I can reproduce this.

First the good news:

  1. I don't get this error (and watchfiles seems to correctly detect changes) if I remove --platform linux/amd64
  2. Same if I use force_polling=True
  3. As per the traceback, watchfiles is catching this error, so we should be able to catch the error and gracefully fall back to polling

It might also be worth honouring an env variable to enable polling for those who are using watchfiles via a third party library and therefore aren't able to use the kwarg directly.

In terms of the underlying error and how to handle it, I'm not really sure where to go.

@messense do you have any ideas?

@samuelcolvin
Copy link
Owner

samuelcolvin commented Jul 19, 2022

From the logs, pip is installing watchfiles-0.15.0-cp37-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

I tried hacking pip to allow me to install other wheels directly and tried installing watchfiles-0.15.0-cp37-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl and watchfiles-0.15.0-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl on the off-chance that worked, but it gave me a wrong ELF class: ELFCLASS32 error.

I also tried install 0.14.1 in case it was an issue with abi3, but it doesn't seem to be.

@messense
Copy link
Collaborator

I cannot reproduce this with lima on M1 Pro Mac.

root@49fb5ab8d2ea:/# watchfiles watch /hello.py
[02:19:00] watchfiles 👀  path="/hello.py" target="watch" (command) filter=DefaultFilter...

Usage:
 watch [options] command

Options:
  -b, --beep             beep if command has a non-zero exit
  -c, --color            interpret ANSI color and style sequences
  -d, --differences[=<permanent>]
                         highlight changes between updates
  -e, --errexit          exit if command has a non-zero exit
  -g, --chgexit          exit when output from command changes
  -n, --interval <secs>  seconds to wait between updates
  -p, --precise          attempt run command in precise intervals
  -t, --no-title         turn off header
  -w, --no-wrap          turn off line wrapping
  -x, --exec             pass command to exec instead of "sh -c"

 -h, --help     display this help and exit
 -v, --version  output version information and exit

For more details see watch(1).
^C[02:19:03] KeyboardInterrupt caught, stopping watch
[02:19:03] process already dead, exit code: 1

@samuelcolvin
Copy link
Owner

I think I'll ask on notify and see if they have any ideas.

@samuelcolvin
Copy link
Owner

Partial fix / work around building now in #170.

Reported upstream, see notify-rs/notify#423.

@messense
Copy link
Collaborator

FYI, https://docs.docker.com/desktop/mac/apple-silicon/#known-issues

However, attempts to run Intel-based containers on Apple silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation.

@samuelcolvin
Copy link
Owner

Amazing, thanks for finding that!

Maybe one day notify will catch this and fall back to polling, in the meantime we can catch this and fallback in watchfiles, I'll create a PR.

@samuelcolvin
Copy link
Owner

samuelcolvin commented Jul 20, 2022

Okay, this is fixed in #172.

Please could those experiencing this problem @daniel-butler @br3ndonland @gravy-jones-locker try this and confirm it fixes the problem for you.

To install from wheel from the PR:

  • Download pypi_files from the CI run
  • Extract the zip
  • pip install pypi_files/watchfiles-0.0.0-cp37-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl

@daniel-butler
Copy link
Author

daniel-butler commented Jul 20, 2022

It is working for me when I use the updated wheel! Thank you for fixing!

PostgreSQL is available
=== Starting migration
Operations to perform:
  Apply all migrations: account, address, admin, auth, contenttypes, customers, orders_tenant, permissions, sessions, socialaccount, users
Running migrations:
  No migrations to apply.
INFO:     Will watch for changes in these directories: ['/app']
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [18] using WatchFiles
INFO:     Started server process [24]
INFO 2022-07-20 10:56:05,658 server 24 274910471360 Started server process [24]
INFO:     Waiting for application startup.
INFO 2022-07-20 10:56:05,662 on 24 274910471360 Waiting for application startup.
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO 2022-07-20 10:56:05,664 on 24 274910471360 ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.
INFO 2022-07-20 10:56:05,665 on 24 274910471360 Application startup complete.
INFO:     172.18.0.1:64966 - "GET /hub/work/order/ HTTP/1.1" 200 OK
INFO:     172.18.0.1:64974 - "GET /static/debug_toolbar/css/toolbar.css HTTP/1.1" 200 OK
INFO:     172.18.0.1:64972 - "GET /static/css/project.min.css HTTP/1.1" 200 OK
INFO:     172.18.0.1:64976 - "GET /static/debug_toolbar/js/toolbar.js HTTP/1.1" 200 OK
INFO:     172.18.0.1:64972 - "GET /static/debug_toolbar/css/print.css HTTP/1.1" 200 OK
INFO:     172.18.0.1:64972 - "GET /static/debug_toolbar/js/utils.js HTTP/1.1" 200 OK
INFO:     172.18.0.1:64988 - "GET /hub/work/home/ HTTP/1.1" 200 OK
INFO:     172.18.0.1:64992 - "GET /static/css/project.min.css HTTP/1.1" 304 Not Modified
INFO:     172.18.0.1:64994 - "GET /static/debug_toolbar/js/toolbar.js HTTP/1.1" 304 Not Modified
INFO:     172.18.0.1:64996 - "GET /static/debug_toolbar/css/toolbar.css HTTP/1.1" 304 Not Modified
INFO:     172.18.0.1:64994 - "GET /static/debug_toolbar/css/print.css HTTP/1.1" 304 Not Modified
INFO:     172.18.0.1:64992 - "GET /static/debug_toolbar/js/utils.js HTTP/1.1" 304 Not Modified
INFO 2022-07-20 11:00:44,302 workhub 24 275185379072 form_action = /hub/work/task-work/2/
INFO:     172.18.0.1:65008 - "GET /hub/work/task-work/2/ HTTP/1.1" 200 OK
INFO:     172.18.0.1:65014 - "GET /static/debug_toolbar/css/toolbar.css HTTP/1.1" 304 Not Modified
INFO:     172.18.0.1:65016 - "GET /static/debug_toolbar/js/toolbar.js HTTP/1.1" 304 Not Modified
INFO:     172.18.0.1:65012 - "GET /static/css/project.min.css HTTP/1.1" 304 Not Modified
INFO:     172.18.0.1:65014 - "GET /static/debug_toolbar/js/utils.js HTTP/1.1" 304 Not Modified
INFO:     172.18.0.1:65016 - "GET /static/debug_toolbar/css/print.css HTTP/1.1" 304 Not Modified
WARNING:  WatchFiles detected changes in 'listing_service/orders_tenant/models.py'. Reloading...
INFO:     Shutting down
INFO 2022-07-20 11:01:38,647 server 24 274910471360 Shutting down
INFO:     Finished server process [24]
INFO 2022-07-20 11:01:38,749 server 24 274910471360 Finished server process [24]
INFO:     Started server process [49]
INFO 2022-07-20 11:01:48,922 server 49 274910471360 Started server process [49]
INFO:     Waiting for application startup.
INFO 2022-07-20 11:01:48,926 on 49 274910471360 Waiting for application startup.
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO 2022-07-20 11:01:48,929 on 49 274910471360 ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.
INFO 2022-07-20 11:01:48,930 on 49 274910471360 Application startup complete.
INFO 2022-07-20 11:01:51,662 workhub 49 275016066816 form_action = /hub/work/task-work/2/
INFO:     172.18.0.1:65038 - "GET /hub/work/task-work/2/ HTTP/1.1" 200 OK
INFO:     172.18.0.1:65042 - "GET /static/debug_toolbar/css/toolbar.css HTTP/1.1" 304 Not Modified
INFO:     172.18.0.1:65046 - "GET /static/debug_toolbar/js/toolbar.js HTTP/1.1" 304 Not Modified
INFO:     172.18.0.1:65044 - "GET /static/css/project.min.css HTTP/1.1" 304 Not Modified
INFO:     172.18.0.1:65042 - "GET /static/debug_toolbar/js/utils.js HTTP/1.1" 304 Not Modified
INFO:     172.18.0.1:65046 - "GET /static/debug_toolbar/css/print.css HTTP/1.1" 304 Not Modified
WARNING:  WatchFiles detected changes in 'listing_service/orders_tenant/models.py'. Reloading...
INFO:     Shutting down
INFO 2022-07-20 11:02:02,787 server 49 274910471360 Shutting down
INFO:     Finished server process [49]
INFO 2022-07-20 11:02:02,889 server 49 274910471360 Finished server process [49]
INFO:     Started server process [61]
INFO 2022-07-20 11:02:13,613 server 61 274910471360 Started server process [61]
INFO:     Waiting for application startup.
INFO 2022-07-20 11:02:13,616 on 61 274910471360 Waiting for application startup.
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO 2022-07-20 11:02:13,619 on 61 274910471360 ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.
INFO 2022-07-20 11:02:13,621 on 61 274910471360 Application startup complete.

@samuelcolvin
Copy link
Owner

samuelcolvin commented Jul 20, 2022

Great, thank you for confirming. I'll make a new release tomorrow.

samuelcolvin added a commit that referenced this issue Jul 21, 2022
* fallback to poll watcher on OS Error 38, fix #167

* tweak error on fallback poll watcher
@samuelcolvin
Copy link
Owner

v0.16.0 has just be released.

@daniel-butler
Copy link
Author

Nice! It works with 0.16.0 from PyPI, too.

@gravy-jones-locker
Copy link

missed the notifications for these. but the immediate issue is definitely fixed. thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants