windows support?

[sunwangme]

wonderful project！when to suppert windows?

[1st1]

When we find a volunteer to implement it ;)

[brickgao]

@1st1 Hi, I'm interested in implementing this proposal, is anyone already working on it ;)?

[1st1]

@brickgao As far as I know no one is.

[brickgao]

@1st1 I have a little trouble porting posix.signal to Windows in https://github.com/MagicStack/uvloop/blob/master/uvloop/os_signal.pyx#L1 . I could not find a good way to implement it because of the difference in signal between Windows and *nix. Could you please give me some advice?

[1st1]

I don't think Windows supports signals, so you should probably add checks like if not __WINDOWS__ around code that touches loop.py_signals and loop.uv_signals.

[1st1]

@brickgao Do you think we should find more people to look into this problem?

[brickgao]

@1st1 Sure, I find that there isn't a quick way to port uvloop with the native Windows API. I would try to port uvloop with Cygwin these days first, cuz I find Cygwin provide most of the *NIX API on Windows.

[brickgao]

@1st1 I find that Cygwin haven't provided Python3.5 officially :(, so we should work on it with the native Windows API again. Maybe we should find more people to look into this problem.

There are two major parts we need to port, socket and signal.

For socket, most of socket functions could be easy to ported by changing the header file, but Windows don't provide socketpair. Referring to https://github.com/ncm/selectable-socketpair and https://github.com/python/cpython/blob/c6ae2fcc724cbebb14e7c434b89eabdd64802cb3/Lib/asyncio/windows_utils.py#L37, it's possible to implement a socketpair that support AF_INET and AF_INET6, but it doesn't support AF_UNIX.
For signal, Windows provides a function signal(https://msdn.microsoft.com/en-us/library/xdkz3x12.aspx) to change the handlers of specific signal, but it doesn't provide the function to get the current handler of specific signal and it also doesn't provide sigfillset.

[schlamar]

Libuv already provides a platform agnostic interface to signals and sockets (e.g. http://docs.libuv.org/en/v1.x/signal.html). You shouldn't have to deal with the Windows API at all.

[1st1]

@schlamar You should have looked at uvloop source code first, before assuming that we don't know some basic libuv APIs.

@brickgao Sorry for the delayed reply. For signals specifically I use sigaction to make sure that:

1. uvloop restores CPython's signal handlers
2. Ctrl+C is handled correctly, i.e. PyErr_SetInterrupt is called when uvloop runs Python callbacks.

For the Windows port we don't need (1). We might need to play with Windows APIs to workaround (2), I suggest you to look at how CPython handles Ctrl+C on Windows.

As for socketpair–the whole subprocess handling code is complicated. I had to copy the logic from subprocess.py (and its C-level helper) to make uvloop's subprocess behave exactly like CPython's. Good news is that we can learn how subprocess.py uses Windows APIs, and copy that approach to uvloop.

[iceboy233]

Had a quick hack to make it buildable in windows (win7, python 3.5.2 x64): https://github.com/iceb0y/uvloop/commit/fa2b7db1d7940149bb8d16e856e7d1a642f3e6b8

build snippet:
in /vendor/libuv, use python2

vcbuild.bat x64 release

in /, use python3, do the same as makefile

cython -3 -a -p uvloop/loop.pyx
python setup.py build_ext --inplace

Had a test with aiohttp server - the current stucking problem is that uvloop uses os_dup in multiple places to duplicate UVSocketHandle._fileno, where os_dup = os.dup and it duplicates posix file handles. In windows, posix file handle is implemented as an additional layer on NT handles where the handle numbers are not equal. I workaround this problem and aiohttp server works.

Notice that os_dup is also used to duplicate posix handle in uvloop/loop.pyx and uvloop/handles/process.pyx. So we need to differentiate between this two kinds of dups. Also note that duplicating socket handles in windows is fragile (see http://bugs.python.org/issue14522). If we can share socket by sharing handle objects instead of duplicating OS handles, we can resolve this problem and avoid the dup system call.

[iceboy233]

dup is introduced in 6e9c43b

maybe we need to find another way to solve the problem described by the commit?

[1st1]

Your PR looks very good, thanks a lot for working on this. I believe we don't need to do anything special for signals or forks. As for dup, I'll fix a bug in asyncio and change how uvloop implements get_extra_info in a couple of weeks.

[iceboy233]

Sharing handle objects seems not working. uv still complains Assertion failed: (loop->watchers[w->fd] == w), function uv__io_stop, file src/unix/core.c, line 888.

Changing socket.socket to socket.fromfd, which uses _socket.dup which has special implementation for windows, without dup just works (tested in osx and windows). But I still feel a little strange to duplicate a socket handle.

What do you plan for dup?

[brickgao]

Some workarounds to the signal based on @iceb0y 's commits: brickgao@eb9b9b7

[1st1]

What do you plan for dup?

Just to remove it completely. Right now it's there to save uvloop from segfaulting when someone uses sockets from transports with add_writer API.

[asvetlov]

@1st1 I've fixed sendfile support on aiohttp master.
Funny fact: on travis dupped socket was in blocking mode.
Honestly I've not dug into but had added just socket.setblocking(False) call.

[1st1]

@1st1 I've fixed sendfile support on aiohttp master.

Super cool! Thanks!

Funny fact: on travis dupped socket was in blocking mode.

Wow. Maybe this is something worth investigating? Is it something that asyncio causes?

[iceboy233]

Some workarounds to the signal based on @iceb0y 's commits: brickgao/uvloop@eb9b9b7

@1st1 I think the windows signal code is actually platform-independent

1. Is it correct?
2. Should we just use it for all platforms for simplification?

[asvetlov]

Funny fact: on travis dupped socket was in blocking mode.

Wow. Maybe this is something worth investigating? Is it something that asyncio causes?

@1st1 I cannot reproduce the problem locally. The only error what I have is travis report: https://travis-ci.org/KeepSafe/aiohttp/builds/157838616
Something like this:

    resp = yield from sender().send(request, filepath)
  File "/opt/python/3.5.0/lib/python3.5/asyncio/coroutines.py", line 105, in __next__
    return self.gen.send(None)
  File "/home/travis/build/KeepSafe/aiohttp/aiohttp/file_sender.py", line 148, in send
    yield from self._sendfile(request, resp, f, file_size)
  File "/opt/python/3.5.0/lib/python3.5/asyncio/coroutines.py", line 105, in __next__
    return self.gen.send(None)
  File "/home/travis/build/KeepSafe/aiohttp/aiohttp/file_sender.py", line 86, in _sendfile_system
    ''.join(headers).encode('utf-8'))
  File "uvloop/loop.pyx", line 1831, in sock_sendall (uvloop/loop.c:29654)
ValueError: the socket must be non-blocking

sock.setblocking(False) has solved the problem.
I have no Idea why it works with some configurations but fails with others.

[1st1]

@iceb0y In the latest version of uvloop I've redesigned how signals are implemented. No more libuv or system API calls, I now only use Python's signal module. This should help with the Windows port.

[iceboy233]

tested on windows - the signal handling code doesn't work on windows, probably due to usage of socketpair.

Loop._setup_signals -> Loop._add_reader -> UVPoll.new -> UVPoll._init -> uv.uv_poll_init -> OSError: [Errno 4050] Unknown error.

Why does signal handling use a socketpair which seems to post notification to the loop?

[1st1]

FWIW I'm working on to add Windows support in https://github.com/MagicStack/uvloop/tree/win. Any help is welcome.

[1st1]

Here's the PR #62

[Martmists-GH]

For those on windows 10, I believe you can use bash on ubuntu and install uvloop on there for now, or at least that seems to have worked for me.

[1st1]

For those on windows 10, I believe you can use bash on ubuntu and install uvloop on there for now, or at least that seems to have worked for me.

Great! FWIW I don't have time to focus on the IOCP win port right now, but I can definitely help with reviews/suggestions if someone is interested in it.

[floqqi]

@1st1 @luckcolors would be interesting for me, too! Any chance? Would be great to see frameworks like Sanic working on Windows :)

[1st1]

Would be great to see frameworks like Sanic working on Windows :)

Nothing prevents sanic from working on Windows. uvloop should be an optional dependency.

[aknaebel]

Some project like uvicorn (https://github.com/encode/uvicorn) use uvloop and cannot be used on windows which is very restrictive

[1st1]

uvloop should be an optional dependency. It doesn't provide any extra APIs on top of vanilla asyncio, it's an accelerator. It happens that uvloop only works on posix right now and it's OK.

Please file a bug report to uvicorn to make uvloop dep optional.

[dakyri]

Yuri, what's the state of play with getting the windows port working? I might be interested getting this done (I have a use case for it) ... at least as one contributor. I've worked with libuv in c/c++, so I know my way around that. Any idea how much work would be invoved time wise to get something functional?

[1st1]

I almost got it working, except cases where the API accepts an existing socket to work with. I.e. create_server(sock=sock), not create_server(addr). The best approach would be to build uvloop on Windows with the patch (#62) applied and work from that point.

[dakyri]

Cool! That sounds like it's viable or close enough for me as it stands already, but I'll be happy to pick up the ball and contribute moving forwards.

[matemax]

@1st1 ,can you merge master into win branch?

[1st1]

I'll take a look on Monday. Are you using the win branch?

[matemax]

No, but our project is using uvloop and python 3.6. I am going to make it cross-platform. 

[1st1]

I've rebased it. Have no idea if it still compiles on Windows, but it does compile OK on *nix after the rebase.

[matemax]

I tried to instal branch 'win' and got folowing errors.

Failed build

warning: uvloop\handles/stream.pyx:318:8: Unreachable code warning: uvloop\loop.pyx:2750:4: Unreachable code

Error compiling Cython file:
------------------------------------------------------------
...
            self._fatal_error(exc, True)
            return

        cdef:
            int backend_id
            system.epoll_event dummy_event
           ^
------------------------------------------------------------

uvloop\handles/poll.pyx:67:12: 'epoll_event' is not a type identifier

Error compiling Cython file:
------------------------------------------------------------
...

        cdef:
            int backend_id
            system.epoll_event dummy_event

        if system.PLATFORM_IS_LINUX:
                ^
------------------------------------------------------------

uvloop\handles/poll.pyx:69:17: cimported module has no attribute 'PLATFORM_IS_LINUX'

Error compiling Cython file:
------------------------------------------------------------
...
            # after calling uv_poll_stop.

            backend_id = uv.uv_backend_fd(self._loop.uvloop)
            if backend_id != -1:
                memset(&dummy_event, 0, sizeof(dummy_event))
                system.epoll_ctl(
                     ^
------------------------------------------------------------

uvloop\handles/poll.pyx:82:22: cimported module has no attribute 'epoll_ctl'

Error compiling Cython file:
------------------------------------------------------------
...
            backend_id = uv.uv_backend_fd(self._loop.uvloop)
            if backend_id != -1:
                memset(&dummy_event, 0, sizeof(dummy_event))
                system.epoll_ctl(
                    backend_id,
                    system.EPOLL_CTL_DEL,
                         ^
------------------------------------------------------------

uvloop\handles/poll.pyx:84:26: cimported module has no attribute 'EPOLL_CTL_DEL'

Error compiling Cython file:
------------------------------------------------------------
...
                    with open(self._errpipe_write, 'wb') as f:
                        f.write(str(ex.__class__.__name__).encode())
                        f.write(b':')
                        f.write(str(ex.args[0]).encode())
                finally:
                    system._exit(255)
                         ^
------------------------------------------------------------

uvloop\handles/process.pyx:173:26: cimported module has no attribute '_exit'

Error compiling Cython file:
------------------------------------------------------------
...
                    with open(self._errpipe_write, 'wb') as f:
                        f.write(str(ex.__class__.__name__).encode())
                        f.write(b':')
                        f.write(str(ex.args[0]).encode())
                finally:
                    system._exit(255)
                         ^
------------------------------------------------------------

uvloop\handles/process.pyx:173:26: cimported module has no attribute '_exit'

Error compiling Cython file:
------------------------------------------------------------
...
cdef __socketpair():
    cdef:
        int fds[2]
        int err

    err = system.socketpair(uv.AF_UNIX, uv.SOCK_STREAM, 0, fds)
               ^
------------------------------------------------------------

uvloop\handles/process.pyx:719:16: cimported module has no attribute 'socketpair'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Maxim\AppData\Local\Temp\pip-joj6i769-build\setup.py", line 370, in <module>
    test_suite='tests.suite'
  File "c:\python36\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "c:\python36\lib\distutils\dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "c:\python36\lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "c:\python36\lib\site-packages\setuptools\command\egg_info.py", line 279, in run
    self.find_sources()
  File "c:\python36\lib\site-packages\setuptools\command\egg_info.py", line 306, in find_sources
    mm.run()
  File "c:\python36\lib\site-packages\setuptools\command\egg_info.py", line 533, in run
    self.add_defaults()
  File "c:\python36\lib\site-packages\setuptools\command\egg_info.py", line 562, in add_defaults
    sdist.add_defaults(self)
  File "c:\python36\lib\site-packages\setuptools\command\py36compat.py", line 36, in add_defaults
    self._add_defaults_ext()
  File "c:\python36\lib\site-packages\setuptools\command\py36compat.py", line 119, in _add_defaults_ext
    build_ext = self.get_finalized_command('build_ext')
  File "c:\python36\lib\distutils\cmd.py", line 299, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "c:\python36\lib\distutils\cmd.py", line 107, in ensure_finalized
    self.finalize_options()
  File "C:\Users\Maxim\AppData\Local\Temp\pip-joj6i769-build\setup.py", line 138, in finalize_options
    annotate=self.cython_annotate)
  File "c:\python36\lib\site-packages\Cython\Build\Dependencies.py", line 1039, in cythonize
    cythonize_one(*args)
  File "c:\python36\lib\site-packages\Cython\Build\Dependencies.py", line 1161, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: uvloop/loop.pyx
uvloop/loop.pyx: cannot find cimported module '.includes'
uvloop/loop.pxd: cannot find cimported module '.includes'
Compiling uvloop/loop.pyx because it changed.
[1/1] Cythonizing uvloop/loop.pyx

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in C:\Users\Maxim\AppData\Local\Temp\pip-joj6i769-build\

[nsavelyeva]

Hi All,
I'm looking forward to this feature, too. Respect and many thanks to all devs involved.
I tried today 'pip install uvloop' on Windows 10 and it failed with RuntimeError: uvloop does not support Windows at the moment.

[AymanEG]

Hello from 2018 , good luck guys completing it :)

[griimnak]

Good luck, windows support would still be appreciated

[ofek]

Any progress?

[tweakimp]

It would be helpful if it at least said in the readme that it currently has no Windows support.

[ma5oud180]

Hi Yuri, I understand that you are super busy, but I myself & as I see lots of people, are not able and have not the skills to make it work on Windows. I would like to kindly ask you to finally, easily make it possible to install this package on Windows 64 bit systems. Lots of people are looking for it. Many thanks

[jmclemo6]

Any updates on this? If I could be caught up to speed on what's been done and discovered, I think I could start working on it.

[ofek]

I'd love to help test!

[1st1]

Any updates on this? If I could be caught up to speed on what's been done and discovered, I think I could start working on it.

Sure, by all means. I'd start with trying to recover the win branch. Please open new issues as you progress through this.

[1st1]

I'm locking this issue. I don't have time to work on this in my personal time. It doesn't make sense to work on this during my work time. If the community really needs this then someone will start submitting PRs which I'll happily review and help with advice.