Make Ray work on Windows.

[robertnishihara]

Click here for instructions on how to use Ray on Windows (natively)

This is long term, but I'm creating this issue so we have a place to collect our thoughts. I'd like to have a better understanding of what's involved.

There has been substantial progress on this in #93 by @mehrdadn. The PR is outdated, but that should give a sense of what's involved.

One of the primary issues I believe was the heavy usage of system calls that don't have exact equivalents in Windows. Related to this, see code under src/common/shims/windows/.

Another issue had to do with thirdparty libraries (e.g., flatbuffers, Redis) that had issues on Windows. One approach is to fork these projects and make necessary modifications. There is a Windows port of Redis https://github.com/MSOpenTech/redis, but it's not clear how up-to-date it is, and we're relying on some pretty recent Redis features.

[mehrdadn]

For future reference: if my memory serves me right (it might not be), the main problem I recall having was the impedance mismatch between the networking I/O models. I was using Redis's compatibility layer to make the rest of Ray to work properly on Windows, so it wasn't just a matter of getting the Redis server itself to run. I ended up getting tons of hangs/crashes that were difficult to track down.

[pcmoritz]

Also, arrow is now officially supporting Windows. If there is a good portable C++ socket implementation that we could use to make the porting easier let us know.

[mehrdadn]

Arrow was never really a problem on Windows.

For sockets in C++, I believe the one and only consistent recommendation I've seen has been Boost.Asio. It's fairly heavyweight as you might expect. Apparently it has a non-Boost version too. Other recommendations for C++ can be found here but I don't really know anything about them.

[pcmoritz]

The standalone asio library seems the best candidate here. Boost::asio seems to be popular, a variant of it might even become part of the c++ standard in the future (see https://github.com/chriskohlhoff/networking-ts-impl) and it would also make it easier to support IPv6 in the future I think.

[flyingofftheshelves]

Any new developments with this?

[robertnishihara]

@flyingofftheshelves This is on hold for now until we find a Windows expert to help make it happen. In the meantime, some people have been successfully using Ray on the Linux subsystem for Windows (cc @devin-petersohn).

[richardliaw]

There's another open source fork of Redis for Windows that seems to be well-maintained here: https://github.com/tporadowski/redis

[robertnishihara]

Nice!

[jengelman]

For what it's worth, we use the MSOpenTech Redis port pretty heavily in production without major issues
Edit: we've switched over to the version @richardliaw linked, pretty close to the current version of Redis and big performance improvement

[felixludos]

How close are you currently to getting Ray to work on Windows?

[ErnstDinkelmann]

Hi there
I want to use Ray Tune for hyper parameter optimisation of a tensorflow graph. I work in windows.
I've been able to use a remote interpreter in Pycharm that sits on the Ubuntu (on WSL).
But now, even though Ray is able to execute, there is no support for CUDA/GPUs yet in WSL, the way I understand this.
I hate to ask this here, but are there good alternatives to Ray that do work on Windows. which make use of similar concepts to find the best hyper-parameters?

[eromoe]

Does windows support on progress ?

[iandanforth]

Any progress here? Seems like making cross-platform compatible architectural changes might be in order.

[robertnishihara]

There actually has been some progress, but there's still a bunch of work to do. @suquark has started replacing the underlying event loop in the Plasma shared memory object store (which uses the same event loop as Redis and which doesn't work on Windows) with the asio library. From there it will be easier to make the object store work with Windows. Once the shared memory details are worked out, then there are some more things to change (Unix domain sockets, some system calls, and using a Windows port of Redis).

[jengelman]

FWIW, you can run regular Redis on Windows 10 using Windows Subsystem for Linux (https://redislabs.com/blog/redis-on-windows-10/) with basically native performance, so no need to rely on the port. We recently switched from using the Windows port as a Celery backend and no complaints so far.

[robertnishihara]

@jengelman is there an easy way to programmatically start a Redis server on the Windows Linux Subsystem from a Python script running outside of the Subsystem?

[mehrdadn]

You might want to check if WSL Redis would have native performance—if it uses some kind of tmpfs such as for memory-mapped files (shared memory etc.) then on WSL that will hit the disk rather than occurring in RAM.

[jengelman]

@robertnishihara You can invoke wsl from cmd (ex: wsl redis-server start), so just need a call to subprocess.Popen

[gowthamnatarajan]

Since the windows support is being worked on, if I have to use RLLIB now, the only was is to install linux directly right? Installing linux as a VM inside Windows won't work because it cannot access my GPU?

[juliusfrost]

Thank you for doing this! It really helps even when just developing on windows and running large scale experiments on a Linux cluster.

[mehrdadn]

Hi everyone! Exciting news:

Experimental Windows support is now officially released. :)

Just a heads up: it is alpha quality. It is currently incomplete and some things are known to be unstable/broken. Best results will come from running it in a typical, "friendly" environment. 🙂 Please read the release notes for important details.

If you encounter any issues, please check the latest nightly wheels since the release. If the issue still exists, check the known Windows issues page (#9114), as well as other issues (as some issues may not be Windows-specific). If you encounter any problems that are not yet being addressed, please let us know!

We'll have more information coming soon, including in the documentation. But most importantly, to run Ray on Windows, ensure you have the VC++ 2019 runtime, otherwise you may get errors about dependencies not being found.

I'll close this issue, so you can track the Known Issues page (#9114) for further updates on Windows development.

P.S. Shout-out to https://github.com/tporadowski/redis for making my life much easier with Redis on Windows. Without it, we'd have probably needed to introduce a WSL dependency just for Redis. :)