Replace libuv

[talex5]

libuv is not suitable for an Eio backend. Problems include:

• Resources are attached to loops, which doesn't work if you want to share them between domains (Luv: deadlock sharing resources between domains? #386).
• It interferes with signals (luv fails to handle signals promptly #400), requiring a work-around.
• The OCaml bindings are unmaintained (eio_luv: write with wrong size on s390x #433, Eio.Path.load loads only first 4gb on big files #335, eio_luv: Process.spawn → “Switch hook removed from wrong domain!” #446).
• It interferes with other effect handlers (eio_luv: Losing effect handler scope #427).
• File operations can't be cancelled (Can't cancel reading on stdin with Luv #373, Opening a named pipe hangs libuv #357).
• It provides a second abstraction layer (Eio chooses a backend for your platform, then libuv chooses another one). It would be simpler to have only one OS abstraction.
• It doesn't support many useful features (e.g. reuse_addr and reuse_port parameters are ignored in luv #366, openat).

The main reason to use it was to get Windows support for free. However, that doesn't fully work (aantron/luv#142) and we hope to provide a native Windows backend instead.

On other systems, we now use the much simpler Eio_posix backend.

[patricoferris]

In case it is useful I have a mostly working (but needs more testing and of course reviewing!) kqueue-based backend https://github.com/patricoferris/eio/tree/kqueue/lib_eio_kqueue. I'm a little more confident in this backend for macOS than GCD at the moment and we might get BSDs out of it too (although kqueue implementations vary between them all it seems)

[tmcgilchrist]

Out of interest @patricoferris what is the issue with using macOS GCD? Are there technical limitations or is it a matter of developer time?

[patricoferris]

Out of interest @patricoferris what is the issue with using macOS GCD? Are there technical limitations or is it a matter of developer time?

Good question! I've been hacking away without really completing both a kqueue-based backend and a GCD backend whilst reading quite a bit about both, and here's my opinionated thoughts on that matter:

• From the outside it would seem GCD is not the best option for something like macOS -- I think it can still make sense for some applications (e.g. iOS) but after scouring many a forum, this is the general picture I'm getting from devs working in the Apple ecosystem. Have a read of this forum discussion https://forums.swift.org/t/task-safe-way-to-write-a-file-asynchronously/54639/ (I find it quite compelling that SwiftNIO did not choose GCD for macOS, they do have an extension to support more Apple devices that does use GCD for some things https://github.com/apple/swift-nio-extras).
• I feel like GCD is an all or nothing approach when used how we want to use it. GCD is supposed to be clever enough to manage how many threads are spawned to handle your queues (in the case of concurrent queues it may even thread explode), I'm worried how this will play with a domainslib-style domain pool mechanism.
• All callbacks, just like luv, have to be wrapped and are then called in another thread so there's a price to pay for acquiring the runtime lock and then this opens us up to strange behaviour I think similar to some of the issues talex5 mentioned above.

In the long run, on macOS, I think we'll benefit from being in total control of the OS resources ourselves. We can use kqueues for networking and then do some systhread worker pool magic for File I/O, this way we only have to fight with ourselves and not some other library too! I don't think the GCD backend is completely dead, but I think the kqueue support in Eio_posix is probably the first most important thing to do :))

[talex5]

Status update: we now have an Eio_posix backend (#448) that fixes these issues (though there are a few things to improve: #456). Eio_luv is now only used on Windows.

[samwgoldman]

we hope to provide a native Windows backend instead

The Flow team at Meta would be interested in this. We current use Lwt but have run into a number of Windows-related issues, so we're looking closely at Eio as we move to OCaml 5. I'm happy to help contribute to this.

[avsm]

@samwgoldman, glad to hear it! Just to help scope out your use of Windows here, are you primarily interested in a CLI to generate a flow.exe, or other Win32 API integration points?

[avsm]

The direct issue to supporting Windows is in #125, so best to switch discussion there from this one about libuv, @samwgoldman.