Function not implemented (os error 38) when running on docker in M1 MacOS

[samuelcolvin]

System details

See samuelcolvin/watchfiles#167 for initial report

• OS/Platform name and version: docker running on M1 mac with the --platform linux/amd64 flag set
• Rust version (if building from source): rustc --version: rustc 1.61.0 (fe5b13d68 2022-05-18)
• Notify version (or commit hash if building from git): =5.0.0-pre.15
• If you're coming from a project that makes use of Notify, what it is, and a link to the downstream issue if there is one:
  • watchfiles, issue: M1 mac with Docker WatchfilesRustInternalError: Error creating watcher: Function not implemented (os error 38) samuelcolvin/watchfiles#167
  • and upstream from there: [Apple M1 Docker] (_rust_notify.WatchfilesRustInternalError: Error creating watcher: Function not implemented (os error 38)) adamchainz/django-watchfiles#3
• Filesystem type and options: Mac OS with M1 architecture running docker using the python:3.10 image

What you did (as detailed as you can)

When running watchfiles, the following error occurs: Function not implemented (os error 38)

In watchfiles code, the error is caught and raised here.

This occurs when using RecommendedWatcher, when using PollWatcher everything seems to be fine.

There are a number of other similar reports of the same error message on other rust packages running in docker in the same way, see adamchainz/django-watchfiles#3. Those seem to related to missing binaries for MacOS M1, but watchfiles provides (AFAIK) all applicable binaries.

I think this is a problem with docker not providing the underlying functions which notify expects to find.

I think I can catch this error and fall back to PollWatcher in watchfiles but I wanted to report the issue here in case you had any insights or suggestions.

[samuelcolvin]

A bit more detail on this:

As discovered by @messense, https://docs.docker.com/desktop/mac/apple-silicon/#known-issues

However, attempts to run Intel-based containers on Apple silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation.

So I think notify should detect this and fallback to PollWatcher.

In case it helps, debug of the error is: { kind: Io(Os { code: 38, kind: Unsupported, message: "Function not implemented" }), paths: [] }.

I'm working round this in watchfiles, see samuelcolvin/watchfiles#172.

[0xpr03]

I sadly can't test this at all, as I don't own an m1 device. And this can't be simulated by gh-actions?

Is there a specific conditional compilation architecture we could detect for this ? If possible I'd like to actively detect the feature instead of trying for error 38 first on macos.

[samuelcolvin]

Ye, I get the problem.

You won't be able to test on gh-actions or any other cloud service AFAIK.

I don't know of any to actively detect, but I guess there must be a way?

[messense]

Is there a specific conditional compilation architecture we could detect for this ?

I don't think there is. The target architecture is x86_64. You'd need to detect QEMU emulation at runtime, but that's not pretty either. https://stackoverflow.com/a/71643762

[0xpr03]

So the question is: Do we want to add this error-code workaround into notify ?

The default-watcher is just an alias, so if you'd create an inotify watcher manually, you'd still have a chance of receiving a pollwatcher without any warning. I see this as bad behavior.

Ultimately the QEMU setup is lying to us..

[samuelcolvin]

I hadn't quite appreciated the run-time vs. compile-time distinction at play here - that's my fault for thinking in python.

Do we want to add this error-code workaround into notify?

Well, I don't really have a dog in this fight anymore since it's fixed/worked-around in watchfiles.

My recommendation would be:

• document the behaviour - kind of done by this issue
• report it upstream to docker/QEMU if we can be bothered - I assume they already know and aren't that interested in fixing it, so I'm not sure of the point
• move on

But ultimately, up to the maintainers of this package.

Feel free to close this issue.

[0xpr03]

document the behaviour

Definitely.

I was thinking about a way to make the default-init able to catch such a case without inducing a ton of overhead. Currently the default watcher is selected at compile time.

[0xpr03]

I'm adding a note about this in #395

A correct fix is a little bit more tricky.

[samuelcolvin]

Looks good to me, thanks.

[samuelcolvin]

For anyone else who comes to this issue here is the logic in watchfiles which catches this error and falls back to the poll watcher.

In particular, we catch this specific error with

                match RecommendedWatcher::new(event_handler.clone()) {
                    Ok(watcher) => {
                        ...
                    }
                    Err(error) => {
                        match &error.kind {
                            ErrorKind::Io(io_error) => {
                                if io_error.raw_os_error() == Some(38) {
                                    ...