jupyter fails with cannot execute binary file: Exec format error

[sdbbs]

Using msys2/mingw64 updated today, on a Windows 10 (64-bit) machine. I want to install Jupyter - so first, I tried to look at https://jupyter.org/install which says "Installing Jupyter with pip"; turns out Jupyter install via pip fails in MSYS2 shell:

user@DESKTOP-PC MSYS ~
$ python3 -m pip install jupyter
Collecting jupyter
  Downloading https://files.pythonhosted.org/packages/83/df/0f5dd132200728a86190397e1ea87cd76244e42d39ec5e88efd25b2abd7e/jupyter-1.0.0-py2.py3-none-any.whl
...
  Running setup.py install for pyzmq ... error
    ERROR: Complete output from command /usr/bin/python3.exe -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-akpo_2c9/pyzmq/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-93nj111m/install-record.txt --single-version-externally-managed --compile:
...
    build/temp.msys-3.0.7-x86_64-3.7/scratch/vers.c:4:10: fatal error: zmq.h: No such file or directory
        4 | #include "zmq.h"
          |          ^~~~~~~
    compilation terminated.

    error: command 'x86_64-pc-msys-gcc' failed with exit status 1
...
    In file included from bundled/zeromq/src/address.cpp:38:
    bundled/zeromq/src/tipc_address.hpp:43:10: fatal error: linux/tipc.h: No such file or directory
       43 | #include <linux/tipc.h>
          |          ^~~~~~~~~~~~~~
    compilation terminated.
    error: command 'x86_64-pc-msys-gcc' failed with exit status 1
...

... and it turns out, I cannot use pip in MINGW64 ( python - Pip on MINGW64? - Stack Overflow )

So, then I opened a MINGW64 shell, and tried to install the binary packages:

user@DESKTOP-PC MINGW64 ~
$ pacman -S mingw-w64-x86_64-python3-notebook
resolving dependencies...
looking for conflicting packages...

Packages (49) mingw-w64-x86_64-libsodium-1.0.18-1
              mingw-w64-x86_64-python3-attrs-19.1.0-1
...
(49/49) installing mingw-w64-x86_64-python3-no...  [#####################] 100%

user@DESKTOP-PC MINGW64 ~
$ pacman -S mingw-w64-x86_64-python3-jupyter_console
resolving dependencies...
looking for conflicting packages...

Packages (1) mingw-w64-x86_64-python3-jupyter_console-6.0.0-1
...
(1/1) installing mingw-w64-x86_64-python3-jupy...  [#####################] 100%

Fine - now I should test the start with jupyter notebook but:

user@DESKTOP-PC MINGW64 ~/jupyter_test
$ jupyter notebook
bash: /mingw64/bin/jupyter: cannot execute binary file: Exec format error

user@DESKTOP-PC MINGW64 ~/jupyter_test
$ file /mingw64/bin/jupyter
/mingw64/bin/jupyter: MS-DOS executable

user@DESKTOP-PC MINGW64 ~/jupyter_test
$ cygpath -w `which jupyter`
C:\msys64\mingw64\bin\jupyter

Just in case, I tried this in MINGW32 too:

user@DESKTOP-PC MINGW32 ~
$ jupyter notebook
failed to create process (C:\building\msys64\mingw64\bin\python3.exe "C:\msys64\mingw64\bin\jupyter-notebook-script.py").

The same error for MINGW32 also appears for MSYS2 shell too, which I guess is expected.

But I certainly did not expect cannot execute binary file: Exec format error when running jupyter notebook in MINGW64.

So, how can I run jupyter on MINGW64?

---

EDIT: just tried to run jupyter.exe in cmd.exe:

C:\msys64\mingw64\bin>jupyter.exe

... and I got a popup window with:

---------------------------
Unsupported 16-Bit Application
---------------------------
The program or feature "\??\C:\msys64\mingw64\bin\jupyter.exe" cannot start or run due to incompatibity with 64-bit versions of Windows. Please contact the software vendor to ask if a 64-bit Windows compatible version is available.
---------------------------
OK   
---------------------------

... it is recognized as 16-bit application?!

[sdbbs]

Ok, I managed to get it to run, it's a bit weird; this is still from cmd.exe:

C:\msys64\mingw64\bin>jupyter.exe
This version of C:\msys64\mingw64\bin\jupyter.exe is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher.

C:\msys64\mingw64\bin>jupyter-notebook.exe
failed to create process (C:\building\msys64\mingw64\bin\python3.exe "C:\msys64\mingw64\bin\jupyter-notebook-script.py").

Then, I kind of started suspecting the path C:\building\msys64\mingw64\bin\python3.exe:

C:\msys64\mingw64\bin>C:\building\msys64\mingw64\bin\python3.exe
The system cannot find the path specified.

C:\msys64\mingw64\bin>C:\msys64\mingw64\bin\python3.exe
Python 3.7.3 (default, Jul  6 2019, 21:32:41)  [GCC 9.1.0 64 bit (AMD64)] on win32
...
>>> quit
Use quit() or Ctrl-Z plus Return to exit
>>> ^Z

Ok, so it seems the path to python which has \building in it is wrong; let's try to run that "failed to create process" command with the right python path:

C:\msys64\mingw64\bin>C:\msys64\mingw64\bin\python3.exe "C:\msys64\mingw64\bin\jupyter-notebook-script.py"
[I 10:48:54.318 NotebookApp] Writing notebook server cookie secret to C:\Users\sd\AppData\Roaming\jupyter\runtime\notebook_cookie_secret
[W 10:48:55.888 NotebookApp] Terminals not available (error was No module named 'winpty')
[I 10:48:55.888 NotebookApp] Serving notebooks from local directory: C:\msys64\mingw64\bin
[I 10:48:55.888 NotebookApp] The Jupyter Notebook is running at:
[I 10:48:55.888 NotebookApp] http://localhost:8888/?token=88cdb775be0c3a2f182d9a0c0436da49318e9e59823d4f07
[I 10:48:55.888 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 10:48:55.948 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://localhost:8888/?token=88cdb775be0c3a2f182d9a0c0436da49318e9e59823d4f07
[I 10:48:56.458 NotebookApp] Accepting one-time-token-authenticated connection from ::1
[I 10:49:05.948 NotebookApp] Interrupted...
[I 10:49:05.948 NotebookApp] Shutting down 0 kernels

... and here I get a browser raised too - so it is all good... or maybe it isn't, because maybe the jupyter executable raises other processes too?! In any case, this \building ends up in the shebang line for all these python scripts:

user@DESKTOP-PC MINGW64 /mingw64/bin
$ grep -r ....building...... jupyter*
jupyter-bundlerextension-script.py:#!C:/building/msys64/mingw64/bin/python3.exe
jupyter-nbextension-script.py:#!C:/building/msys64/mingw64/bin/python3.exe
jupyter-notebook-script.py:#!C:/building/msys64/mingw64/bin/python3.exe
jupyter-serverextension-script.py:#!C:/building/msys64/mingw64/bin/python3.exe

Not sure how to handle this ... Though, in #1540 there is:

Strangely, I downloaded a PKGBUILD from
https://github.com/Alexpux/MINGW-packages
and built mingw-w64-x86_64-python3-notebook myself. It is able to start fine.

.... which I'd try, but I cannot find a directory containing the string "notebook" anywhere on that page ...

---

EDIT: Turns out, if I call jupyter-notebook.exe from MINGW64 shell, it also starts:

user@DESKTOP-PC MINGW64 ~
$ jupyter-notebook
[I 11:05:02.392 NotebookApp] The port 8888 is already in use, trying another port.
[W 11:05:02.397 NotebookApp] Terminals not available (error was No module named 'winpty')
[I 11:05:02.397 NotebookApp] Serving notebooks from local directory: D:/msys64/home/sd
[I 11:05:02.397 NotebookApp] The Jupyter Notebook is running at:
[I 11:05:02.397 NotebookApp] http://localhost:8889/?token=e6cc65930c9ff8f805387c82ef7f439ce1a88aa120c9c55e
[I 11:05:02.397 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 11:05:02.442 NotebookApp]
...

... but it doesn't really work, possibly because of this bug? Kernel not connected Python 3.7 · Issue #4444 · jupyter/notebook - here it is linked to mingw-w64-x86_64-python3-tornado 6.0.3-1, but apparently, to get it work, one needs a version 5 of tornado ....

[peterbud]

@sdbbs : with the latest update I see the same, before that it was working. I'll look into this.

Also you need to start jupyter notebook only from MINGW64 shell, so not too much sense to start from cmd prompt.

[Alexpux]

Need update nodejs and then update Jupiter notebook, in other case we can’t get it working

[sdbbs]

Hi guys,

Thanks a lot for the responses!

My OP was from 08 July 2019; at that time, /var/log/pacman.log says I've had:

[2019-07-08 10:04] [ALPM] installed mingw-w64-x86_64-python3-jupyter_client (5.2.4-1)

Some days ago, I noticed something was updated for jupyter - checking again through /var/log/pacman.log I can see only this:

[2019-07-23 09:03] [ALPM] upgraded mingw-w64-x86_64-python3-jupyter_client (5.2.4-1 -> 5.3.1-1)

So, I decided to try it again; again, first with MINGW64, fails:

user@PC MINGW64 ~
$ jupyter notebook
bash: /mingw64/bin/jupyter: cannot execute binary file: Exec format error

user@PC MINGW64 ~
$ which jupyter
/mingw64/bin/jupyter

user@PC MINGW64 ~
$ file /mingw64/bin/jupyter
/mingw64/bin/jupyter: MS-DOS executable

So, I thought - hey, if it complains about "MS-DOS", and it doesn't run in MSYS2 - maybe it will run in MINGW32? And yeah, it does:

user@PC MINGW32 /c/bin
$ jupyter notebook
[W 08:38:30.200 NotebookApp] Terminals not available (error was No module named 'winpty')
[I 08:38:30.215 NotebookApp] Serving notebooks from local directory: C:/bin
[I 08:38:30.215 NotebookApp] The Jupyter Notebook is running at:
[I 08:38:30.215 NotebookApp] http://localhost:8888/?token=9d64d0212e47a1658c99e4b612dc4d2d312eddabae03593c
[I 08:38:30.215 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 08:38:30.246 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://localhost:8888/?token=9d64d0212e47a1658c99e4b612dc4d2d312eddabae03593c
[I 08:38:30.684 NotebookApp] Accepting one-time-token-authenticated connection from ::1

... although, when you open a new Python 3 notebook, it starts a kernel, but then it is just stuck on "Connecting to kernel".

Not sure if this latest jupyter_client had anything to do with this (nor am I sure on, how and why would a mingw-w64-x86_64* package have an influence on something not running in MINGW64, but running in MINGW32 ?!) But that is what I can see...

Btw, i got worried about Terminals not available (error was No module named 'winpty'), and so I tried this:

user@PC MINGW32 ~
$ pacman -S winpty
warning: winpty-0.4.3-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Packages (1) winpty-0.4.3-1

Total Installed Size:  3.17 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] y
...

That, unfortunately, did not solve the issue with No module named 'winpty' - so went to the Python side of things:

user@PC MINGW32 ~
$ pip3 install --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/62/ca/94d32a6516ed197a491d17d46595ce58a83cbb2fca280414e57cd86b84dc/pip-19.2.1-py2.py3-none-any.whl (1.4MB)
     |████████████████████████████████| 1.4MB 2.3MB/s
Installing collected packages: pip
  Found existing installation: pip 19.1.1
    Uninstalling pip-19.1.1:
      Successfully uninstalled pip-19.1.1
Successfully installed pip-19.2.1

user@PC MINGW32 ~
$ pip3 install winpty
Collecting winpty
  ERROR: Could not find a version that satisfies the requirement winpty (from versions: none)
ERROR: No matching distribution found for winpty

user@PC MINGW32 ~
$ pip3 search winpty
pywinpty (0.5.5)             - Python bindings for the winpty library
yawinpty (0.4.3)             - yet another winpty binding for python
yawinpty-extra (0.1.0.dev1)  - yet another winpty binding for python (extra
                               package)

user@PC MINGW32 ~
$ pip3 install pywinpty
Collecting pywinpty
  Downloading https://files.pythonhosted.org/packages/21/ae/acbedcee475d049647e45ee949e23ca492764fcd8027a073fcf07646d47c/pywinpty-0.5.5.tar.gz (48kB)
     |████████████████████████████████| 51kB 1.2MB/s
Installing collected packages: pywinpty
  Running setup.py install for pywinpty ... done
Successfully installed pywinpty-0.5.5

This did not solve the No module named 'winpty' either - and the reason is this:

user@PC MINGW32 ~
$ python3
Python 3.7.4 (default, Jul 11 2019, 09:35:14)
[GCC 9.1.0] on msys
Type "help", "copyright", "credits" or "license" for more information.
>>> import winpty
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.7/site-packages/winpty/__init__.py", line 11, in <module>
    from .ptyprocess import PtyProcess
  File "/usr/lib/python3.7/site-packages/winpty/ptyprocess.py", line 21, in <module>
    from .winpty_wrapper import PTY, PY2
  File "/usr/lib/python3.7/site-packages/winpty/winpty_wrapper.py", line 7, in <module>
    from ctypes import windll
ImportError: cannot import name 'windll' from 'ctypes' (/usr/lib/python3.7/ctypes/__init__.py)
>>>

So, I guess I "just" need to find and install ctypes.windll for this Python, and maybe Jupyter will start working?! I'll try to do some more testing when I have time, and I'll report back if I manage to get forward ...

EDIT: Just wanted to say I found this, from 2013: Windows build (ghc-devs):

msys2-compiled python is cygwin-like and it's 'ctypes' has no 'windll'.

... although, I found also this from 2013, which implies otherwise: Building a DLL with MinGW and loading it with Python's ctypes module - Stack Overflow:

I can ctypes.windll.loadlibrary(...) an ordinary windows DLL found in the windows/system32 without an issue.

[sbrisard]

Hello,
I have exactly the same problems with a fresh install of MSYS2.

If I try to run jupyter notebook (no dash), I get the following error message

sbrisard@EliteBook MINGW64 ~
$ jupyter notebook
bash: /mingw64/bin/jupyter: cannot execute binary file: Exec format error

Now, running jupyter-notebook (with the dash)

sbrisard@EliteBook MINGW64 ~
$ jupyter-notebook
failed to create process (C:\building\msys64\mingw64\bin\python3.exe "C:\msys64\mingw64\bin\jupyter-notebook-script.py").

Regarding the second error message, this "building" in the path is probably reminiscent of the machine on which the binaries where compiled?

Regarding the first error message, it is strange that jupyter.exe is recognized as a MS-DOS executable, rather than a PE32+ executable. For what it's worth, there are quite a few jupyter related executables that are MS-DOS executables:

sbrisard@EliteBook MINGW64 ~
$ file /mingw64/bin/jupyter*.exe
/mingw64/bin/jupyter.exe:                  MS-DOS executable
/mingw64/bin/jupyter-bundlerextension.exe: PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows
/mingw64/bin/jupyter-kernel.exe:           MS-DOS executable
/mingw64/bin/jupyter-kernelspec.exe:       MS-DOS executable
/mingw64/bin/jupyter-migrate.exe:          MS-DOS executable
/mingw64/bin/jupyter-nbconvert.exe:        PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows
/mingw64/bin/jupyter-nbextension.exe:      PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows
/mingw64/bin/jupyter-notebook.exe:         PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows
/mingw64/bin/jupyter-qtconsole.exe:        MS-DOS executable
/mingw64/bin/jupyter-run.exe:              MS-DOS executable
/mingw64/bin/jupyter-serverextension.exe:  PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows
/mingw64/bin/jupyter-troubleshoot.exe:     MS-DOS executable

In fact, they are the only executables that are listed as MS-DOS in the whole mingw64/bin directory:

sbrisard@EliteBook MINGW64 ~
$ file /mingw64/bin/*.exe|grep MS-DOS
/mingw64/bin/jupyter.exe:                            MS-DOS executable
/mingw64/bin/jupyter-kernel.exe:                     MS-DOS executable
/mingw64/bin/jupyter-kernelspec.exe:                 MS-DOS executable
/mingw64/bin/jupyter-migrate.exe:                    MS-DOS executable
/mingw64/bin/jupyter-qtconsole.exe:                  MS-DOS executable
/mingw64/bin/jupyter-run.exe:                        MS-DOS executable
/mingw64/bin/jupyter-troubleshoot.exe:               MS-DOS executable

So the recipe that builds the jupyter packages must be broken.

[Alexpux]

@sbrisard building jupiter is broken as updating nodejs too. There are no currently maintainers for this packages and I dont have time and wish to support them myself. So if you want to see issue solved you can try to do this job, in other case wait if someone want to do it.

[sbrisard]

@Alexpux Fair enough! Thank you for this quick answer. I am not competent enough to take on maintenance of this package. Will try and find another solution.
Thank you anyway for this great suite of tools, that makes working under windows so much fun!