Include MNE-ICALabel (and PyTorch) in the Windows installers

[hoechenberger]

No description provided.

[hoechenberger]

If we merge this, please don't squash the history – I separated the commits so it'll be easier to compile the changelog.

[hoechenberger]

Dependency resolution seems to work; however, installer creation on Windows fails for unknown reasons :(

[hoechenberger]

@larsoner Have we seen such an issue before? I'm not sure

[larsoner]

No, I'm guessing it's unrelated to this PR (and probably due to conda installer updates) but it's worth checking

[larsoner]

Okay on mamba 1.1.0 / 87f96bf the failure is:

fetching: mne-nirs-0.5.0-pyhd8ed1ab_0.conda
fetching: pyprep-0.4.2-pyhd8ed1ab_0.tar.bz2
Checking for duplicate files ...
Warning: File 'Library/bin/libiomp5md.dll' found in multiple packages: intel-openmp-2023.0.0-h57928b3_25922.conda, openmp-5.0.0-vc14_1.tar.bz2
Warning: File 'Library/README.txt' found in multiple packages: libhwloc-2.8.0-h039e092_1.tar.bz2, gl2ps-1.4.2-h0597ee9_0.tar.bz2
Warning: File 'site-packages/tests/__init__.py' found in multiple packages: pycircstat-0.0.2-pyh8a188c0_0.tar.bz2, neurokit2-0.2.3-pyha21a80b_0.conda
Checking for 'C:\Users\runneradmin\micromamba-root\envs\constructor-env\NSIS\makensis.exe'
NSIS version: v3.01
Reading: C:\Users\runneradmin\micromamba-root\envs\constructor-env\lib\site-packages\constructor\nsis\main.nsi.tmpl
Created C:\Users\RUNNER~1\AppData\Local\Temp\tmpt3ds4579\main.nsi file
Calling: ['C:\\Users\\runneradmin\\micromamba-root\\envs\\constructor-env\\NSIS\\makensis.exe', '/V2', 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpt3ds4579\\main.nsi']
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\Users\\runneradmin\\micromamba-root\\envs\\constructor-env\\NSIS\\makensis.exe', '/V2', 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpt3ds4579\\main.nsi']' returned non-zero exit status 1.
Error: Process completed with exit code 1.

Going to try reverting our NSIS hotfix to see if that helps...

[hoechenberger]

@larsoner Failing :(

Checking for 'C:\Users\runneradmin\micromamba-root\envs\constructor-env\NSIS\makensis.exe'
[588](https://github.com/mne-tools/mne-installers/actions/runs/4167033792/jobs/7212114930#step:9:589)
NSIS version: v3.08
[589](https://github.com/mne-tools/mne-installers/actions/runs/4167033792/jobs/7212114930#step:9:590)
Reading: C:\Users\runneradmin\micromamba-root\envs\constructor-env\lib\site-packages\constructor\nsis\main.nsi.tmpl
[590](https://github.com/mne-tools/mne-installers/actions/runs/4167033792/jobs/7212114930#step:9:591)
Created C:\Users\RUNNER~1\AppData\Local\Temp\tmpwq_ql9_b\main.nsi file
[591](https://github.com/mne-tools/mne-installers/actions/runs/4167033792/jobs/7212114930#step:9:592)
Calling: ['C:\\Users\\runneradmin\\micromamba-root\\envs\\constructor-env\\NSIS\\makensis.exe', '/V2', 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpwq_ql9_b\\main.nsi']
[592](https://github.com/mne-tools/mne-installers/actions/runs/4167033792/jobs/7212114930#step:9:593)
Error: Process completed with exit code 1.

[larsoner]

@jaimergp our nsis=3.01 hotfix you kindly suggested worked for a while but stopped working on Windows in the last ~2 months:

https://github.com/mne-tools/mne-installers/actions/runs/4166630449/jobs/7211209939

...
fetching: pyprep-0.4.2-pyhd8ed1ab_0.tar.bz2
Checking for duplicate files ...
Warning: File 'Library/bin/libiomp5md.dll' found in multiple packages: intel-openmp-2023.0.0-h57928b3_25922.conda, openmp-5.0.0-vc14_1.tar.bz2
Warning: File 'Library/README.txt' found in multiple packages: libhwloc-2.8.0-h039e092_1.tar.bz2, gl2ps-1.4.2-h0597ee9_0.tar.bz2
Warning: File 'site-packages/tests/__init__.py' found in multiple packages: pycircstat-0.0.2-pyh8a188c0_0.tar.bz2, neurokit2-0.2.3-pyha21a80b_0.conda
Checking for 'C:\Users\runneradmin\micromamba-root\envs\constructor-env\NSIS\makensis.exe'
NSIS version: v3.01
Reading: C:\Users\runneradmin\micromamba-root\envs\constructor-env\lib\site-packages\constructor\nsis\main.nsi.tmpl
Created C:\Users\RUNNER~1\AppData\Local\Temp\tmpt3ds4579\main.nsi file
Calling: ['C:\\Users\\runneradmin\\micromamba-root\\envs\\constructor-env\\NSIS\\makensis.exe', '/V2', 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpt3ds4579\\main.nsi']
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\Users\\runneradmin\\micromamba-root\\envs\\constructor-env\\NSIS\\makensis.exe', '/V2', 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpt3ds4579\\main.nsi']' returned non-zero exit status 1.

And switching to napari/bundle_tools_2 and removing the pins does not help:

https://github.com/mne-tools/mne-installers/actions/runs/4169356013/jobs/7217229034

...
fetching: pyprep-0.4.2-pyhd8ed1ab_0.tar.bz2
Checking for duplicate files ...
Warning: File 'Library/bin/libiomp5md.dll' found in multiple packages: openmp-5.0.0-vc14_1.tar.bz2, intel-openmp-2023.0.0-h[579](https://github.com/mne-tools/mne-installers/actions/runs/4169356013/jobs/7217229034#step:8:580)28b3_25922.conda
Warning: File 'Library/README.txt' found in multiple packages: gl2ps-1.4.2-h0597ee9_0.tar.bz2, libhwloc-2.8.0-h039e092_1.tar.bz2
Warning: File 'site-packages/tests/__init__.py' found in multiple packages: neurokit2-0.2.3-pyha21a80b_0.conda, pycircstat-0.0.2-pyh8a188c0_0.tar.bz2
Checking for 'C:\Users\runneradmin\micromamba-root\envs\constructor-env\NSIS\makensis.exe'
NSIS version: v3.08
Reading: C:\Users\runneradmin\micromamba-root\envs\constructor-env\lib\site-packages\constructor\nsis\main.nsi.tmpl
Created C:\Users\RUNNER~1\AppData\Local\Temp\tmpqzoz9q5k\main.nsi file
Calling: ['C:\\Users\\runneradmin\\micromamba-root\\envs\\constructor-env\\NSIS\\makensis.exe', '/V2', 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpqzoz9q5k\\main.nsi']
Error: Process completed with exit code 1.

Any ideas here?

[jaimergp]

Maybe try running constructor -v for higher verbosity. I updated the builds on bundle_tools_2 yesterday and it is now up to date with 3.4.3.

[larsoner]

TL;DR: Thanks to your help @jaimergp, it looks like the problem is that something new we add here (PyTorch) makes the installers too big (>2GB I guess?) which is problematic.

Verbose mode was informative in e07c9d3 / CI:

File: "pytorch-1.12.1-py3.10_cuda11.6_cudnn8_0.tar.bz2"

Note: you may have one or two (large) stale temporary file(s) left in your temporary directory (Generally this only happens on Windows 9x).
makensis stderr:
Internal compiler error #12345: error mmapping file (1331871499, 33554432) is out of range.
Successfully created 'D:\a\mne-installers\mne-installers\MNE-Python-1.3.0_1-Windows.exe'.

Although it says it successfully created the file (?), it appears not to have done so as our simple filename glob fails:

Finding matches
Extracting fname
Found name: 
Want name:  MNE-Python-1.3.0_1-Windows.exe
Error: Process completed with exit code 1.

After a bit of googling it sounded like we might be hitting a 2GB limitation of NSIS. I suspected PyTorch adds a lot of code, so I removed mne_iclabel / pytorch (which was the point of this PR!) in c9836b0 / CI, and this build successfully created the .exe -- seemingly confirming the NSIS 2GB limitation issue :(

[hoechenberger]

OUCH

@larsoner Let's try without Spyder, WDYT?

[larsoner]

I don't think we should drop Spyder for PyTorch on Windows

[hoechenberger]

My idea was that maybe we could create two separate installers, one with more basic functionality and including Spyder, and one for more advanced users

[larsoner]

I think I'd rather just give the advanced users the instructions for how to get PyTorch installed on Windows

[hoechenberger]

There is a project that removes the 2 GB limitation, but I have no clue how we could make use if it in constructor

https://sourceforge.net/projects/nsisbi/

@jaimergp WDYT?

[jaimergp]

Great find! I'll open an issue on conda/constructor.