Plugin Manager shows conda as option for installation source in non-conda environment

[aganders3]

🐛 Bug

As noted in napari/napari#5622 but deemed out of scope, indeed when running napari from a venv-based virtualenv (and all CONDA env vars removed) the Plugin Manager still shows conda as an option for plugin source in the Installation Info dropdown:

If I try to install a plugin using conda anyway, I get ValueError: Prefix has not been specified!.

Full stack trace

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/src/napari/napari/_qt/dialogs/qt_plugin_dialog.py:593, in QPluginList.handle_action(self=<napari._qt.dialogs.qt_plugin_dialog.QPluginList object>, item=<PyQt6.QtWidgets.QListWidgetItem object>, pkg_name='affinder==0.2.3', action_name='install', version='0.2.3', installer_choice='PyPI')
    588     pkg_name += (
    589         f"=={item.widget.version_choice_dropdown.currentText()}"
    590     )
    591 widget.set_busy(trans._("installing..."), action_name)
--> 593 job_id = self.installer.install(
        self = <napari._qt.dialogs.qt_plugin_dialog.QPluginList object at 0x2c970e830>
        self.installer = <napari._qt.dialogs.qt_package_installer.InstallerQueue object at 0x297c44550>
        tool = <InstallerTools.CONDA: 'conda'>
        pkg_name = 'affinder==0.2.3'
        [pkg_name] = ['affinder==0.2.3']
    594     tool=tool,
    595     pkgs=[pkg_name],
    596     # origins="TODO",
    597 )
    598 if self._warn_dialog:
    599     self._warn_dialog.exec_()

File ~/src/napari/napari/_qt/dialogs/qt_package_installer.py:311, in InstallerQueue.install(self=<napari._qt.dialogs.qt_package_installer.InstallerQueue object>, tool=<InstallerTools.CONDA: 'conda'>, pkgs=['affinder==0.2.3'], prefix=None, origins=(), **kwargs={})
    284 """Install packages in `pkgs` into `prefix` using `tool` with additional
    285 `origins` as source for `pkgs`.
    286 
   (...)
    301     ID that can be used to cancel the process.
    302 """
    303 item = self._build_queue_item(
    304     tool=tool,
    305     action=InstallerActions.INSTALL,
   (...)
    309     **kwargs,
    310 )
--> 311 return self._queue_item(item)
        item = CondaInstallerTool(action=<InstallerActions.INSTALL: 'install'>, pkgs=['affinder==0.2.3'], origins=(), prefix=None)
        self = <napari._qt.dialogs.qt_package_installer.InstallerQueue object at 0x297c44550>

File ~/src/napari/napari/_qt/dialogs/qt_package_installer.py:466, in InstallerQueue._queue_item(self=<napari._qt.dialogs.qt_package_installer.InstallerQueue object>, item=CondaInstallerTool(action=<InstallerActions.INST...kgs=['affinder==0.2.3'], origins=(), prefix=None))
    464 def _queue_item(self, item: AbstractInstallerTool) -> JobId:
    465     self._queue.append(item)
--> 466     self._process_queue()
        self = <napari._qt.dialogs.qt_package_installer.InstallerQueue object at 0x297c44550>
    467     return item.ident

File ~/src/napari/napari/_qt/dialogs/qt_package_installer.py:476, in InstallerQueue._process_queue(self=<napari._qt.dialogs.qt_package_installer.InstallerQueue object>)
    474 self.setProgram(str(tool.executable()))
    475 self.setProcessEnvironment(tool.environment())
--> 476 self.setArguments([str(arg) for arg in tool.arguments()])
        tool = CondaInstallerTool(action=<InstallerActions.INSTALL: 'install'>, pkgs=['affinder==0.2.3'], origins=(), prefix=None)
        self = <napari._qt.dialogs.qt_package_installer.InstallerQueue object at 0x297c44550>
    477 # this might throw a warning because the same process
    478 # was already running but it's ok
    479 self._log(
    480     trans._(
    481         "Starting '{program}' with args {args}",
   (...)
    484     )
    485 )

File ~/src/napari/napari/_qt/dialogs/qt_package_installer.py:189, in CondaInstallerTool.arguments(self=CondaInstallerTool(action=<InstallerActions.INST...kgs=['affinder==0.2.3'], origins=(), prefix=None))
    188 def arguments(self) -> Tuple[str, ...]:
--> 189     prefix = self.prefix or self._default_prefix()
        self.prefix = None
        self = CondaInstallerTool(action=<InstallerActions.INSTALL: 'install'>, pkgs=['affinder==0.2.3'], origins=(), prefix=None)
    190     if self.action == InstallerActions.UPGRADE:
    191         args = ['update', '-y', '--prefix', prefix]

File ~/src/napari/napari/_qt/dialogs/qt_package_installer.py:252, in CondaInstallerTool._default_prefix(self=CondaInstallerTool(action=<InstallerActions.INST...kgs=['affinder==0.2.3'], origins=(), prefix=None))
    250 if (Path(sys.prefix) / "conda-meta").is_dir():
    251     return sys.prefix
--> 252 raise ValueError("Prefix has not been specified!")

ValueError: Prefix has not been specified!

To Reproduce

Steps to reproduce the behavior:

1. Create and activate a non-conda virtual environment (e.g. python -m venv venv && source venv/bin/activate)
2. Install napari in the venv (e.g. pip install 'napari[pyqt6]', note I had to use qt6 because I'm on an ARM Macbook)
3. Launch napari, open the Plugin Manager from the "Plugins" menu, click the "Installation Info" dropdown under the affinder plugin, check the "Source" dropdown to see it offers both PyPI and conda.

Expected behavior

I expect the Plugin Manager to not offer and not attempt to install using conda unless in a conda-based environment.

Environment

napari: 0.4.17rc4.dev210+gc43a2e7c0.d20230302
Platform: macOS-13.2.1-arm64-arm-64bit
System: MacOS 13.2.1
Python: 3.10.9 (main, Feb 3 2023, 15:40:08) [Clang 14.0.0 (clang-1400.0.29.202)]
Qt: 6.4.2
PyQt6: 
NumPy: 1.24.1
SciPy: 1.10.0
Dask: 2023.1.1
VisPy: 0.12.2.dev38+dirty
magicgui: 0.6.1
superqt: unknown
in-n-out: 0.1.6
app-model: 0.1.1
npe2: 0.6.2

OpenGL:
- GL version: 2.1 Metal - 83
- MAX_TEXTURE_SIZE: 16384

Screens:
- screen 1: resolution 2560x1440, scale 1.0

Settings path:
- /Users/aandersoniii/Library/Application Support/napari/venv_640f7def1935afdf07a142187e645430c6d70fe6/settings.yaml

Additional context

Note I don't have an entirely conda-free computer to test on, but I would still expect the behavior to be different in the setup I have here.

Summary of prior discussion - NB @Czaki @psobolewskiPhD:

And that's where the widget is made. As far as I can tell there is no checks for when the widget shows conda. Unless I've missed something, I'd expect the widget to show conda regardless of how the env was created.
If conda isn't actually available then I think the only checks are here:

So it looks like there is a bug in the logic. Maybe out of the scope of this PR but at least should have issue about this.

I think for sure out of scope, this is a small bug fix for a rather niche case.
If indeed conda is offered when not available or offered when non-condo env is used, that's a bigger deal I think. But best someone without conda test this first to make sure?

Originally posted by @psobolewskiPhD in napari/napari#5622 (comment)

[psobolewskiPhD]

CC: @ppwadhwa @goanpeca
Can y'all look into this and maybe give the backend aspects of Plugin Manager a thorough shakedown for any other bugs related to the conda implementation?
We'd like to get a release going sooner rather than later and this looks like it needs a bit more time in the oven.
Edit: also, the docs page finding and installing plugins (https://napari.org/stable/plugins/find_and_install_plugin.html#finding-and-installing-a-napari-plugin) is now outdated. I've made a docs issue napari/docs#122

BTW, related other thing I ran into in the original PR, which isn't strictly a bug, but is a bit of an issue is that if napari was pip installed and the user installs plugin via conda, you can get the whole env rebuilt, which can mean multiple Qt involved, etc.
napari/napari#5198 (comment)
(Note the most common install instructions say make a conda env and then pip install, so this is probably a fairly common scenario)
Does it make sense to consider checking the source of napari? or maybe giving a heads up warning?

[goanpeca]

Hi @psobolewskiPhD and @aganders3 thanks for raising this issue. I am working on a fix for the two things now :)

Cheers!

[goanpeca]

Hi @psobolewskiPhD

Regarding:

(Note the most common install instructions say make a conda env and then pip install, so this is probably a fairly common scenario) Does it make sense to consider checking the source of napari? or maybe giving a heads up warning?

There is still the case of using napari in development mode... that is installing from the latest source.

So that would need some extra logic to be handled correctly. That is installing everything from conda in a conda environment but using napari in development mode

[psobolewskiPhD]

@goanpeca I'm not sure I follow.
I'm talking about a user who does as instructed here:
https://napari.org/stable/#from-pip-with-batteries-included
They will have a conda env but everything will be from pip.
If they then install a plugin from conda (as some folks in different places may suggest), then my understanding is that conda will ignore their existing packages that were pip installed and install everything from scratch.
I can mock this behavior by installing making an env and installing napari with pip as described above, then e.g. conda install napari-animation
This installs everything from conda (276 packages), results in duplicate pyqt:

╰─ pip show PyQt5                                                  (nap-test) ─╯
Name: PyQt5
Version: 5.15.9
Summary: Python bindings for the Qt cross platform application toolkit
Home-page: https://www.riverbankcomputing.com/software/pyqt/
Author: Riverbank Computing Limited
Author-email: info@riverbankcomputing.com
License: GPL v3
Location: /Users/piotrsobolewski/Dev/miniforge3/envs/nap-test/lib/python3.9/site-packages
Requires: PyQt5-Qt5, PyQt5-sip
Required-by: 
╭─    ~ ······························································ ✔ ─
╰─ mamba list | grep pyqt                                          (nap-test) ─╯
napari                    0.4.17          pyh275ddea_0_pyqt    conda-forge
pyqt                      5.15.7           py39h327fbb6_3    conda-forge
pyqt5                     5.15.9                   pypi_0    pypi
pyqt5-qt5                 5.15.2                   pypi_0    pypi
pyqt5-sip                 12.11.1                  pypi_0    pypi

and now napari doesn't start:

objc[44078]: Class QMacAutoReleasePoolTracker is implemented in both /Users/piotrsobolewski/Dev/miniforge3/envs/nap-test/lib/libQt5Core.5.15.8.dylib (0x16591b2d8) and /Users/piotrsobolewski/Dev/miniforge3/envs/nap-test/lib/python3.9/site-packages/PyQt5/Qt5/lib/QtCore.framework/Versions/5/QtCore (0x17e9ca198). One of the two will be used. Which one is undefined.
objc[44078]: Class QT_ROOT_LEVEL_POOL__THESE_OBJECTS_WILL_BE_RELEASED_WHEN_QAPP_GOES_OUT_OF_SCOPE is implemented in both /Users/piotrsobolewski/Dev/miniforge3/envs/nap-test/lib/libQt5Core.5.15.8.dylib (0x16591b350) and /Users/piotrsobolewski/Dev/miniforge3/envs/nap-test/lib/python3.9/site-packages/PyQt5/Qt5/lib/QtCore.framework/Versions/5/QtCore (0x17e9ca210). One of the two will be used. Which one is undefined.
objc[44078]: Class KeyValueObserver is implemented in both /Users/piotrsobolewski/Dev/miniforge3/envs/nap-test/lib/libQt5Core.5.15.8.dylib (0x16591b378) and /Users/piotrsobolewski/Dev/miniforge3/envs/nap-test/lib/python3.9/site-packages/PyQt5/Qt5/lib/QtCore.framework/Versions/5/QtCore (0x17e9ca238). One of the two will be used. Which one is undefined.
objc[44078]: Class RunLoopModeTracker is implemented in both /Users/piotrsobolewski/Dev/miniforge3/envs/nap-test/lib/libQt5Core.5.15.8.dylib (0x16591b3c8) and /Users/piotrsobolewski/Dev/miniforge3/envs/nap-test/lib/python3.9/site-packages/PyQt5/Qt5/lib/QtCore.framework/Versions/5/QtCore (0x17e9ca288). One of the two will be used. Which one is undefined.
WARNING: QObject::moveToThread: Current thread (0x600003391110) is not the object's thread (0x600003380fe0).
Cannot move to target thread (0x600003391110)

18:48:38 WARNING QObject::moveToThread: Current thread (0x600003391110) is not the object's thread (0x600003380fe0).
Cannot move to target thread (0x600003391110)

WARNING: You might be loading two sets of Qt binaries into the same process. Check that all plugins are compiled against the right Qt binaries. Export DYLD_PRINT_LIBRARIES=1 and check that only one set of binaries are being loaded.
18:48:38 WARNING You might be loading two sets of Qt binaries into the same process. Check that all plugins are compiled against the right Qt binaries. Export DYLD_PRINT_LIBRARIES=1 and check that only one set of binaries are being loaded.
WARNING: Could not load the Qt platform plugin "cocoa" in "" even though it was found.
18:48:38 WARNING Could not load the Qt platform plugin "cocoa" in "" even though it was found.
WARNING: This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: cocoa, minimal, offscreen, webgl.

18:48:38 WARNING This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: cocoa, minimal, offscreen, webgl.

[goanpeca]

@psobolewskiPhD, sorry I was not clear, let me elaborate :)

So the case you expose, yes, someone does

conda create -n napari python -c conda-forge
pip install napari

I have taken care of this case :)

---

I am now referring to the use case where someone is using the main branch napari (installing napari from source)

conda create -n napari python -c conda-forge
git clone https://github.com/napari/napari.git
conda install ...(all napari dependencies) 
pip install napari -e . --no-deps

So, here we installed everything in a conda environment and all dependencies are conda packages, except napari which is installed in dev mode (using pip), so this case is special, and you should be able to install things with conda from the plugin manager, but you may also run into the issue you found.

[psobolewskiPhD]

@goanpeca OK, yeah, I get it now, thanks for being patient with me.
Yeah, that is an interesting case. Talley's tool pamba can facilitate that actually.
I think it's fairly deliberate and likely only by more sophisticated users, who might not use the GUI to install? Anyhow, I think I wouldn't block conda installs in these napari from pypi cases, just warn the user?
(Note: I see the PR now, haven't looked yet, so this comment may be outdated/updated.)

[psobolewskiPhD]

@goanpeca Should we migrate this issue to the plugin-manager repo?

[goanpeca]

Migrated @psobolewskiPhD :)