[WIP] install to envs by default

[minrk]

Continuing the discussion in #61

Main proposed change: default installation of support files is in the current env, rather than /usr/local (or equivalent)

• add paths.install_data_path
  • Central API for specifying the installation path for a given file for user=True|False.
  • Should ensure consistency for defaults, etc.
  • If accepted, this should be used exclusively to determine install paths for nbextensions/kernelspecs, etc.

This also asks disutils where data_files would go for both user=True|False, ensuring that Python packages can provide kernelspecs, extensions, etc. via data_files (in wheels).

There is a small number of common cases where default data_files and sys.prefix disagree (OS X, Debian), which is the reasoning behind asking distutils instead of hardcoding sys.prefix. When they disagree, sys.prefix is also added to paths, but we could consider not doing that. In envs (virtualenv, conda, custom Python install), these never disagree.

Unfortunately, neither Python nor distutils has a reasonable INSTALL_PREFIX API to ask for where files would go, so building a distutils object seems to be the only way to get the answer to this question.

cc @jdemeyer @takluyver @ellisonbg

[asmeurer]

Main proposed change: default installation of support files is in the current env, rather than /usr/local (or equivalent)

+:100:

[rgbkrk]

-:100:

I'm very much against Python specifics for kernel spec installation. There are more than one non-Python environments - primarily in Atom, Electron apps, as well as Go.

[takluyver]

I'm not clear what the advantage of this would be. AIUI, the reason to consider sys.prefix is that it's where Python packaging installs data_files. But we already put that location in the data files search path, so stuff installed by Python packaging should be found. And installing data_files with Python packaging doesn't involve our installation mechanisms, so that's not affected by this. So what exactly are we trying to solve?

[Carreau]

What should we do with this ? It seem to be stale, I'd like to have a clean PR queue.

@fperez might be needed to make a BDFL decision.

[jdemeyer]

I'm very much against Python specifics for kernel spec installation.

I don't think that the current proposal is so Python-specific. Sure, it might use some Python implementation to determine the directory, but that does not mean that the resulting directory is Python-specific. And in any case, it is important that the resulting directory is one where the notebook (which is Python-specific) will look for kernelspecs.

Moreover, I'm very much against inconsistency. Right now, we are in a situation where every kernel package essentially does its own thing. My package uses the data_files option of distutils but other packages do different things.

[jdemeyer]

So what exactly are we trying to solve?

As absolute minimum, there is a documentation problem. You have a very nice echokernel module explaining how to write a basic wrapper kernel, but that's only half of the story. As I mentioned in #61, this should be expanded to a complete package with setup.py and kernel spec. I spent quite some time to figure this out for my kernel.

But before we can write such an example package, we first need to agree on what the recommended way is to install kernel specs. If it's data_files in setup.py, that's fine for me. For non-Python kernels, we need some way to determine this directory.

[Carreau]

But before we can write such an example package, we first need to agree on what the recommended way is to install kernel specs. If it's data_files in setup.py, that's fine for me. For non-Python kernels, we need some way to determine this directory.

The problem is for project like http://nteract.io/ that can and do connect to Python kernel, but are not written in Python. data_files and installing in env by default breaks it because as they are written in nodeJS they can't know about the environment. That would also break things like Hydrogen.

[asmeurer]

I agree. The concept of an install prefix is in no way specific to Python. It's a POSIX thing that you'll see all over the place (e.g., autoconf and cmake both use the same terminology to refer to the same thing).

[takluyver]

I don't like installing kernelspecs into the env by default because I want to have kernels from different envs available in one notebook server, and that only works if the kernelspecs live somewhere else.

[rgbkrk]

When I see sys.prefix used from an Anaconda install, it's deep inside a conda environment and not discoverable by a system process.

[jdemeyer]

I want to have kernels from different envs available in one notebook server, and that only works if the kernelspecs live somewhere else.

I consider that advanced usage. There is no way a single default value will be the correct one for everybody. It's just a default: if you want to be special, you can still manually specify the installation prefix.

[Carreau]

I consider that advanced usage. There is no way a single default value will be the correct one for everybody. It's just a default: if you want to be special, you can still manually specify the installation prefix.

Seen that IPython will drop Python 2 support next year, followed by other Jupyter component following that mean that every Python 2 user will likely have to to enter this special command. and make the Python 2 vs Python 3 install instruction differens. Plus all extensions have to be installed on a per environment basis. So I don't think that's the right way.

Continuum will ship the notebook server with a kernel manager that is aware of environments so you won't have to register each environment. You will "just" have to pip-install ipykernel

[fperez]

Will look next into this, but for now, it's worth noting the failing Travis...

[Carreau]

Will look next into this, but for now, it's worth noting the failing Travis...

That's failing only on Python 3.6a nightly. Yes we are testing on 3.6a nightly and did complain upstream that they are breaking things.

[fperez]

Ah, ok!

[Carreau]

That's failing only on Python 3.6a nightly. Yes we are testing on 3.6a nightly and did complain upstream that they are breaking things.

My bad responding through email because Apple Mail mixed up the thread, This one is actually failing.

[fperez]

Ah, ok ok! ;)

[minrk]

Personally, I'm opposed to this change now, as well.

[jdemeyer]

:-(

[jdemeyer]

Denying the problem or not agreeing on how to fix a problem does not magically fix the problem.