Would be nice if .pxd import could be done without requiring the depedency to be fully installed

[ghost]

Ok, I am now having the following problem:

Edit: go here for much better description: pypa/pip#6411

I have a .pxd dependency from an external lib. Because easy_install breaks Cython sometimes and since setup_requires is deprecated anyway, I have moved this .pxd-included dependency to pyproject.toml's build-system.requires. (since install_requires is too late, that doesn't guarantee it will be around during build_ext for .pxd inclusion but only later when it actually runs)

Now the problem appears to be that with a pyproject.toml & pip19 you get this new fancy build isolation, and the result of that appears to be that everything in build-system.requires is always locally reinstalled from scratch even if it's already in the system site packages (since the build is isolated). But this installation will always run through build_ext as well - that is entirely unnecessary for .pxd inclusion but pip/setuptools don't know that, they think it's a build requirement so surely it needs to be fully installed to be available including compilation.

This reinstall from scratch means that for bigger .pxd Cython dependencies, this time is always added up on the regular build time which for big Cython projects can be significant. (for mine multiple minutes!) And it's a waste because the package is not actually needed to be fully installed, only the uncompiled .pxd files would be needed.

It would be therefore nice if you could sit together with pip & setuptools folks to figure out something like sdist_requires on top of install_requires/build-system.requires that somehow makes it available for Cython .pxd include use WITHOUT installing it fully. Otherwise all Cython builds around the world using .pxd includes will waste quite some additional time moving to the new pyproject.toml/pip19 ways of doing things

[ghost]

A real world example where this makes a huge impact for the testing tool tox:

• tox prepares a virtualenv of all dependencies the first time it runs the tests
• then with each later run, it will usually only make an sdist of the project, and install the project directly without deps to rerun the newer version's tests

Historic result:

The dependencies, even if they take very long to build, don't get too much into the way because they're only built once and when really needed. (version bump/resetting the virtualenv due to some other issue) This also applies to setup_requires dependencies needed for .pxd imports

Degradation with pyproject.toml/.pxd imports:

With a pyproject.toml and a package from which I want to .pxd import specified as build-system.requires in that toml file (which is the new, only not-broken way to ensure external .pxds are around, right?) pip 19 will enable build isolation.

Result: deps, even if already installed by tox and in the virtualenv, are not around during build. However, the .pxd dependency is needed during build, and hence will be rebuilt from scratch every single time the main project is also rebuilt. (my dep actually has another indirect Cython dep, so both are rebuilt in addition!)

This is not the case for 99% of the other deps since almost all will just be install_requires, and even things like setuptools that are truly needed for build don't need long to install, unlike substantial Cython projects - the problem is purely limited to external .pxd import targets.

Overall result/problem: potentially many minutes of additional time on every single test run.

And that all because there's no good way of stating "Give me the .pxd files during install but don't actually build it please" - kind of the equivalent of needing the development headers only, and being forced to rebuild the entire external lib to get them

[ghost]

(Of course setup_requires has kinda the same issue, but without the new pyproject.toml build isolation being enabled it's not as bad because it will only impact the first run. Also if the dep is in install_requires as well it's not useless that it's built at some point, so one might not actually lose any time in overall.)

[scoder]

I guess the canonical answer to this is:

1. make sure your dependencies provide wheels
2. if your goal is to build only an sdist and you have non-wheel binary build dependencies, then set the CFLAGS to something fast, e.g. CFLAGS=-O0 python setup.py sdist.

[ghost]

make sure your dependencies provide wheels

That helps a little, but still the wheel building itself will be still unnecessarily slow. Also there is the problem that I can't actually do this right now I believe, since I mostly use it in python-for-android which is a cross-compile environment, I'd be surprised if wheels are set up to work properly. But maybe we need to fix that then, but that might take a while. (not sure pypi will even accept the unusual architecture entry at this point) But yeah, that does sound like it might be the best idea, I'll ponder this

if your goal is to build only an sdist and you have non-wheel binary build dependencies, then set the CFLAGS to something fast, e.g. CFLAGS=-O0 python setup.py sdist.

Nice! A bit of a hack though, and again not that easily done in a cross-compile environment where CFLAGS may already be set to other things and merging the value might be more complicated

[ghost]

Wait, can I even easily provide a wheel when I install from github/git master? This sounds like it might make quite complicated, I'm really not sure if that's a realistic option for a fast-moving in-development dependency. (which it currently is) Of course for a package on pypi it might work, but then again I'd also need to put android wheels onto pypi (and preferably all other arcane and non-arcane platforms) or we're back to the original problem. Thinking more about this it sounds like the better idea would be still to simply not require compilation when it's evidently not required at all, since that really is the source of all trouble

[scoder]

You can point pip to your own "PyPI" instance. Many people do that, e.g. using devpi or some other kind of caching proxy.

[ghost]

Yes but isn't that horribly complicated? I appreciate the constructive ideas though, I'm just wondering if it wouldn't be worth it to make some sort of change in the tooling to make this case easier