problem for libraries that depend on psycopg2 going forward

[cjw296]

Just to get this out the way, I think issuing a UserWarning when there's nothing a user can do to make that warning stop, other than writing code to ignore the warning which will then become a wart to be forgotten about and stumbled over alter is a mistake, I've read #701 and while @matsaman's tone wasn't ideal, I can appreciate where they're coming from.

So, as a library author, I have libraries such as https://github.com/Mortar/mortar_mixins that depend on psycopg2. It's unclear to me what I put in my dependencies going forward; it seems that regardless of whether I put pyscopg2 or psycopg2-binary, I'm forcing a choice on my end users that isn't my place to make.

As a concrete example: if I put psycopg2, I currently get that UserWarning, which most people will rightly understand as "I'm doing something wrong". If I put psycopg2-binary, then I break people using conda, because I'm forcing them to no-longer user the conda-provided packages that are optimised for the conda environment.

I see there's plenty of discussion of this over on #674, but I wanted to to ask this concrete question, because as a library author, it seems like there's nothing "correct" I can do here :-/

[dvarrazzo]

Using psycopg2 --no-binary psycogp2 as explained in the docs.

I should be able to release 2.8 in a few days which shouldn't have the problem anymore.

[cjw296]

How do I specify --no-binary in a requirements.txt?
How do I tell conda to use --no-binary?

[dvarrazzo]

@cjw296 it's written in the manual.

I don't know conda, nobody ever asked me to support it.

[sjamaan]

@dvarrazzo Sorry to comment on this ticket again (I'm sure you're tired of this mess), but I don't understand exactly what libraries should do, dependency-wise.

I just received an issue in a library package of mine (a custom field for Django that uses the Postgres INTERVAL column type). An application that uses my package would like to specify psycopg2-binary in their dependencies for easy installation for the end user. However, they also use my library, which has psycopg2>=2.7.0 as a dependency in setup.py. That dependency line forces installation from source, which kind of destroys the intended ease of use.

I'm a little confused as to what I, as a library author, should do here. I could remove the dependency completely, but that would be strictly wrong, I think, since psycopg2 is really a dependency of my package. Simply changing the dependency to psycopg2-binary seems wrong too, given that this may potentially cause segfaults for some of my users if I understand correctly why the split was made.

Perhaps the proper solution could be added to the FAQ section of the docs as well, as this seems to affect multiple users, judging by #674.

[dvarrazzo]

@dvarrazzo Sorry to comment on this ticket again (I'm sure you're tired of this mess), but I don't understand exactly what libraries should do, dependency-wise.

I think your library should depend on psycopg2 and if the OP of CodeYellowBV/django-relativedelta#6 is removing the psycopg2-binary dependency, as they say in their comment, that's the way to go.

[altaurog]

I realize the decision to split off the psycopg2-binary package was made already over a year ago, and I understand the motivation. I also realize it may be too late to enter the conversation, but I hadn’t fully appreciated the ramifications until today.

While splitting off the psycopg2-binary package provides a direct user with the same install options as before, I am not sure it works now for dependent libraries such as my pgcopy. My setup.py specifies "psycopg2" in install_requires, but for users relying on the psycopg2 binary distribution this no longer works. Obviously, changing setup.py to require "psycopg2-binary" isn’t an acceptable solution, but I am not sure how else to address this. pip doesn’t recognize psycopg2-binary as a variant of psycopg2, so installing the binary first won’t prevent pip from trying to install psycopg2 from source as a pgcopy dependency subsequently.

I imagine the following have been discussed before:

• Trying to convince the pip/python packaging folks to provide a mechanism for a python package to set no-binary as default installation method.

• Modifying the binary package so pip will recognize it as psycopg2 and won’t try to install psycopg2 as a dependency of other packages (request support for this from python packaging folks, if it isn’t currently possible)

I don’t know if either of the above are feasible or likely, but I would argue that until one or the other are supported, recommending --no-binary psycopg2 for production use and wherever bugs arise with binary installs is superior to the split-package solution, as undesirable as it might be, since it will allow dependent packages to offer the same installation options that psycopg2 itself offers.

Thanks again for your efforts

[dvarrazzo]

@altaurog I don't understand what you are even asking from this project. pip behaviour is not within the realm of the things we can change. If you can make so that the binary packages declares "provides psycopg" then propose a patch to our setup.py. If pip doesn't support it propose a change to setuptools. They could provide a way to specify that a package depends on "X or Y" for instance. Whatever, it is not a thing you can resolve complaining in a closed bug.

We are not changing to distribute psycopg2 wheels by default again, until someone provides a way to reproduce #543, and a fix for it. This is not even remotely open for discussion. Your argument is "it works for me, so sod off everyone else". I have a different opinion: in my view psycopg's reliability comes before the property of being installable without a compiler and a dev package.

There is plenty you could fix. Fix pip. Provide a script to reproduce #543 (because someone is working on packaging libssl 1.1, which may work or not, without reproducing #543 there is not a chance those packages will be default. Don't ask me to reduce psycopg reliability for your convenience.

[cjw296]

@dvarrazzo - I understand that this is an inflammatory issue but I feel @altaurog's comments were pretty reasonable. People who write libraries that depend on psycopg2 are currently put in a tough spot: it's not our place to mandate either psycopg2 or psycopg2-binary but we should really specify a dependency we have.

I think the inclusion of the warning in the last binary-psycopg2 releases was unfortunate, but pragmatically all library authors can currently do is not depend on neither but then explain to downstream library users that they will have to pick one of the two. That's ugly and unpleasant, but not unsurmountable ;-)

From what you've said, it sounds like your preferred choice for library developers would be to depend on psycopg2 as the safest option?

How about just killing psycopg2-binary completely if it causes possible segfaults?

[dvarrazzo]

@cjw296 libraries developers have no problem: they can specify psycopg2 and tell their users to install the required dev packages. That is definitely my suggestion and I consider doing otherwise wrong.

Another option us to depend on none and ask your user to install psycopg themselves.

Another user is to depend on <2.8.

Killing binary is an option yes. It won't fix people opening bugs to report a failed installation though. I've lost my hopes about that.

You can still lobby setuptools team asking to improve setup metadata.