Segmentation fault when connecting using SSL #543
Comments
|
I think I know what it is, and it is not pretty :( you are using a wheel package, which is self-contained and is bundled with its own version of libssl. Python uses the system version of the library. The two libraries together probably clash on some global object, or step onto each other's private structure. I think the problem cannot be solved, which is sad: the wheel package is probably not compatible with the
Thorny problem. |
|
Improve docs by 2.7.2 |
|
Wow. This one burned me for a couple of hours. All I was getting was "segmentation fault" on ubuntu 16.04 with a tornado + aiopg app. I have to investigate the core to get here. Rolling back to psycopg2==2.6.2 seems to have resolved things for me. |
|
@oliverseal Yeah, that's very unfortunate and unfortunately I don't see a way to solve it. You can still use 2.7 forcing source installation. I've added a warning in the docs about the problem, but if you had to dig deeper it may mean that warning is not enough. Please feel free to suggest how we can improve the docs further to help people not waste too much time. |
|
Would it be possible to not bundle libssl in the wheel and use the one that comes with python instead? |
|
@rubendv I don't think so because it's the libpq to bind to the libssl, not psycopg directly. Maybe there could be a way to exclude the libssl from the wheel generation; then we should check the binary compatibility libssl offers. It would be actually interesting to take a look into that. |
|
Why keep the wheel, in the meantime, given how broken it is? Building from source should work for far more environments than the wheel, or at least, error out if the appropriate libraries / headers are available. Presently, naïvely attempting to install |
|
@thanatos I have maintained this library for more 10 years and I can tell you that people incapable of reading the first line of the manual, of installing the -dev packages, opening bugs because of the mysterious writing "I can't find pg_config: please add it to the path" are an endless army. The wheel is presently only broken for people also using libssl, which are a minority. People who have automated deployment by just sticking For every person complaining about this segfault I have 100 OSX dummies who stopped complaining. It is a net win for me. |
|
Issue re-opened, because documenting a workaround doesn't effectively close it. |
My understanding is that the standard library's
My options here are also fairly limited: we actually do attempt to pin dependencies s.t. we don't pull in behavioral changes inadvertently. However, we have a large number of dependencies, and our time is severely limited; we don't always have time to chase down libraries to see if they use some custom, home-grown versioning scheme, and what that is. For the most part, we don't bother, as it is usually not specified anywhere and thus a fool's errand; my life would be a lot easier if folks would just use semver (and if
I wasn't aware that this was even valid syntax for a requirements file. However, it unfortunately seems to apply to the entire set, not the single requirement (and thus doesn't work around the issue of subsequently needing to rebuild everything). (If I make a requirements file with that and a single other requirement, This also ignores that things being pulled in through other dependencies are going to need to push their dependencies through
Thank you.
I'm not against bundling the postgres library; that seems fairly safe to do; it's the bundled |
|
Of course I'm not happy that psycopg segfaults in any condition, let alone in conjunction with the use of a stdlib module. It would be great if we could exclude libssl from the wheel package; unfortunately this is not possible because libssl doesn't have a stable ABI. It is true that it pretty much exists everywhere; the problem is that we don't know what shape it has. It would also be great if the workflow could be: however there is no "prefer source" or "prefer binary" option in pip, AFAICS. Maybe it's worth asking them? As per the issue that As per the semantic versioning: we do it. The 2 is there to stay in So to recap, the problem with psycopg wheels is caused by a combination of
on the first being fixed I don't hold my breath. Point three could be just a bug, the second would be nice, but I don't have the energy to lobby the pip guys. If you can help with some of the above, be my guest. If not, please don't insist that we drop the wheel support because some random guy in pypi rolls three dice to choose their version numbers and you don't feel like adding version guards in your req's file. I'm happy to work on the shortcomings of the libraries I maintain and to give alternatives to work around problems, but I have limited agency upstream (the libraries we use) and downstream (the users we have). |
|
Looked better into all the packages going no-binary in The right solution is to use Installs: where lxml is installed from wheel and psycopg from source. I will update the documentation to suggest this workaround. |
:all: applies to the entire file. --no-binary psycopg2 is the solution. See issue #543
|
I would like to work to solve this problem, but I can't actually reproduce it. Tried importing libssl both by psycopg and by urllib/requests with a script like the following, which in my understanding should have triggered the segfault from __future__ import print_function
import requests
s = requests.get('https://github.com')
print("from requests:", len(s.content))
try:
from urllib2 import urlopen
except ImportError:
from urllib.request import urlopen
f = urlopen('https://github.com')
print("from urllib:", len(f.read()))
import psycopg2
cnn = psycopg2.connect('host=localhost sslmode=require')
cur = cnn.cursor()
cur.execute("select 1")
print("from psycopg:", cur.fetchone())checking in however the program completes without segfaulting. Tested on Python 2.7.12 and 3.5.2 on Ubuntu 16.04.3 64 bit. Any suggestion about how to trigger the segfault reliably? |
|
@rubendv would you be able to provide a standalone test to reproduce the issue? I've tried in an ubuntu 14:04 docker image containing: I couldn't reproduce the segfault, tested with psycopg 2.7.1 (the one you tested with) and 2.7.3.2 (the current one). Your test depends on a module whose behaviour is not clear (just importing fabric.api doesn't seem to import A more self-contained test in a dockerfile would be welcome to fix the problem, thank you. |
|
Whilst I can't easily reproduce it, I have experienced the same issue using In my cases, it seems to occur more frequently (and even then, not actually frequently) when Below is the debug information I have from the OSX console, which is ... possibly useless? I certainly wouldn't be able to diagnose anything from it. Given its up to thread 7 (or 8 in some cases) I assume I was running a And now what I assume is some assembly: I appreciate that more anecdata probably doesn't offer up any new information or help, but just in case :) |
|
Observation:
|
The idea is to release a package 'psycopg2-binary' to allow installing binary, and leave the psycopg2 package to be source only, to avoid pushing the unreliability of the wheel pacakge by default (see issue #543). Version number bumped to test with new packages.
|
Starting from psycopg 2.7.4, a new package will be introduced: psycopg2-binary. The psycopg2 wheel package will be still available but will importing it will raise an explanatory warning. Starting from psycopg 2.8, only psycopg2-binary will be available as wheel package and 'pip install psycopg2' will require build tools on the client. In case of missing build prerequisites, setup.py will also inform about the availability of the psycopg2-binary package. |
The idea is to release a package 'psycopg2-binary' to allow installing binary, and leave the psycopg2 package to be source only, to avoid pushing the unreliability of the wheel pacakge by default (see issue #543). Version number bumped to test with new packages.
The wheel package for `psycopg2` is not compatible with the Python `ssl` module due to their dependency on different versions of OpenSSL (the wheel packages its own version, while `ssl` uses the system version). Attempts to resolve by installing the dependency from source proved challenging because we install `psycopg2` via Ansible (`postgresql-support`) in a way that makes it difficult to supply the `--no-binary` override for `pip`. Instead, we've downgraded to a version prior to wheel support. Lastly, the `libpq5` and `libpq-dev` dependencies needed to be downgraded as well. This is because `psycopg2==2.6.2` is unable to make sense of a `libpq` version greater than 9. See: psycopg/psycopg2#543
The wheel package for `psycopg2` is not compatible with the Python `ssl` module due to their dependency on different versions of OpenSSL (the wheel packages its own version, while `ssl` uses the system version). Attempts to resolve by installing the dependency from source proved challenging because we install `psycopg2` via Ansible (`postgresql-support`) in a way that makes it difficult to supply the `--no-binary` override for `pip`. Instead, we've downgraded to a version prior to wheel support. Lastly, the `libpq5` and `libpq-dev` dependencies needed to be downgraded as well. This is because `psycopg2==2.6.2` is unable to make sense of a `libpq` version greater than 9. See: psycopg/psycopg2#543
|
I'm happy to help fix the wheels, as above - where should I do a PR? |
|
@matthew-brett That would be awesome! cc @dvarrazzo |
|
@matthew-brett I guess the https://github.com/psycopg/psycopg2-wheels package is the right place. |
We believe that OpenSSL 1.1.x will fix the threading issues previously reported with binary builds: psycopg#543 Require the OpenSSL 1.1.x Windows library names.
|
@rubendv - and others on this thread - we may well have a fix for the threading problems, in psycopg/psycopg2-wheels#8, but getting default wheels back, depends on having a test for this issue - see #837. We need a test case that we can run, that crashes using current |
167: Update psycopg2 requirement from ~=2.8.3 to ~=2.8.4 in /config r=MiklerGM a=dependabot-preview[bot] Updates the requirements on [psycopg2](https://github.com/psycopg/psycopg2) to permit the latest version. <details> <summary>Changelog</summary> *Sourced from [psycopg2's changelog](https://github.com/psycopg/psycopg2/blob/master/NEWS).* > Current release > --------------- > > What's new in psycopg 2.8.4 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > - Fixed building with Python 3.8 (:ticket:`[#854](https://github.com/psycopg/psycopg2/issues/854)`). > - Don't swallow keyboard interrupts on connect when a password is specified > in the connection string (:ticket:`[#898](https://github.com/psycopg/psycopg2/issues/898)`). > - Don't advance replication cursor when the message wasn't confirmed > (:ticket:`[#940](https://github.com/psycopg/psycopg2/issues/940)`). > - Fixed inclusion of ``time.h`` on linux (:ticket:`[#951](https://github.com/psycopg/psycopg2/issues/951)`). > - Fixed int overflow for large values in `~psycopg2.extensions.Column.table_oid` > and `~psycopg2.extensions.Column.type_code` (:ticket:`[#961](https://github.com/psycopg/psycopg2/issues/961)`). > - `~psycopg2.errorcodes` map and `~psycopg2.errors` classes updated to > PostgreSQL 12. > - Wheel package compiled against OpenSSL 1.1.1d and PostgreSQL at least 11.4. > > > What's new in psycopg 2.8.3 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > - Added *interval_status* parameter to > `~psycopg2.extras.ReplicationCursor.start_replication()` method and other > facilities to send automatic replication keepalives at periodic intervals > (:ticket:`[#913](https://github.com/psycopg/psycopg2/issues/913)`). > - Fixed namedtuples caching introduced in 2.8 (:ticket:`[#928](https://github.com/psycopg/psycopg2/issues/928)`). > > > What's new in psycopg 2.8.2 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > - Fixed `~psycopg2.extras.RealDictCursor` when there are repeated columns > (:ticket:`[#884](https://github.com/psycopg/psycopg2/issues/884)`). > - Binary packages built with openssl 1.1.1b. Should fix concurrency problems > (:tickets:`[#543](psycopg/psycopg2#543), [#836](https://github.com/psycopg/psycopg2/issues/836)`). > > > What's new in psycopg 2.8.1 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > - Fixed `~psycopg2.extras.RealDictRow` modifiability (:ticket:`[#886](https://github.com/psycopg/psycopg2/issues/886)`). > - Fixed "there's no async cursor" error polling a connection with no cursor > (:ticket:`[#887](https://github.com/psycopg/psycopg2/issues/887)`). > > > What's new in psycopg 2.8 > ------------------------- > > New features: ></tr></table> ... (truncated) </details> <details> <summary>Commits</summary> - See full diff in [compare view](https://github.com/psycopg/psycopg2/commits) </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com): - Update frequency (including time of day and day of week) - Pull request limits (per update run and/or open at any time) - Automerge options (never/patch/minor, and dev/runtime dependencies) - Out-of-range updates (receive only lockfile updates, if desired) - Security updates (receive only security updates, if desired) </details> Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
This is due to a segfault issue as reported in psycopg/psycopg2#543
230: Update psycopg2 requirement from ~=2.8.4 to ~=2.8.5 in /config r=whirish a=dependabot-preview[bot] Updates the requirements on [psycopg2](https://github.com/psycopg/psycopg2) to permit the latest version. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/psycopg/psycopg2/blob/master/NEWS">psycopg2's changelog</a>.</em></p> <blockquote> <h2>Current release</h2> <p>What's new in psycopg 2.8.5 ^^^^^^^^^^^^^^^^^^^^^^^^^^^</p> <ul> <li>Fixed use of <code>!connection_factory</code> and <code>!cursor_factory</code> together (:ticket:<code>[#1019](https://github.com/psycopg/psycopg2/issues/1019)</code>).</li> <li>Added support for <code>~logging.LoggerAdapter</code> in <code>~psycopg2.extras.LoggingConnection</code> (:ticket:<code>[#1026](https://github.com/psycopg/psycopg2/issues/1026)</code>).</li> <li><code>~psycopg2.extensions.Column</code> objects in <code>cursor.description</code> can be sliced (:ticket:<code>[#1034](https://github.com/psycopg/psycopg2/issues/1034)</code>).</li> <li>Added AIX support (:ticket:<code>[#1061](https://github.com/psycopg/psycopg2/issues/1061)</code>).</li> <li>Fixed <code>~copy.copy()</code> of <code>~psycopg2.extras.DictCursor</code> rows (:ticket:<code>[#1073](https://github.com/psycopg/psycopg2/issues/1073)</code>).</li> </ul> <p>What's new in psycopg 2.8.4 ^^^^^^^^^^^^^^^^^^^^^^^^^^^</p> <ul> <li>Fixed building with Python 3.8 (:ticket:<code>[#854](https://github.com/psycopg/psycopg2/issues/854)</code>).</li> <li>Don't swallow keyboard interrupts on connect when a password is specified in the connection string (:ticket:<code>[#898](https://github.com/psycopg/psycopg2/issues/898)</code>).</li> <li>Don't advance replication cursor when the message wasn't confirmed (:ticket:<code>[#940](https://github.com/psycopg/psycopg2/issues/940)</code>).</li> <li>Fixed inclusion of <code>time.h</code> on linux (:ticket:<code>[#951](https://github.com/psycopg/psycopg2/issues/951)</code>).</li> <li>Fixed int overflow for large values in <code>~psycopg2.extensions.Column.table_oid</code> and <code>~psycopg2.extensions.Column.type_code</code> (:ticket:<code>[#961](https://github.com/psycopg/psycopg2/issues/961)</code>).</li> <li><code>~psycopg2.errorcodes</code> map and <code>~psycopg2.errors</code> classes updated to PostgreSQL 12.</li> <li>Wheel package compiled against OpenSSL 1.1.1d and PostgreSQL at least 11.4.</li> </ul> <p>What's new in psycopg 2.8.3 ^^^^^^^^^^^^^^^^^^^^^^^^^^^</p> <ul> <li>Added <em>interval_status</em> parameter to <code>~psycopg2.extras.ReplicationCursor.start_replication()</code> method and other facilities to send automatic replication keepalives at periodic intervals (:ticket:<code>[#913](https://github.com/psycopg/psycopg2/issues/913)</code>).</li> <li>Fixed namedtuples caching introduced in 2.8 (:ticket:<code>[#928](https://github.com/psycopg/psycopg2/issues/928)</code>).</li> </ul> <p>What's new in psycopg 2.8.2 ^^^^^^^^^^^^^^^^^^^^^^^^^^^</p> <ul> <li>Fixed <code>~psycopg2.extras.RealDictCursor</code> when there are repeated columns (:ticket:<code>[#884](https://github.com/psycopg/psycopg2/issues/884)</code>).</li> <li>Binary packages built with openssl 1.1.1b. Should fix concurrency problems (:tickets:<code>[#543](psycopg/psycopg2#543), [#836](https://github.com/psycopg/psycopg2/issues/836)</code>).</li> </ul> </tr></table> ... (truncated) </blockquote> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/psycopg/psycopg2/commits">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com): - Update frequency (including time of day and day of week) - Pull request limits (per update run and/or open at any time) - Automerge options (never/patch/minor, and dev/runtime dependencies) - Out-of-range updates (receive only lockfile updates, if desired) - Security updates (receive only security updates, if desired) </details> Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
tl;dr
This segfault only affects the psycopg wheel package: the workaround is to build psycopg from source, e.g. using:
or the entry in the
requirements.txtfile:Running the delete_database function in this snippet against our database causes a segmentation fault in psycopg2==2.7 and 2.7.1, but not in 2.6.2. The segmentation fault seems to happen inside OpenSSL.
Debugging revealed that SSL_library_init is being called twice, once by the Python ssl module and once by the libpq library that comes with psycopg2, where it segfaults. Not sure if this function is allowed to be called twice or not.
The system openssl is 1.0.1f-1ubuntu2.22 on Ubuntu 14.04.
On psycopg2==2.6.2 this works fine.
On psycopg2==2.7 we get the following segmentation fault:
On psycopg2==2.7.1 we get the following segmentation fault:
The text was updated successfully, but these errors were encountered: