support for freeze and python 3.7

[chris21k]

Since Daniele told me how to use psycopg together with freezing my app under
https://psycopg.lighthouseapp.com/projects/62710/tickets/201
this worked like a charm. Still, one gets a performance boost.
Precisely, I am using

sys.path.insert(0, '/path/to/psycopg2')
import _psycopg  # this is line 7
sys.modules['psycopg2._psycopg'] = _psycopg
sys.path.pop(0)

import psycopg2

on top of an example dbtest.py file. (It also suffices to have only this shown code in dbtest.py to show the behavior). Running python3 dbtest.py and freezing and running the resulting binary dbtest used to work like a charm. However, today I got an update to Python 3.7.5 on my Ubuntu 20.04/focal development/testing installation. psycopg2 is in verstion 2.8.3
Freezing and running the dbtest binary works as ususall, however, now running it with python3 dbtest.py delivers

Traceback (most recent call last):
  File "./dbtest.py", line 7, in <module>
    import _psycopg
SystemError: initialization of _psycopg raised unreported exception

I belive that with python 3.7.4 everything was ok, but I cannot swear it. Commenting out the lines of the hack fixes the python3 dbtest.py run. However, both needs to work...

Thanks,
Chris

[chris21k]

Again, as in the old bug report, adding

from pkgutil import extend_path
__path__ = extend_path(__path__, __name)

to the top of psycopg2/__init__.el (and removing the above code which extends sys.path) helps.
Other (similar built up) libraries use this code. Look at /usr/lib/python3/dist-packages/gi

Chris

[dvarrazzo]

I don't understand what is this report. What are we supposed to do?

[chris21k]

Dear Daniele,

I need to freeze my Python cgi project to a binary with the in Python included Tool/command freeze.py (see Python gzipped src tgz under Tools/freeze). This makes problems in combination with the usage of psycopg2, i.e. to find the psycopg2 package/library when running as freezed binary. However, you told me some years ago (see https://psycopg.lighthouseapp.com/projects/62710/tickets/201) to introduce the code snippet/hack above (the very first in this report) to the start of my programm. This worked great until the newest versions Python 3.7.5 and pyscopg2 2.8.3 in Ubuntu 20.04 development.

Here freezing and running the resulting binary works as desired (with the hack), however, when running live/interpreted a failure message (see above) is triggered.

So now I am heavily addicted to an alternative for a tool in productive use at my University. The best from my point of view is psycopg2 would support freezing out of the box. My second posting above shows the way of other libraries with a native part which does support this. I do not know if this is the best way. Or could you give at least some advice/fix of the hack to also work in interpreted mode?

Thanks,
Chris

[dvarrazzo]

We didn't change anything in psycopg 2.8.3: it must be something that changed on Python 3.7.5.

How to freeze the library is a responsibility of whatever tools you use to freeze, not ours. If you or they get in touch explaining how to allow freezing it we will be happy to support it, but it's not something we should research into ourselves.

Please explain how to support it out of the box and provide a patch for it, otherwise it will not happen.

[chris21k]

My second posting from top already provides a patch for an out of the box solution. At least I think this is the way other libraries do that (see my reasoning in the old posting from May 17, 2014 in issue #201 which is a copy of https://psycopg.lighthouseapp.com/projects/62710/tickets/201). This solution is also a suggestion of a freeze developer, as far as I can interpolate, see the last paragraph of this comment.

On Aug 25, 2014 you added a patch to psycopg2 to support the version of modifying sys.path as shown in my first posting in this issue, see dvarrazzo/psycopg@af00be8 . Maybe here is (now) some incompatibility?

How to freeze the library is a responsibility of whatever tools you use to freeze, not ours.

Each of the two parties says it is the other ones fault... :) I already tried to investigate also with the python/freeze developers in https://bugs.python.org/issue16047. The interesting part starts here
https://bugs.python.org/issue16047#msg214844 . And freeze does work with Python 3 and with shared libraries.

[dvarrazzo]

My second posting from top already provides a patch for an out of the box solution

What file is psycopg2/__init__.el? Not one of psycopg. From #201 I see you meant "py", not "el"? If so, no, can't be accepted. It's irresponsible for psycopg to manipulate a shared resource such as sys.path (extend_path is used "if one wants to distribute different parts of a single logical package as multiple directories", docs say, and that's not psycopg case). Manipulating the sys.path is totally the responsibility of the importing machinery. We don't do any magic import trick, we are a normal package, and as such I don't expect I need to mess with the pythonpath in order to work.

In psycopg 2.8.3 nothing changed for which whatever used to work now it doesn't - or if you find what it was let us know and we can try and revert it. It is more likely that newer Python versions changed its import semantic.

[chris21k]

What file is psycopg2/init.el?
Sorry for the typo. I mean init.py as you have suggested. In my stream of confusedness I probably mixed it up with init.el of Emacs :(

Ok, I understand your argumentation. So I also will not use the path -extension option in init.py in production. My current ugly solution I will be using is

import sys
try:
    __file__
    frozen_mode = False
except NameError:       # __file__ undefined
    frozen_mode = True
if frozen_mode:
    sys.path.insert(0, '/usr/lib/python3/dist-packages/psycopg2')
    import _psycopg
    sys.modules['psycopg2._psycopg'] = _psycopg
    sys.path.pop(0)

at the beginning of my program. However, this may be insecure, as there is no guarrante that file is undefinded in future versions of Python when running as frozen binary.

It is more likely that newer Python versions changed its import semantic.

Hmm, it was a bugfix version step from 3.7.4 (or 3.7.x) to 3.7.5 . At a quick look I found nothing about that in some release notes.

Still I wonder why the body of the if-statement, i.e., the line import _psycopg, cannot be executed in non-frozen/interpreted mode any more.

Thanks again,
Chris

[dvarrazzo]

Closing as manipulating the import machinery is not in the psycopg2 realm of things to do