Python freezed binary cannot use psycopg.so

[psycoteam]

Originally submitted by: Christian Bachmaier

I use psycopg2 2.4.5-1buidl5 under Ubuntu 14.04 LTS x86_64. After pachting Python 3.4.0 final as described under http://bugs.python.org/issue16047 to work again since Python 3.2 under Python 3.3 and 3.4 I can sucessfully freeze my Python cgi project to a binary with the in Python included Tool freeze.py command.

However, a freezed script hello.py containing only the one line

import psycopg2

does not operate, since it does not find the psycopg.so library. It delivers after execution:

Traceback (most recent call last):
  File "hello.py", line 18, in <module>
    import psycopg2
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2203, in _find_and_load_unlocked
    module = _SpecMethods(spec)._load_unlocked()
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 1200, in _load_unlocked
    self._exec(module)
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 1129, in _exec
    self.spec.loader.exec_module(module)
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 1336, in exec_module
    exec(code, module.__dict__)
  File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 67, in <module>
    from psycopg2._psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2201, in _find_and_load_unlocked
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'psycopg2._psycopg'

In Python 3.2 it helped to create a link
_psycopg.so -> /usr/lib/python3/dist-packages/psycopg2/_psycopg.cpython-32mu.so
in a subdir psycopg2 of the current directory of the execution of the freezed binary.

The same issue is on Python3.4r2 under Debian Wheezy/Sid.

[dvarrazzo]

It doesn't seem a psycopg bug to me: it seems a freeze problem. We just provide a setup.py for the Python toolchain to build. Or is there anything we have to change in setup.py to support new features?

[dvarrazzo]

Taken a look at the above bug report. Closing as not a psycopg bug.

[psycoteam]

Originally submitted by: Christian Bachmaier

Adding

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

to the top of psycopg2/init.py resolves the problem and the frozen binary works as expected. I had to learn this in a painful and time wasting way without any help from here or the Python bug databse http://bugs.python.org/issue16047 :(

I am not quite sure if this is due to a flaw of Python/freeze as running in interpreted mode does not need it. At least the behaviours in interpreted and frozen mode should not be different. Or it is a bug of psycopg2. However, the above works in both modes. Other libraries like janitor or gi do already contain these lines.

[dvarrazzo]

from extend_path docs:

"This is useful if one wants to distribute different parts of a single logical package as multiple directories"

This doesn't sound what psycopg wants to do. Could there be a way to adjust psycopg imports to make it compatible with freeze? This comment from M.-A. L. seems suggesting it is possible.

If you can fix psycopg by adjusting its imports but remaining in the realm of the canonical ways to import internal modules I'll be happy to apply the patch. If you want to see the patch applied in psycopg 2.5.x it must be compatible with everything between Python 2.5 and 3.4.

I'll leave the bug open for you but I won't be working on this problem: the ball is yours.

[psycoteam]

Originally submitted by: Christian Bachmaier

As far as I can tell, this is exactly as needed. Using debug outputs like

print(__path__)

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

print(__path__)

show that running in interpreted mode does not change the search path path of the package:

['/usr/lib/python3/dist-packages/psycopg2']
['/usr/lib/python3/dist-packages/psycopg2']

So there is not any change for that at all. Everything stays as before.

In compiled mode path seems to be empty. See the link in my previous postings. The guys there told me that this is by design. Don't know why, this may be a bug in freeze if you are asking me. Hower running in compiled mode, the above code only adds the psycopg2 directory which contains the native so-library and which is there in interpreted mode. Then the native library can be found at runtime by a frozen binary.

[]
['/usr/lib/python3/dist-packages/psycopg2']

Other (similar built up) libraries use this code. Look at /usr/lib/python3/dist-packages/gi or .../janitor.

Fazit: The 2 lines fix the problem enitrely. There is nothing to work on, but to test with older python version and to put into the official code. Unfortunately, I can only test with 3.4 at the moment. Theoretically this should work also with older Python versions. But maybe the maintainer of psycopg have some testing installations?

Remember to use the fixed version of freeze (e.g. hg clone http://hg.python.org/cpython) as prior versions in Python 3.3 and 3.4 were broken.
Then it is easy:

$ /$HOME/cpython-xxx/Tools/freeze/freeze.py hello.py
$ make
$ ./hello

For your convenience I have also attached an archive with an operating version of freeze. So at least for 3.3 and 3.4 you need not to download it.
hello.py contains some code which uses psycopg2. Consiting only of a line

import psycopg2

already suffices.

[dvarrazzo]

Christian, extend_path is not designed to fix freeze but for a different use case that doesn't apply to psycopg, so I'm very reluctant to add it. In the past I've been pyinstaller maintainer and I remember having no problem with psycopg. For what I (don't) know your solution may end up breaking other freeze solutions that currently work.

Please talk with freeze authors and have the issue fixed there. If we are doing our internal imports wrong I'm ready to change every single one of them, but I'm not happy to add a call to a function that I don't know what it does and the docs say it's not for our use; the fact two packages I don't know anything about do the same is not enough for me to accept it as a quality solution.

[dvarrazzo]

Christian, I've taken a look at your problem.

Freeze cannot freeze psycopg just because it cannot work with C extension. This should be enough to rule out what you want to do. I've taken a look at why adding the unrelated hack of extend_path() work: it works just because it finds an externally available _psycopg.so, however we don't want to distribute psycopg "as multiple directories": that's not for us.

Note that using absolute or relative imports doesn't change anything. However we will likely move to relative imports as they are the right thing and we don't target Python 2.4 anymore.

In order to solve your problem you can add an import hack into your frozen application:

sys.path.insert(0, '/path/to/psycopg2')
import _psycopg
sys.modules['psycopg2._psycopg'] = _psycopg
sys.path.pop(0)

# this will now work
import psycopg2

so you can ship _psycopg.so in a separate directory together with your frozen application and make it available before psycopg2 itself.

The above hack only works with a psycopg version patched with this changeset; without it importing _psycopg would fail because it would try to import psycopg2.tz.

This is the best I can do for your issue. Please acknowledge that you are trying to use a tool designed to freeze pure Python modules: if it doesn't work for psycopg2 it isn't our fault and we cannot do much. Please test the workaround above: if it works I'll try and ship the patch in the next 2.5.4 release (it still needs some testing).

[psycoteam]

Originally submitted by: Christian Bachmaier

Daniele, thanks.

Your code sys.path Manipulation works with the version from

cd /scratch
git clone https://github.com/dvarazzo/psycopg -b freeze
cd psycopg
python3 setup.py build

and using

import sys
sys.path.insert(0, '/scratch/psycopg2/build/lib.linux-x86_64-3.4/psycopg2')
...

Please let me know from which release version on the changes will be contained.
I assume that in init.py

from pkgutil import extend_path
path = extend_path(path, name)

does something similar, as many other libraries (see above) use that. The advantage is that the library does the trick then and not the user programm.

However, there should be a clean way. Either this is a bug in freeze or in psycopg2. I tend to say it is in freeze as the interpreted and compiled beavior should be identical. However, the freeze guys are not very amused about my opionion (http://bugs.python.org/issue16047). Maybe you could open a ticket?

Please acknowledge that you are trying to use a tool designed to freeze pure
Python modules

The docu (I have found) says nothing about not supporting .so-files, nor the maintainer of freeze do, if I interpret them right (see above link).

[dvarrazzo]

Hi Christian,

I consider the refactoring I've made to _psycopg.so generally useful and I've tested it more yesterday. If there is no surprise from the test grid (with python versions I've not tested yet) I'll release soon, in the next 2.5.4.

Have another test: I suspect if _psycopg.so is in the same directory of the executable you don't even need the sys.path hack, but I may be wrong.

The docu (I have found) says nothing about not supporting .so-files,
nor the maintainer of freeze do, if I interpret them right (see above link).

Freeze doesn't support non-pure modules at all: read its docstring:

The script should not use modules provided only as shared libraries;
if it does, the resulting binary is not self-contained.

so, that's what happens. Freeze doesn't freeze the .so, but the resulting executable can still use them if it finds them, i.e. with the regular python importing machinery. It's up to you to make the .so available to the exe. I've fixed psycopg2 so that the .so can be imported outside the package and that's about what I can do to help whatever program needs pythonpath hacking. But you are really asking too much to freeze: other programs are better suited to create executables out of non-pure modules (e.g. pyinstaller).

Closing the bug. Have a nice day.

[psycoteam]

Originally submitted by: Christian Bachmaier

Have another test: I suspect if _psycopg.so is in the same directory of the
executable you don't even need the sys.path hack, but I may be wrong.

Unfortunately, this does not work, even not if _psycopg.so is in a local subdirectory psycopg. The latter worked with python 2.7 and its freeze.