How to use external python libraries in a godot project with virtualenv?

[creikey]

As changing the python interpreter to that of one in a virtualenv breaks linting and library import in vscode, I don't understand how to use the python executable bundled with the library ( and its associated site packages ) with a virtual env and pip

[touilleMan]

Hi @creikey,

The idea is you don't need a virtual env given your game project has it own virtual env (well it ships it own python interpreter entirely separated from your system's one).

[creikey]

@touilleMan I tried doing this but couldn't figure it out, can you call the pip that's installed in the game project? Whenever I ran it it couldn't link to the library in another folder

[touilleMan]

I've made some test and it seems the pip command is broken:

$ cd examples/pong
$ ./pythonscript/bin/pip
Failed to execute process './pythonscript/bin/pip'. Reason:
The file './pythonscript/bin/pip' specified the interpreter '/build/out/python/install/bin/python3', which is not an executable command.

However python command works fine, and you can use python -m pip to replace pip:

$ cd examples/pong
$ ./pythonscript/bin/python3 -m pip install requests
[...]
$ ls pythonscript/lib/python3.7/site-packages/
certifi/                       chardet-3.0.4.dist-info/  _godot.so*           pip-19.3.1-py3.7.egg/  requests-2.23.0.dist-info/   urllib3/
certifi-2019.11.28.dist-info/  easy-install.pth          idna/                README.txt             setuptools-41.4.0-py3.7.egg  urllib3-1.25.8.dist-info/
chardet/                       godot/                    idna-2.9.dist-info/  requests/              setuptools.pth
$ ./pythonscript/bin/python3 -c "import requests; print(requests.__version__)"
2.23.0

[chrisgraf]

I tried the workaround you mentioned above but get the following error message in the latest release (0.30.0):
$ ./pythonscript/x11-64/bin/python3
./pythonscript/x11-64/bin/python3: error while loading shared libraries: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory

[touilleMan]

hi @chrisgraf, thanks for pointing this out !

$ ldd ./pythonscript/x11-64/bin/python3
	linux-vdso.so.1 (0x00007ffe13b89000)
	libpython3.7m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.7m.so.1.0 (0x00007f7659668000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7659477000)
	libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f7659449000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f765942d000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f765940a000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7659404000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f76593fd000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f76592ae000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f7659b7b000)

On my computer, ./pythonscript/x11-64/bin/python3 loads the wrong libpython3.7 (i.e. the one in /usr/lib instead of the one embedded in pythonscript !), can you check you have the same trouble on your side ?

The quick'n dirty solution is to use LD_PRELOAD to force python to load the right lib:

$ env LD_PRELOAD=./pythonscript/x11-64/lib/libpython3.7m.so ldd ./pythonscript/x11-64/bin/python3

To fix this properly, we should customize the link step of the python3 compilation use -Wl,-rpath option in order to provide a relative path to the libpython (this is what is already done for the cython modules)

[chrisgraf]

@touilleMan : thanks for your answer! It indeed seems to be the case. In my case the link to the library is not found at all:

$ ldd ./pythonscript/x11-64/bin/python3
	linux-vdso.so.1 (0x00007ffea5aa2000)
	libpython3.7m.so.1.0 => not found
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f801e6c4000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f801e6be000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f801e6b9000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f801e56a000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f801e378000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f801e711000)

[chrisgraf]

I could fix the library linking issue with a hint from @matheus2740 in #174

what I did was:

pythonscript/x11-64/bin$ patchelf --replace-needed libpython3.7m.so.1.0 <path-to-project>/pythonscript/x11-64/lib/libpython3.7m.so.1.0 python3
pythonscript/x11-64/bin$ patchelf --replace-needed libpython3.7m.so.1.0 <path-to-project>/pythonscript/x11-64/lib/libpython3.7m.so.1.0 python3.7
pythonscript/x11-64/bin$ patchelf --replace-needed libpython3.7m.so.1.0 <path-to-project>/pythonscript/x11-64/lib/libpython3.7m.so.1.0 python3.7m

but when I try to execute python3 -m pip the following happens:

$ ./pythonscript/x11-64/bin/python3 -m pip install requests
Traceback (most recent call last):
  File "/home/graf/landscapelab/godot-python-test/pythonscript/x11-64/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/graf/landscapelab/godot-python-test/pythonscript/x11-64/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/graf/landscapelab/godot-python-test/pythonscript/x11-64/lib/python3.7/site-packages/pip/__main__.py", line 16, in <module>
    from pip._internal import main as _main  # noqa
  File "/home/graf/landscapelab/godot-python-test/pythonscript/x11-64/lib/python3.7/site-packages/pip/_internal/__init__.py", line 42, in <module>
    from pip._internal import cmdoptions
  File "/home/graf/landscapelab/godot-python-test/pythonscript/x11-64/lib/python3.7/site-packages/pip/_internal/cmdoptions.py", line 16, in <module>
    from pip._internal.index import (
  File "/home/graf/landscapelab/godot-python-test/pythonscript/x11-64/lib/python3.7/site-packages/pip/_internal/index.py", line 25, in <module>
    from pip._internal.download import HAS_TLS, is_url, path_to_url, url_to_path
  File "/home/graf/landscapelab/godot-python-test/pythonscript/x11-64/lib/python3.7/site-packages/pip/_internal/download.py", line 39, in <module>
    from pip._internal.utils.glibc import libc_ver
  File "/home/graf/landscapelab/godot-python-test/pythonscript/x11-64/lib/python3.7/site-packages/pip/_internal/utils/glibc.py", line 3, in <module>
    import ctypes
  File "/home/graf/landscapelab/godot-python-test/pythonscript/x11-64/lib/python3.7/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: libffi.so.6: cannot open shared object file: No such file or directory

[matheus2740]

this seems to be a problem with LD_LIBRARY_PATH... @touilleMan can you check out what's happening with build on linux?

[touilleMan]

@chrisgraf your trick with patchelf is really neat ! Do you know if we can provide a relative path with this tool ? This way we could easily fix the python interpreter without having to recompile it

@matheus2740 do you think this will work on macOS ?

Considering the libffi error, I guess you don't have libffi installed on your system:

$ apt search libffi
Sorting... Done
Full Text Search... Done
[...]
libffi6/eoan,now 3.2.1-9 amd64 [installed]
  Foreign Function Interface library runtime
[...]

Of course we cannot expect for 3rd party user to be required to install libffi prior to play on a game ! Pyoxidizer has an interesting part about libffi used in python
So basically we should compile&ship libffi (along with libssl which I think we don't support correctly currently)
Pyoxidizer's sister project python-build-standalone is really interesting, ideally I would like to use it to provide us a ready to use embededdable python distribution ;-)

[chrisgraf]

@touilleMan : sadly I was unsuccessful using a relative path with patchelf

libffi is installed at my system:

$ apt list libffi7
Listing... Done
libffi7/focal,now 3.3-4 amd64 [installed,automatic]
libffi7/focal,now 3.3-4 i386 [installed,automatic]

[chrisgraf]

Ok there is a version mismatch in libffi .. the latest ubuntu ships with libffi7 but libffi.so.6 is searched for in the error mentioned above. So maybe a static linked version as you proposed would be a good long term solution!

I will try to manually provide libffi6 now for further testing. Are there more dependencies (I think you mentioned libssl somewhere) that are required in a specific version?

[chrisgraf]

For documentation reasons: I simply put a symlink from libffi7 to libffi6 an it SEAMS to work:

/usr/lib/x86_64-linux-gnu$ ln -s libffi.so.7.1.0 libffi.so.6

[matheus2740]

@touilleMan I'm already doing this in macOS, except the tool that modifies Mach-O (mac binaries) is called install_name_tool. Take a look at the macOS Sconscript file.

[chrisgraf]

The original issue (broken pip in distribution) is not fixed in 0.40.0

$ ./pip3.8 
bash: ./pip3.8: /build/out/python/install/bin/python3: bad interpreter: No such file or directory

the workaround with python3 -m pip does work now however

[touilleMan]

@chrisgraf yep, I've opened issue #155 for the pip command not working

[chrisgraf]

sorry, didn't see that! Thanks for 0.40.0!

[whogben]

hi @chrisgraf, i'm trying to get some packages installed in godot python and stuck - you mentioned you got it working with a workaround, would you mind summarizing what you did to get it working?

It sounds like @touilleMan is describing the workaround in this thread, but I can't tell which files need to be edited / with what values: #155

Alternatively, if you know where site packages would go, I could try manually moving the site packages in from another python interp after I pip install them there - that worked for me on the older godot-python version, but I couldn't find a site_packages folder in the new one.

Many thanks for taking a look!

[chrisgraf]

@whogben: I documented my steps above - I did nothing extra as far as I recall. Sorry that I can not help you with your problem here.