Cannot use tensorflow 1.15.2 anymore with pex==2.1.5 #913
Assignees
Labels
Comments
|
I don't repro either the warning line or the import error. I add steps after yours to build a pex file and then dump its metadata so we might compare how our repro attempts differ: |
|
Only diff is the python version and the hash of tensorflow & wrapt is different. I'm on centos7: |
|
When I compile Python 3.6.10 from scratch is seems to work. |
|
I see no obvious change that would affect this either @fhoering. I'm not sure what to say. I suspect, under your python 3.6.8, |
|
No it is not broken inside the virtual env, there tensorflow actually works. Also this happens on multiple machines for different users. I also get it inside a docker container. There must be something, it is too deterministic, maybe the Python installation of Centos7. In my docker I use this: So I will do more tests like recompiling python 3.6.8 and keep you posted. |
|
I think I actually could even provide you a dockerfile where this fails. Maybe even pulling centos7 and installing python as above is enough. |
|
This looks like a bug fixed in the tensorflow 2 series here: tensorflow/tensorflow@c40c5df In older versions, the variable would then be where purelib and platlib are defined to be for the python in question. On a stock CentOS7 I find: On my arch linux box I find: And on the canonical python 3.6 docker image (Debian Buster, custom build) I find: Beyond noting the likely variable in success vs failure here, I have not yet determined exactly how this variable leads to the success / failure split, but maybe that pushes the ball along to help your investigations. |
|
I observed a similar behavior
For now, using |
|
Ok, circling back, this was fixed by 2.1.133. Alas, centos 7 is no more (at least the centos:7 image isn't, but I can use rockylinux:8 to get a facsimile: :; cat <<EOF | docker build -t pex-issues-913 - FROM rockylinux:8
RUN yum -y install python3 python3-devel python3-pip
EOF
:; docker run --rm -it pex-issues-913 python3.6 -c 'import sysconfig; print("\n".join(p + ": " + sysconfig.get_paths()[p] for p in ("purelib", "platlib")))'
purelib: /usr/lib/python3.6/site-packages
platlib: /usr/lib64/python3.6/site-packagesSo, same problematic split purelib / platlib that tripped # Good in Pex 2.1.4:
:; docker run --rm -it pex-issues-913 python3.6 -c 'import sysconfig; print("\n".join(p + ": " + sysconfig.get_paths()[p] for p in ("purelib", "platlib")))'
purelib: /usr/lib/python3.6/site-packages
platlib: /usr/lib64/python3.6/site-packages
# Bad in Pex 2.1.5
:; docker run --rm -it pex-issues-913 bash -c 'python3.6 -mvenv pex.venv && source pex.venv/bin/activate && pip -q install pex==2.1.5 && pex --version && pex --disable-cache tensorflow==1.15.2 -- -c
"import tensorflow; print(tensorflow.__file__)"'
pex 2.1.5
WARNING: Target directory /tmp/tmpny0v8b4t/installed_wheels/94f9f1d978d9665ef102abf2e89004602c84f25f/tensorflow-1.15.2-cp36-cp36m-manylinux2010_x86_64.whl.d6aa0a8a68a34b01a4e61c8ff709d6d2/tensorflow_core already exists. Specify --upgrade to force replacement.
Traceback (most recent call last):
File "/tmp/tmpllc_n8ca/.bootstrap/pex/pex.py", line 393, in execute
exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)
File "/tmp/tmpllc_n8ca/.bootstrap/pex/pex.py", line 325, in _wrap_coverage
return runner(*args)
File "/tmp/tmpllc_n8ca/.bootstrap/pex/pex.py", line 356, in _wrap_profiling
return runner(*args)
File "/tmp/tmpllc_n8ca/.bootstrap/pex/pex.py", line 441, in _execute
return self.execute_interpreter()
File "/tmp/tmpllc_n8ca/.bootstrap/pex/pex.py", line 479, in execute_interpreter
self.execute_content('-c <cmd>', content, argv0='-c')
File "/tmp/tmpllc_n8ca/.bootstrap/pex/pex.py", line 530, in execute_content
exec_function(ast, globals_map)
File "/tmp/tmpllc_n8ca/.bootstrap/pex/compatibility.py", line 77, in exec_function
exec(ast, globals_map, locals_map)
File "-c <cmd>", line 1, in <module>
File "/tmp/tmpllc_n8ca/.deps/tensorflow-1.15.2-cp36-cp36m-manylinux2010_x86_64.whl/tensorflow/__init__.py", line 102, in <module>
from tensorflow.python.util import deprecation_wrapper as _deprecation
File "/tmp/tmpllc_n8ca/.deps/tensorflow-1.15.2-cp36-cp36m-manylinux2010_x86_64.whl/tensorflow/__init__.py", line 50, in __getattr__
module = self._load()
File "/tmp/tmpllc_n8ca/.deps/tensorflow-1.15.2-cp36-cp36m-manylinux2010_x86_64.whl/tensorflow/__init__.py", line 44, in _load
module = _importlib.import_module(self.__name__)
File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'tensorflow_core.python'Things are still bad (but differently) in Pex 2.1.132 but are fixed in Pex 2.1.133: # Bad, differently:
:; docker run --rm -it pex-issues-913 bash -c 'python3.6 -mvenv pex.venv && source pex.venv/bin/activate && pip -q install pex==2.1.132 && pex --version && pex --disable-cache tensorflow==1.15.2 -- -
c "import tensorflow; print(tensorflow.__file__)"'
2.1.132
pid 36 -> /tmp/tmpw32mmgd9/venvs/9089b13598b90e0f8047ceb38dcae8c26b52ec97/2e582d52f642776ae1b90ff40fa6891e2f01b326/bin/python -sE /tmp/tmpw32mmgd9/venvs/9089b13598b90e0f8047ceb38dcae8c26b52ec97/2e582d52f642776ae1b90ff40fa6891e2f01b326/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /tmp/tmpw32mmgd9/pip_cache download --dest /tmp/tmpw32mmgd9/downloads/resolver_download.cejdcn2v/pex.venv.bin.python3.6 tensorflow==1.15.2 --index-url https://pypi.org/simple --retries 5 --timeout 15 exited with 1 and STDERR:
Re-execing from /tmp/tmpw32mmgd9/venvs/9089b13598b90e0f8047ceb38dcae8c26b52ec97/2e582d52f642776ae1b90ff40fa6891e2f01b326/bin/python
Traceback (most recent call last):
File "/tmp/tmpw32mmgd9/venvs/9089b13598b90e0f8047ceb38dcae8c26b52ec97/2e582d52f642776ae1b90ff40fa6891e2f01b326/pex", line 50, in <module>
os.execv(python, argv)
FileNotFoundError: [Errno 2] No such file or directory
# Good again:
:; docker run --rm -it pex-issues-913 bash -c 'python3.6 -mvenv pex.venv && source pex.venv/bin/activate && pip -q install pex==2.1.133 && pex --version && pex --disable-cache tensorflow==1.15.2 -- -
c "import tensorflow; print(tensorflow.__file__)"'
2.1.133
/tmp/tmpmphq3gyr/installed_wheels/34551b8cc446342dac8fdcf79e775e0841aeea38e927b73e2ecf4b2d01277264/tensorflow-1.15.2-cp36-cp36m-manylinux2010_x86_64.whl/tensorflow/__init__.pyAnd things remain good to this day: :; docker run --rm -it pex-issues-913 bash -c 'python3.6 -mvenv pex.venv && source pex.venv/bin/activate && pip -q install pex && pex --disable-cache tensorflow==1.15.2 -- -c "import tensorflow; prin
t(tensorflow.__file__)"'
/tmp/tmp50v576j2/installed_wheels/34551b8cc446342dac8fdcf79e775e0841aeea38e927b73e2ecf4b2d01277264/tensorflow-1.15.2-cp36-cp36m-manylinux2010_x86_64.whl/tensorflow/__init__.pyIt looks like the fix came in #1867. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We have an error with tensorflow now that seems to be linked to the latest pex release 2.1.5.
Same protocol works with pex==2.1.4
Note that there is also a new warning that is displayed (my /tmp is all empty before an after the run)
The text was updated successfully, but these errors were encountered: