Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyCall not found error #515

Open
abianco88 opened this issue Jan 16, 2020 · 15 comments
Open

PyCall not found error #515

abianco88 opened this issue Jan 16, 2020 · 15 comments

Comments

@abianco88
Copy link

After a fresh installation of Julia on CentOS 7.2, I added TensorFlow, ran the "basic usage" test in README.md and passed it. Then, after installing also the Distributions and Printf packages, I tried to run the "logistic regression example" in the same README and, when I executed minimize_op = train.minimize(optimizer, Loss), I got an error claiming PyCall is not found in current path (see snippet below).

ERROR: On worker 2:
LoadError: ArgumentError: Package PyCall not found in current path:
- Run `import Pkg; Pkg.add("PyCall")` to install the PyCall package.

require at ./loading.jl:823
include at ./boot.jl:317 [inlined]
include_relative at ./loading.jl:1044
include at ./sysimg.jl:29
include at ./client.jl:392
#5 at /home/adude/.julia/packages/TensorFlow/q9pY2/src/TensorFlow.jl:160
#116 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:276
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:56
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:65
#102 at ./task.jl:259
in expression starting at /home/adude/.julia/packages/TensorFlow/q9pY2/src/py.jl:1
Stacktrace:
 [1] #remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:421
 [2] remotecall_wait(::Function, ::Distributed.Worker, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:412
 [3] #remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:433
 [4] remotecall_wait(::Function, ::Int64, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:433
 [5] top-level scope at /home/adude/.julia/packages/TensorFlow/q9pY2/src/TensorFlow.jl:159
 [6] eval at ./boot.jl:319 [inlined]
 [7] eval at ./sysimg.jl:68 [inlined]
 [8] #load_python_process#1(::Bool, ::Function) at /home/adude/.julia/packages/TensorFlow/q9pY2/src/TensorFlow.jl:156
 [9] load_python_process at /home/adude/.julia/packages/TensorFlow/q9pY2/src/TensorFlow.jl:149 [inlined]
 [10] macro expansion at /home/adude/.julia/packages/TensorFlow/q9pY2/src/TensorFlow.jl:187 [inlined]
 [11] add_gradients_py(::Tensor{Float64}, ::Array{Any,1}, ::Nothing) at /home/adude/.julia/packages/TensorFlow/q9pY2/src/core.jl:1548
 [12] gradients at /home/adude/.julia/packages/TensorFlow/q9pY2/src/core.jl:1536 [inlined] (repeats 2 times)
 [13] compute_gradients(::TensorFlow.train.AdamOptimizer, ::Tensor{Float64}, ::Nothing) at /home/adude/.julia/packages/TensorFlow/q9pY2/src/train.jl:49
 [14] #minimize#1(::Nothing, ::Nothing, ::Nothing, ::Function, ::TensorFlow.train.AdamOptimizer, ::Tensor{Float64}) at /home/adude/.julia/packages/TensorFlow/q9pY2/src/train.jl:41
 [15] minimize(::TensorFlow.train.AdamOptimizer, ::Tensor{Float64}) at /home/adude/.julia/packages/TensorFlow/q9pY2/src/train.jl:38
 [16] top-level scope at none:0

Scratching my head, I subsequently added PyCall and re-ran the logistic regression example. Again I was thrown an error at minimize_op = train.minimize(optimizer, Loss), however this time the error stated that py_gradients is not defined (see folowing snippet). How can it be? Any idea on how to fix it?

ERROR: On worker 2:
UndefVarError: py_gradients not defined
#13 at /home/adude/.julia/packages/TensorFlow/q9pY2/src/TensorFlow.jl:189
#116 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:276
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:56
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/process_messages.jl:65
#102 at ./task.jl:259
Stacktrace:
 [1] #remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:421
 [2] remotecall_wait(::Function, ::Distributed.Worker) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:412
 [3] #remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:433
 [4] remotecall_wait(::Function, ::Int64) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:433
 [5] top-level scope at /home/adude/.julia/packages/TensorFlow/q9pY2/src/TensorFlow.jl:188
 [6] eval at ./boot.jl:319 [inlined]
 [7] eval at ./sysimg.jl:68 [inlined]
 [8] add_gradients_py(::Tensor{Float64}, ::Array{Any,1}, ::Nothing) at /home/adude/.julia/packages/TensorFlow/q9pY2/src/core.jl:1548
 [9] gradients at /home/adude/.julia/packages/TensorFlow/q9pY2/src/core.jl:1536 [inlined] (repeats 2 times)
 [10] compute_gradients(::TensorFlow.train.AdamOptimizer, ::Tensor{Float64}, ::Nothing) at /home/adude/.julia/packages/TensorFlow/q9pY2/src/train.jl:49
 [11] #minimize#1(::Nothing, ::Nothing, ::Nothing, ::Function, ::TensorFlow.train.AdamOptimizer, ::Tensor{Float64}) at /home/adude/.julia/packages/TensorFlow/q9pY2/src/train.jl:41
 [12] minimize(::TensorFlow.train.AdamOptimizer, ::Tensor{Float64}) at /home/adude/.julia/packages/TensorFlow/q9pY2/src/train.jl:38
 [13] top-level scope at none:0

I even tried to kill process 2 (rmprocs(2)), but kept facing an error at the same place:

ERROR: ProcessExitedException()
Stacktrace:
 [1] worker_from_id(::Distributed.ProcessGroup, ::Int64) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/cluster.jl:973
 [2] worker_from_id at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/cluster.jl:970 [inlined]
 [3] #remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:433
 [4] remotecall_wait(::Function, ::Int64) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Distributed/src/remotecall.jl:433
 [5] top-level scope at /home/adude/.julia/packages/TensorFlow/q9pY2/src/TensorFlow.jl:188
 [6] eval at ./boot.jl:319 [inlined]
 [7] eval at ./sysimg.jl:68 [inlined]
 [8] add_gradients_py(::Tensor{Float64}, ::Array{Any,1}, ::Nothing) at /home/adude/.julia/packages/TensorFlow/q9pY2/src/core.jl:1548
 [9] gradients at /home/adude/.julia/packages/TensorFlow/q9pY2/src/core.jl:1536 [inlined] (repeats 2 times)
 [10] compute_gradients(::TensorFlow.train.AdamOptimizer, ::Tensor{Float64}, ::Nothing) at /home/adude/.julia/packages/TensorFlow/q9pY2/src/train.jl:49
 [11] #minimize#1(::Nothing, ::Nothing, ::Nothing, ::Function, ::TensorFlow.train.AdamOptimizer, ::Tensor{Float64}) at /home/adude/.julia/packages/TensorFlow/q9pY2/src/train.jl:41
 [12] minimize(::TensorFlow.train.AdamOptimizer, ::Tensor{Float64}) at /home/adude/.julia/packages/TensorFlow/q9pY2/src/train.jl:38
 [13] top-level scope at none:0

If you are making a bug report, please copy and paste the output of the following Julia snippet into the issue:

using TensorFlow
tf_version()

Wording: Please copy-paste the entirely of the below output into any bug reports.
Note that this may display some errors, depending upon on your configuration. This is fine.

----------------
Library Versions
----------------
Trying to evaluate ENV["TF_USE_GPU"] but got error: KeyError("TF_USE_GPU")
Trying to evaluate ENV["LIBTENSORFLOW"] but got error: KeyError("LIBTENSORFLOW")

tf_version(kind=:backend) = 1.12.0
Trying to evaluate tf_version(kind=:python) but got error: RemoteException(2, CapturedException(UndefVarError(:py_tf), Any[(#9 at TensorFlow.jl:189, 1), (#116 at process_messages.jl:276, 1), (run_work_thunk at process_messages.jl:56, 1), (run_work_thunk at process_messages.jl:65, 1), (#102 at task.jl:259, 1)]))
tf_version(kind=:julia) = 0.11.0

-------------
Python Status
-------------
PyCall.conda = false
Trying to evaluate ENV["PYTHON"] but got error: KeyError("PYTHON")
PyCall.PYTHONHOME = /home/adude/anaconda3:/home/adude/anaconda3
String(read(#= /home/adude/.julia/packages/TensorFlow/q9pY2/src/version.jl:104 =# @cmd("pip --version"))) = pip 19.3.1 from /home/adude/anaconda3/lib/python3.6/site-packages/pip (python 3.6)

Trying to evaluate String(read(#= /home/adude/.julia/packages/TensorFlow/q9pY2/src/version.jl:105 =# @cmd("pip3 --version"))) but got error: Base.IOError("could not spawn `pip3 --version`: no such file or directory (ENOENT)", -2)

------------
Julia Status
------------
Julia Version 1.0.5
Commit 3af96bcefc (2019-09-09 19:06 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, haswell)

@oxinabox oxinabox changed the title Logistic regression example erors with train.minimize PyCall not found error Jan 18, 2020
@oxinabox
Copy link
Collaborator

Thus should have been fixed by #493

@oxinabox
Copy link
Collaborator

Ah, you are using v0.11.0
And it isn't fixed until v0.12.

Ah for some reason v0.12 hasn't been registered.
https://github.com/JuliaRegistries/General/blob/master/T/TensorFlow/Versions.toml

For now do ]add TensorFlow#master

@oxinabox oxinabox mentioned this issue Jan 18, 2020
4 tasks
@abianco88
Copy link
Author

abianco88 commented Jan 18, 2020 via email

@oxinabox
Copy link
Collaborator

The eager mode branch is a little bit behind master.
It uses an older version of LibTensorFlow

@abianco88
Copy link
Author

I re-built the system with Julia 1.0.5 and installed TensorFlow#master. I noted that both TensorFlow#master and TensorFlow#eager_mode spit out the following when I execute tf_versioninfo(), so I guess they're both using the same front- and back-end version of TensorFlow, right?

tf_version(kind=:backend) = 1.13.1
tf_version(kind=:python) = 1.13.1
tf_version(kind=:julia) = 0.12.0

Anyway, even with TensorFlow#master, I can run logistic.jl and keras.jl but none of the other scripts in the examples directory. What I don't quite understand is why several other examples (like ae.jl and mnist_*.jl) ask for installing the package MNIST, but when I try installing it, I get the message:

ERROR: The following package names could not be resolved:
 * MNIST (not found in project, manifest or registry)
Please specify by known `name=uuid`.

Should I open a different issue ticket for this?
Since installing either TensorFlow#master or TensorFlow#eager_mode on Julia 1.0.5 allows to successfully run the example logistic.jl without PyCall errors, I feel like this issue has been resolved, right?

@oxinabox
Copy link
Collaborator

Anyway, even with TensorFlow#master, I can run logistic.jl and keras.jl but none of the other scripts in the examples directory. What I don't quite understand is why several other examples (like ae.jl and mnist_*.jl) ask for installing the package MNIST, but when I try installing it, I get the message:

yes, open another issue.
Core of it is the MNIST package doesn't exist anymore and has been replaced with MLDatasets but the examples and docs need to be updated.
You could skip straight to the PR.

@abianco88
Copy link
Author

abianco88 commented Jan 19, 2020 via email

@oxinabox
Copy link
Collaborator

PR = Pull Request.

It's how you can suggest a change to this repository.
The literal meaning is to Request we Pull the changes from your branch.

https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests

@abianco88
Copy link
Author

abianco88 commented Jan 20, 2020 via email

@oxinabox
Copy link
Collaborator

Just to clarify, how do I see that LibTensorFlow is different between TensorFlow#master and TensorFlow#eager_mode?

tf_version(kind=:backend)


It might be that I misread the commit history and they are the same.
But TensorFlow#eager_mode is an old branch that has since been merged into master.
Here is the PR: #491

@ndgnuh
Copy link

ndgnuh commented Feb 25, 2020

The tutorial fails again. Now train.GradientDescentOptimizer(0.0001) throw UndefVarError: constant not defined

Stacktrace:
 [1] #GradientDescentOptimizer#4(::String, ::Type{TensorFlow.train.GradientDescentOptimizer}, ::Float64) at /home/hung/.julia/packages/TensorFlow/JljDB/src/train.jl:67
 [2] TensorFlow.train.GradientDescentOptimizer(::Float64) at /home/hung/.julia/packages/TensorFlow/JljDB/src/train.jl:67
 [3] top-level scope at REPL[20]:1

Edit: I was using #master since the current version gave me py_gradient error.
Edit 2: Ok, someone forgot to import constant, I added constant to the import list, now another error was thrown:

ERROR: On worker 2:
KeyError: key "VERSION" not found
getproperty at /home/hung/.julia/packages/PyCall/kAhnQ/src/PyCall.jl:301
getproperty at /home/hung/.julia/packages/PyCall/kAhnQ/src/PyCall.jl:306 [inlined]
#7 at /home/hung/.julia/dev/TensorFlow/src/TensorFlow.jl:202
#112 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:309
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:79
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:88
#98 at ./task.jl:333
Stacktrace:
 [1] #remotecall_wait#150(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_wait), ::Function, ::Distributed.Worker) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:432
 [2] remotecall_wait(::Function, ::Distributed.Worker) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:423
 [3] #remotecall_wait#153(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_wait), ::Function, ::Int64) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:444
 [4] remotecall_wait(::Function, ::Int64) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:444
 [5] top-level scope at /home/hung/.julia/dev/TensorFlow/src/TensorFlow.jl:201
 [6] eval at ./boot.jl:330 [inlined]
 [7] eval at ./Base.jl:70 [inlined]
 [8] #tf_version#82(::Symbol, ::typeof(tf_version)) at /home/hung/.julia/dev/TensorFlow/src/version.jl:20
 [9] (::TensorFlow.var"#kw##tf_version")(::NamedTuple{(:kind,),Tuple{Symbol}}, ::typeof(tf_version)) at ./none:0
 [10] #py_version_check#83(::Bool, ::Bool, ::typeof(TensorFlow.py_version_check)) at /home/hung/.julia/dev/TensorFlow/src/version.jl:47
 [11] py_version_check() at /home/hung/.julia/dev/TensorFlow/src/version.jl:47
 [12] #load_python_process#1(::Bool, ::typeof(TensorFlow.load_python_process)) at /home/hung/.julia/dev/TensorFlow/src/TensorFlow.jl:179
 [13] load_python_process at /home/hung/.julia/dev/TensorFlow/src/TensorFlow.jl:161 [inlined]
 [14] macro expansion at /home/hung/.julia/dev/TensorFlow/src/TensorFlow.jl:200 [inlined]
 [15] add_gradients_py(::Tensor{Float32}, ::Array{Any,1}, ::Nothing) at /home/hung/.julia/dev/TensorFlow/src/core.jl:1553
 [16] gradients at /home/hung/.julia/dev/TensorFlow/src/core.jl:1541 [inlined] (repeats 2 times)
 [17] compute_gradients(::TensorFlow.train.GradientDescentOptimizer, ::Tensor{Float32}, ::Nothing) at /home/hung/.julia/dev/TensorFlow/src/train.jl:49
 [18] #minimize#1(::Nothing, ::Nothing, ::Nothing, ::typeof(TensorFlow.train.minimize), ::TensorFlow.train.GradientDescentOptimizer, ::Tensor{Float32}) at /home/hung/.julia/dev/TensorFlow/src/train.jl:41
 [19] minimize(::TensorFlow.train.GradientDescentOptimizer, ::Tensor{Float32}) at /home/hung/.julia/dev/TensorFlow/src/train.jl:38

This one is from the minimize. Not GradientDescentOptimizer. My guess is that VERSION is a ENV key, so I just assign ENV["VERSION"] = tf_version(). That error is gone, but then another one appears:

ERROR: On worker 2:
Python error: KeyError("import_meta_graph")
error at ./error.jl:33
#3 at /home/hung/.julia/dev/TensorFlow/src/py.jl:45
py_with at /home/hung/.julia/dev/TensorFlow/src/py.jl:20
make_py_graph at /home/hung/.julia/dev/TensorFlow/src/py.jl:52
py_gradients at /home/hung/.julia/dev/TensorFlow/src/py.jl:73
#9 at /home/hung/.julia/dev/TensorFlow/src/TensorFlow.jl:202
#112 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:309
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:79
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:88
#98 at ./task.jl:333
Stacktrace:
 [1] #remotecall_wait#150(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_wait), ::Function, ::Distributed.Worker) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:432
 [2] remotecall_wait(::Function, ::Distributed.Worker) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:423
 [3] #remotecall_wait#153(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(remotecall_wait), ::Function, ::Int64) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:444
 [4] remotecall_wait(::Function, ::Int64) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/remotecall.jl:444
 [5] top-level scope at /home/hung/.julia/dev/TensorFlow/src/TensorFlow.jl:201
 [6] eval at ./boot.jl:330 [inlined]
 [7] eval at ./Base.jl:70 [inlined]
 [8] add_gradients_py(::Tensor{Float32}, ::Array{Any,1}, ::Nothing) at /home/hung/.julia/dev/TensorFlow/src/core.jl:1553
 [9] gradients at /home/hung/.julia/dev/TensorFlow/src/core.jl:1541 [inlined] (repeats 2 times)
 [10] compute_gradients(::TensorFlow.train.GradientDescentOptimizer, ::Tensor{Float32}, ::Nothing) at /home/hung/.julia/dev/TensorFlow/src/train.jl:49
 [11] #minimize#1(::Nothing, ::Nothing, ::Nothing, ::typeof(TensorFlow.train.minimize), ::TensorFlow.train.GradientDescentOptimizer, ::Tensor{Float32}) at /home/hung/.julia/dev/TensorFlow/src/train.jl:41
 [12] minimize(::TensorFlow.train.GradientDescentOptimizer, ::Tensor{Float32}) at /home/hung/.julia/dev/TensorFlow/src/train.jl:38

@abianco88
Copy link
Author

Does it also fail when TensorFlow#master is installed?

@ndgnuh
Copy link

ndgnuh commented Feb 28, 2020

What do you mean? I was using master.

@abianco88
Copy link
Author

Just asking... I was able to make it work simply removing the package and re-installing it as Lyndon suggested.

@ndgnuh
Copy link

ndgnuh commented Mar 1, 2020

Nope, didn't work, sorry. Which commit did you use?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants