Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"cannot pickle 'module' object" from PyCall #50

Closed
KronosTheLate opened this issue Jun 14, 2023 · 10 comments
Closed

"cannot pickle 'module' object" from PyCall #50

KronosTheLate opened this issue Jun 14, 2023 · 10 comments

Comments

@KronosTheLate
Copy link
Contributor

I realize that it is entirely possible that this is a problem only with PyCall, but I will raise it here, since it only occurs when I use both packages, and there is not error when only using PyCall.

I can use PyImport when I run julia directly from an SSH session:

julia> using PyCall

julia> pyimport("piplates.DAQC2plate")
PyObject <module 'piplates.DAQC2plate' from '/home/pi/.local/lib/python3.9/site-packages/piplates/DAQC2plate.py'>

However, when I do this through RemoteREPL, the follwing happens:

julia> connect_remote("pi@192.168.4.2")
RemoteREPL.Connection("pi@192.168.4.2", 27754, :ssh, ``, nothing, nothing, Sockets.TCPSocket(RawFD(29) paused, 0 bytes waiting), :Main)

julia> @remote using PyCall

julia> @remote pyimport("piplates.DAQC2plate")
[ Info: Connection dropped, attempting reconnect
┌ Error: Network or internal error running remote repl
│   exception =
│    KeyError: key PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] not found
│    Stacktrace:
│      [1] getindex
│        @ ./dict.jl:482 [inlined]
│      [2] root_module
│        @ ./loading.jl:979 [inlined]
│      [3] deserialize_module(s::Serialization.Serializer{Sockets.TCPSocket})
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:962
│      [4] handle_deserialize(s::Serialization.Serializer{Sockets.TCPSocket}, b::Int32)
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:864
│      [5] deserialize(s::Serialization.Serializer{Sockets.TCPSocket})
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:782
│      [6] deserialize_datatype(s::Serialization.Serializer{Sockets.TCPSocket}, full::Bool)
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:1287
│      [7] handle_deserialize(s::Serialization.Serializer{Sockets.TCPSocket}, b::Int32)
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:835
│      [8] deserialize(s::Serialization.Serializer{Sockets.TCPSocket})
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:782
│      [9] handle_deserialize(s::Serialization.Serializer{Sockets.TCPSocket}, b::Int32)
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:842
│     [10] deserialize(s::Serialization.Serializer{Sockets.TCPSocket})
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:782
│     [11] (::Serialization.var"#5#6"{Serialization.Serializer{Sockets.TCPSocket}})(i::Int64)
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:941
│     [12] ntupleany(f::Serialization.var"#5#6"{Serialization.Serializer{Sockets.TCPSocket}}, n::Int64)
│        @ Base ./ntuple.jl:43
│     [13] deserialize_tuple(s::Serialization.Serializer{Sockets.TCPSocket}, len::Int64)
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:941
│     [14] handle_deserialize(s::Serialization.Serializer{Sockets.TCPSocket}, b::Int32)
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:825
│     [15] deserialize(s::Serialization.Serializer{Sockets.TCPSocket})
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:782
│     [16] (::Serialization.var"#5#6"{Serialization.Serializer{Sockets.TCPSocket}})(i::Int64)
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:941
│     [17] ntupleany(f::Serialization.var"#5#6"{Serialization.Serializer{Sockets.TCPSocket}}, n::Int64)
│        @ Base ./ntuple.jl:43
│     [18] deserialize_tuple(s::Serialization.Serializer{Sockets.TCPSocket}, len::Int64)
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:941
│     [19] handle_deserialize(s::Serialization.Serializer{Sockets.TCPSocket}, b::Int32)
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:825
│     [20] deserialize(s::Serialization.Serializer{Sockets.TCPSocket})
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:782
│     [21] handle_deserialize(s::Serialization.Serializer{Sockets.TCPSocket}, b::Int32)
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:888
│     [22] deserialize
│        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:782 [inlined]
│     [23] deserialize(s::Sockets.TCPSocket)
│        @ Serialization /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:769
│     [24] send_and_receive(conn::RemoteREPL.Connection, request::Tuple{Symbol, Expr}; read_response::Bool)
│        @ RemoteREPL ~/.julia/packages/RemoteREPL/BFqrB/src/client.jl:207
│     [25] send_and_receive
│        @ ~/.julia/packages/RemoteREPL/BFqrB/src/client.jl:199 [inlined]
│     [26] (::RemoteREPL.var"#47#48"{RemoteREPL.Connection, Expr})()
│        @ RemoteREPL ~/.julia/packages/RemoteREPL/BFqrB/src/client.jl:382
│     [27] ensure_connected!(f::RemoteREPL.var"#47#48"{RemoteREPL.Connection, Expr}, conn::RemoteREPL.Connection; retries::Int64)
│        @ RemoteREPL ~/.julia/packages/RemoteREPL/BFqrB/src/client.jl:178
│     [28] ensure_connected!
│        @ ~/.julia/packages/RemoteREPL/BFqrB/src/client.jl:174 [inlined]
│     [29] remote_eval_and_fetch(conn::RemoteREPL.Connection, ex::Expr)
│        @ RemoteREPL ~/.julia/packages/RemoteREPL/BFqrB/src/client.jl:380
│     [30] top-level scope
│        @ REPL[25]:1
│     [31] eval
│        @ ./boot.jl:360 [inlined]
│     [32] eval_user_input(ast::Any, backend::REPL.REPLBackend)
│        @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:139
│     [33] repl_backend_loop(backend::REPL.REPLBackend)
│        @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:200
│     [34] start_repl_backend(backend::REPL.REPLBackend, consumer::Any)
│        @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:185
│     [35] run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool)
│        @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:317
│     [36] run_repl(repl::REPL.AbstractREPL, consumer::Any)
│        @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:305
│     [37] (::Base.var"#881#883"{Bool, Bool, Bool})(REPL::Module)
│        @ Base ./client.jl:387
│     [38] #invokelatest#2
│        @ ./essentials.jl:708 [inlined]
│     [39] invokelatest
│        @ ./essentials.jl:706 [inlined]
│     [40] run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool)
│        @ Base ./client.jl:372
│     [41] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:302
│     [42] _start()
│        @ Base ./client.jl:485
└ @ RemoteREPL ~/.julia/packages/RemoteREPL/BFqrB/src/client.jl:190
Error on remote (Pi) side:
┌ Error: RemoteREPL responder crashed
│   exception =
│    PyError ($(Expr(:escape, :(ccall(#= /home/pi/.julia/packages/PyCall/twYvK/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'TypeError'>
│    TypeError("cannot pickle 'module' object")
│
│    Stacktrace:
│      [1] pyerr_check
│        @ ~/.julia/packages/PyCall/twYvK/src/exception.jl:75 [inlined]
│      [2] pyerr_check
│        @ ~/.julia/packages/PyCall/twYvK/src/exception.jl:79 [inlined]
│      [3] _handle_error(msg::String)
│        @ PyCall ~/.julia/packages/PyCall/twYvK/src/exception.jl:96
│      [4] macro expansion
│        @ ~/.julia/packages/PyCall/twYvK/src/exception.jl:110 [inlined]
│      [5] #107
│        @ ~/.julia/packages/PyCall/twYvK/src/pyfncall.jl:43 [inlined]
│      [6] disable_sigint
│        @ ./c.jl:458 [inlined]
│      [7] __pycall!
│        @ ~/.julia/packages/PyCall/twYvK/src/pyfncall.jl:42 [inlined]
│      [8] _pycall!(ret::PyObject, o::PyObject, args::Tuple{PyObject}, nargs::Int32, kw::Ptr{Nothing})
│        @ PyCall ~/.julia/packages/PyCall/twYvK/src/pyfncall.jl:29
│      [9] _pycall!
│        @ ~/.julia/packages/PyCall/twYvK/src/pyfncall.jl:11 [inlined]
│     [10] #pycall#112
│        @ ~/.julia/packages/PyCall/twYvK/src/pyfncall.jl:80 [inlined]
│     [11] pycall
│        @ ~/.julia/packages/PyCall/twYvK/src/pyfncall.jl:80 [inlined]
│     [12] serialize(s::Serialization.Serializer{Sockets.TCPSocket}, pyo::PyObject)
│        @ PyCall ~/.julia/packages/PyCall/twYvK/src/serialize.jl:14
│     [13] serialize(s::Serialization.Serializer{Sockets.TCPSocket}, t::Tuple{PyObject, String}) (repeats 2 times)
│        @ Serialization /buildworker/worker/package_linuxarmv7l/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:201
│     [14] serialize(s::Sockets.TCPSocket, x::Tuple{Symbol, Tuple{PyObject, String}})
│        @ Serialization /buildworker/worker/package_linuxarmv7l/build/usr/share/julia/stdlib/v1.6/Serialization/src/Serialization.jl:745
│     [15] serialize_responses(socket::Sockets.TCPSocket, response_chan::Channel{Any})
│        @ RemoteREPL ~/.julia/packages/RemoteREPL/BFqrB/src/server.jl:170
│     [16] (::RemoteREPL.var"#24#26"{Sockets.TCPSocket, Channel{Any}})()
│        @ RemoteREPL ./task.jl:417
└ @ RemoteREPL ~/.julia/packages/RemoteREPL/BFqrB/src/server.jl:198

Any idea what the root issue is, and if there are potential workarounds?

@KronosTheLate
Copy link
Contributor Author

This resource (first result upon googling the error) might be relevant:
https://stackoverflow.com/questions/2790828/python-cant-pickle-module-objects-error

@c42f
Copy link
Collaborator

c42f commented Jun 16, 2023

@remote fetches the result of evaluation from the server back to the client. For this to work, you need

  1. typeof(result) to be defined on both the client and server side. Usually this means loading the relevant module defining that type on both the client and server (in this case, PyCall needs to be loaded on the client as well as the server)
  2. The result must be serializable and deserializable with the Serialization module. From the server logs it seems this isn't the case because there's a pickle failure.

In this case I guess you didn't intend to serialize the python module piplates.DAQC2plate, rather you just want to load it on the server side. In that case you can try suppressing the output by just returning nothing instead of returning the module to the client:

julia> @remote (pyimport("piplates.DAQC2plate"); nothing)

So, keep in mind that @remote is for fetching data back to the client as julia data structures. If you just want to see the results of showing the data on the server, you can use the REPL for that.

@KronosTheLate
Copy link
Contributor Author

KronosTheLate commented Jun 17, 2023

That all makes a lot of sense. So I can import the modules, just making sure to return nothing. And then should a) load Py(thon)Call on the client side, or convert to julia types remotely. So something like

function DAQC2get_adc(bit, addr)
    measurement = DAQC2.get_adc(bit, addr)
    return pyconvert(Float64, measurement)
end

Should probably do the trick. Fingers crossed - I will test when I have time.

A section could perhaps be added to the docs, something like "Transferring non-standard types", explaining that the package defining the type should be defined on both sides, and also perhaps a specific example with python interop. I can look into it if I get things working as I want them to ^_^

@KronosTheLate
Copy link
Contributor Author

So with PythonCall loaded on both sides, I am able to transfer python variables. I was however still unable to load the package and return nothing. It causes an error about returning Int64 instead of Int32 (or opposite). When I tried to return 4 I got an error about returning a symbol. However, if I load the python library directly over SSH, in the remote Julia session, I am able to call on it using RemoteREPL. So the fix for me seems to be
Run the following on the Pi:

using RemoteREPL
using PythonCall
DAQC2 = pyimport("piplates.DAQC2plate")
serve_repl()

On the host/client side, now do

using RemoteREPL
using PythonCall
connect_remote()

And from there, I could sucessfully run stuff like pyconvert(Vector, @remote(DAQC2.daqc2sPresent)) to get a julia vector of the present plates. To with that, it seems to be working. It is absolutely not straightforward though, so I feel like a guide would be needed if we want new users to be able to get going without this dance of creating issues and testing fixes over several days.

@KronosTheLate
Copy link
Contributor Author

KronosTheLate commented Jun 19, 2023

The contents of this comment are more accuratly covered in #54

Original post For something completely unrelated to this issue: When I run `@time pyconvert(Float32, DAQC2.getADC(0, 0))` directly over SSH, I get around 1 ms. When I run `@time pyconvert(Float32, @Remote(DAQC2.getADC(0, 0)))` from the client/host, I get around 50 ms. Is it the serialization and deserialization that takes so long? I also tried `@time @Remote(pyconvert(Float32, DAQC2.getADC(0, 0)))` to perform the conversion before the serialization process, but the timings remained the same.

With a 50 ms overhead to each call, real-time applications are limited to an update rate of 20 Hz, which is perhaps worthy of mention somewhere in the readme. At least that it is known that this is a potential bottleneck. Perhaps BSON.jl would show different performance, but hard to tell without testing.

@c42f
Copy link
Collaborator

c42f commented Jun 23, 2023

I was however still unable to load the package and return nothing. It causes an error about returning Int64 instead of Int32 (or opposite).

What was the code you ran which caused this error?

Are you using 32-bit Julia on the pi while using 64-bit on the client computer? That would cause issues as Serialization only works between the same Julia versions on the same architecture (binary layout of data structures needs to be the same on both sides).

@KronosTheLate
Copy link
Contributor Author

I actually do not remember the code I ran. Something like @remote DAQC2 = pyimport("piplates.DAQC2plate"); nothing. But after this errored, any call to @remote, including just a numeric literal, returned the same error. So something broke in the session. I just restarted, defined DAQC2 directly in the Pi session, and went on my merry way.

I believe the Pi 400 is 64-bit, which is supported by the fact that I was able to make it work eventually, right?

@c42f
Copy link
Collaborator

c42f commented Jul 1, 2023

You need parentheses for that to work:

@remote (DAQC2 = pyimport("piplates.DAQC2plate"); nothing)

In any case, I think there's not much we can do about this issue (aside from additional docs, perhaps)

@c42f c42f closed this as completed Jul 1, 2023
@KronosTheLate
Copy link
Contributor Author

Right, of course. Yhea, a docs section about python interop would not hurt ^_^

Btw, awesome work on JuliaSyntax.jl. I almost feel bad for diverting your attention with this stuff.

@c42f
Copy link
Collaborator

c42f commented Jul 2, 2023

Thanks, it's all good! I haven't used this package for a while which is why I've been a bit inattentive to it sorry :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants