Importing local module fails

[mlubin]

pyimport doesn't seem to be able to find python modules that are present in the local directory. Any reason for this? What's the best way to import user modules?

[stevengj]

python prepends '' to sys.path, whereas initializing libpython does not, by default.

More technically, we are passing 0 for the updatepath parameter of PySys_SetArgvEx as recommended in the Python library (see here) and as recommended for security reasons.

I'm not sure of the best option here. One option would be to just prepend '' to sys.path automatically all of the time (similar to the python executable?). Another would be to only do this in interactive mode, somehow. Or....?

@loladiro, how does Julia decide whether to import modules from the current directory?

[mlubin]

Maybe a custom option to pyinitialize? This should be available from both interactive and noninteractive modes, as python users would expect.

[Keno]

@stevengj Not sure what you mean. If you're talking about include it depends the most recently included file in the current task (the directory of that, or, if none cwd).

[stevengj]

@loladiro, if you type using Foo at the Julia prompt, it will load the module from a Foo.jl file in the current directory if it exists. This seems like the equivalent of having '' in the Python path. I'm not sure where this behavior is specified in the Julia source...

Furthermore, it looks like Julia has a similar security issue to Python... it looks like the current directory is searched before the other standard package directories.

[Keno]

@stevengj I think it just does require which falls back to include which uses the mechanism mentioned above.

[stevengj]

@loladiro, see my amended comment above... it looks like we may have the same security issue as in Python after all. e.g. if I add a PyPlot.jl to my current directory and do using PyPlot, Julia gets confused.

Or maybe because it is task-dependent this is okay?

[Keno]

It think this is different because using PyPlot in a package will search package directory for a PyPlot.jl. cwd is only applicable at the prompt (I guess we could say the source directory of the prompt is implicitly cwd). I think the issue in python is that the search path gets set globally.

[stevengj]

@mlubin, regarding adding an option to pyinitialize, I don't see why this is better than just telling people to modify sys.path, e.g.

unshift!(PyVector(pyimport("sys")["path"]), "")

although one might provide a convenience function for this. (e.g. I could just define PyCall.path as a PyVector mirror of sys.path.) (Note that this only works with the latest version of PyCall, which adds unshift! to PyVectors.)

However, I'd like to do something sensible by default. If Python users are used to being able to load things in the current directory in python, they should be able to do that in PyCall without manually modifying the path. The question is, how can I do this in a way that is reasonably secure?

[stevengj]

I could mimic Julia's behavior: every pyimport call could modify sys.path based on the (Julia) directory of the current task. This seems reasonable to me, but I'm not sure whether it might have unexpected consequences in Python.

[malmaud]

What if we called Python's os.chdir whenever Julia's cd was called (as well as in pyinitialize)?
EDIT: NM, it's necessary and sufficient to modify sys.path.

[stevengj]

How would we modify Julia's cd?

[malmaud]

I think I'm not understanding the security issue with always prepending sys.path with "" in pyinitialize. If I execute a file test.jl with python like

import numpy
...

and a file called numpy.py is in the same directory as test.py, then that local numpy.py will be imported. PyCall would then have the same behavior. Is that what you're trying to avoid?

[stevengj]

It's not where test.jl is located that is the problem. What we want to avoid is having the current directory in the default path, independent of the location of the file that is being executed. You don't want unexpected behavior to result just from executing Julia from the wrong path.

[malmaud]

It seems the path that julia is run in is independent of the search path of python modules (or I'm still not quite getting the issue):

~/a/test.jl:

using PyCall
println(pwd())
unshift!(PyVector(pyimport("sys")["path"]), "")
@pyimport numpy

~/numpy.py:

print "numpy hijacked"

Then in the shell,

cd ~
julia a/test.jl

loads the numpy package, not my rogue version.

[malmaud]

What do you think, @stevengj?

[stevengj]

You're right, I can't get it to override the numpy module, but maybe I'm missing something in how Python decides what modules to load. See also the Python bug discussion.

[gerrat]

Github should have a system to just say: "+1" for a request.
I'd just like to add a: +1 for searching the local directory first.
I just started looking at Julia (I mostly use Python), and so pretty much the first thing I did was trying to import a module in the local directory, which failed.

PyCall seems like a brilliant stroke of genius by the way. Thanks!

[dhoegh]

👍 for this. If any one needs a workaround the following can be done

@pyimport importlib.machinery as machinery
loader = machinery.SourceFileLoader("module.name","/abs/path/to/file.py")
my_mod = loader[:load_module]("module.name")
my_mod[:myfunc]()

This loads python files by path. I got the idea from: http://stackoverflow.com/questions/67631/how-to-import-a-module-given-the-full-path

[sherifnada]

@dhoegh that solution doesn't work for me. I get the following error:

julia>@pyimport importlib.machinery as machinery
ERROR: PyError (:PyImport_ImportModule) <type 'exceptions.ImportError'>
ImportError('No module named machinery',)

in pyerr_check at /home/shrif/.julia/v0.3/PyCall/src/exception.jl:61
in pyimport at /home/shrif/.julia/v0.3/PyCall/src/PyCall.jl:81

Any idea if there a machinery equivalent I can use?

[dhoegh]

What python version are you using? It depends on the python version. I have a version that works for python 2.7 at: https://github.com/dhoegh/Hawk.jl/blob/master/src/Hawk.jl#L13 alternative see: http://stackoverflow.com/questions/19009932/import-abitrary-python-source-file-python-3-3.

[sherifnada]

I am on 2.7 . After checking now, it seems that python2.7 also has problems importing machinery. Not quite sure why since it's included in site-packages under importlib .

[dhoegh]

Did you try the import I used in https://github.com/dhoegh/Hawk.jl/blob/master/src/Hawk.jl#L7-13

[sherifnada]

Yes -- it works just fine, thanks!

However, now I have a new problem. When the local module imports other local modules, the import statement fails.

import-test.jl :

using PyCall

@pyimport imp

filename = abspath(joinpath(dirname(@__FILE__),"callmefirst.py"))
(path,name) = dirname(filename), basename(filename)
(name,ext) = rsplit(name, '.', 2)
(file, filename, data) = imp.find_module(name,[path])
module = imp.load_module(name,file,filename,data)
print("IMPORTED PYTHON")

callmefirst.py:

print "Importing dependencies"
import callme

callme.py:

print "NESTED MODULE SUCCESSFULLY IMPORTED"

running julia import-test.jl returns

Importing dependencies
ERROR: PyError (:PyObject_Call) <type 'exceptions.ImportError'>
ImportError('No module named callme',)

[sherifnada]

Update: This is fixed when I update callmefirst.py to the following:

import imp
foo = imp.load_source('callme', '/home/shrif/Hyperloop/src/hyperloop/callme.py')
print(foo)

However, this is pretty inconvenient since I'd need to change all my import statements (especially if I'm importing a python framework which relies on calling local modules as such). Any advice on how to get around this?

[stevengj]

@sherifnada, does it work to just the current directory to the path, via unshift!(PyVector(pyimport("sys")["path"]), "")?

[sherifnada]

Worked like a charm. Thanks!!

[xanderdunn]

@dhoegh Thanks for your workaround.

This feature would be great for PyCall.

[rofinn]

Is there any easy way to do the same thing as unshift!(PyVector(pyimport("sys")["path"]), ""), but looking in the Conda.jl package installation directory? It'd be nice if PyCall could find packages installed with Conda.jl even if users aren't running the conda python binary.

[stevengj]

@rofinn, that's a separate issue. You can certainly unshift! any directory you want, in principle. But it is usually not a good idea to "mix" python distros, i.e. to include in your python path the packages from two separate distros. If you want to use the Conda.jl packages in PyCall, you should configure PyCall to use Conda's python.

[KwatMDPhD]

py"""
import sys
sys.path.insert(0, "./directory/path/containing/module")
"""

function_name = pyimport("module_name")["function_name"]