bazel run py_binary loads incorrect module with same name

[drigz]

py_binary can end up importing a file from the source tree that it doesn't depend on. It can also get confused by stale .pyc files that Python writes into the source tree (is that expected behaviour?), even once this file has been removed.

My example may seem contrived, but I ran into this problem when trying to port the rosmaster script to Python - since Bazel rejects Python files without the .py extension, I renamed it to rosmaster.py. This caused it to import itself and fail. After I renamed it to avoid this, I had to manually delete the .pyc file before bazel run would work.

Repro

> echo > WORKSPACE 
> cat lib/hello.py 
def hello():
  print("hello, world!")
> cat bin/hello2.py 
import hello

if __name__ == "__main__":
  print "loaded hello from", hello.__file__
  hello.hello()
> cp bin/hello2.py bin/hello.py  
> cat BUILD.bazel 
py_library(
    name = "hello_lib",
    srcs = ["lib/hello.py"],
    imports = ["lib"],
)

py_binary(
    name = "hello2",
    srcs = ["bin/hello2.py"],
    deps = [":hello_lib"],
)
> bazel run :hello2
[SNIP: build output]
INFO: Running command line: bazel-bin/hello2
loaded hello from /usr/local/google/home/rodrigoq/git/bazeltest/stale_pyc/bin/hello.py
Traceback (most recent call last):
  File "/usr/local/google/home/rodrigoq/.cache/bazel/_bazel_rodrigoq/64c22e296c4c5490bc13ab2111abf684/execroot/__main__/bazel-out/local-fastbuild/bin/hello2.runfiles/__main__/bin/hello2.py", line 5, in <module>
    hello.hello()
TypeError: 'module' object is not callable
ERROR: Non-zero return code '1' from command: Process exited with status 1
> rm bin/hello.py 
> bazel run :hello2
[SNIP: build output]
INFO: Running command line: bazel-bin/hello2
loaded hello from /usr/local/google/home/rodrigoq/git/bazeltest/stale_pyc/bin/hello.pyc
Traceback (most recent call last):
  File "/usr/local/google/home/rodrigoq/.cache/bazel/_bazel_rodrigoq/64c22e296c4c5490bc13ab2111abf684/execroot/__main__/bazel-out/local-fastbuild/bin/hello2.runfiles/__main__/bin/hello2.py", line 5, in <module>
    hello.hello()
TypeError: 'module' object is not callable
ERROR: Non-zero return code '1' from command: Process exited with status 1
> rm bin/hello.pyc
> bazel run :hello2
[SNIP: build output]
INFO: Running command line: bazel-bin/hello2
loaded hello from /usr/local/google/home/rodrigoq/.cache/bazel/_bazel_rodrigoq/64c22e296c4c5490bc13ab2111abf684/execroot/__main__/bazel-out/local-fastbuild/bin/hello2.runfiles/__main__/lib/hello.pyc
hello, world!

Environment info

• Operating System: Ubuntu 14.04
• Bazel version: 0.7.0

[scottcjt]

Same here. Minimized reproducible sample can be found below:

https://github.com/scottcjt/bazelbug

Bazel generates the directory structure <runfiles>/flatbuffers/python/flatbuffers for the reference to @flatbuffers/:runtime_py. Then it assigns two Python paths:

• <runfiles>
• <runfiles>/flatbuffers/python

The second one is what we expected, but Python will always take the first one and throw ImportError.

Environment: OSX 10.13.2 with bazel-0.9.0-homebrew.

[drigz]

@scottcjt If I'm understanding right, I've run into that issue too (Python confusing the external repo with the module directory). We use one of two workarounds:

• for simple modules (with just one file, eg mock), use a genrule to rename mock.py to __init__.py in the repository root, so that Python can find everything in the repo root
• otherwise, make sure the repo name doesn't match the module name (eg use @flatbuffers_repo//:runtime_py)

Hope that helps...

[scottcjt]

@drigz Yes, they do the trick. Thanks for the advice.

[brandjon]

So there are two separate issues discussed here:

1. The source file's original path in the source root gets put on PYTHONPATH. This enables relative imports of other modules from the source root, even when they're not declared as dependencies in the runfiles tree. It also leads to .pyc files polluting the source root.

2. The names of repos are conflated with the names of packages.

Issue 1 is covered by #7091, and 2 is covered (at least in part) by #7067.

[keith]

I wonder if this is the same issue as #7754 ?