Flaky product graph construction: Invalid requirement, parse error at "u''"

[wisechengyi]

https://api.travis-ci.org/v3/job/463150773/log.txt

�[Ktravis_fold:start:BackendTests
�[K�[34m[=== 05:35 Running internal backend python tests ===]�[0m
timestamp: 2018-12-04T04:51:24.356140
Exception caught: (pants.build_graph.address_lookup_error.AddressLookupError)
  File ".bootstrap/_pex/pex.py", line 349, in execute
    exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)
  File ".bootstrap/_pex/pex.py", line 281, in _wrap_coverage
    return runner(*args)
  File ".bootstrap/_pex/pex.py", line 312, in _wrap_profiling
    return runner(*args)
  File ".bootstrap/_pex/pex.py", line 394, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File ".bootstrap/_pex/pex.py", line 506, in execute_entry
    return runner(entry_point)
  File ".bootstrap/_pex/pex.py", line 528, in execute_pkg_resources
    return runner()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/pants_loader.py", line 71, in main
    PantsLoader.run()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/pants_loader.py", line 67, in run
    cls.load_and_execute(entrypoint)
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/pants_loader.py", line 60, in load_and_execute
    entrypoint_main()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/pants_exe.py", line 39, in main
    PantsRunner(exiter, start_time=start_time).run()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/pants_runner.py", line 62, in run
    return runner.run()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/local_pants_runner.py", line 159, in run
    self._run()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/local_pants_runner.py", line 227, in _run
    goal_runner_result = self._maybe_run_v1(run_tracker, reporting)
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/local_pants_runner.py", line 176, in _maybe_run_v1
    return goal_runner_factory.create().run()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/goal_runner.py", line 125, in create
    goals, context = self._setup_context()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/goal_runner.py", line 95, in _setup_context
    self._root_dir
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/init/engine_initializer.py", line 226, in create_build_graph
    for _ in graph.inject_roots_closure(target_roots):
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/engine/legacy/graph.py", line 217, in inject_roots_closure
    for address in self._inject_specs(target_roots.specs):
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/engine/legacy/graph.py", line 270, in _inject_specs
    subjects)
  File "/opt/pyenv/versions/2.7.14/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/engine/legacy/graph.py", line 237, in _resolve_context
    'Build graph construction failed: {} {}'.format(type(e).__name__, str(e))

Exception message: Build graph construction failed: ExecutionError 1 Exception encountered:
Computing Select(Specs(dependencies<=tuple>=(DescendantAddresses(directory=u'pants-plugins/tests/python'),), matcher<=SpecsMatcher>=SpecsMatcher(tags<=tuple>=(), exclude_patterns<=tuple>=())), =TransitiveHydratedTargets)
  Computing Task(transitive_hydrated_targets, Specs(dependencies<=tuple>=(DescendantAddresses(directory=u'pants-plugins/tests/python'),), matcher<=SpecsMatcher>=SpecsMatcher(tags<=tuple>=(), exclude_patterns<=tuple>=())), =TransitiveHydratedTargets, true)
    Computing Task(transitive_hydrated_target, pants-plugins/tests/python/internal_backend_test/sitegen:sitegen, =TransitiveHydratedTarget, true)
      Computing Task(transitive_hydrated_target, 3rdparty/python:future, =TransitiveHydratedTarget, true)
        Computing Task(hydrate_target, 3rdparty/python:future, =HydratedTarget, true)
          Computing Task(hydrate_struct, 3rdparty/python:future, =(symbol table types), true)
            Computing Task(resolve_unhydrated_struct, 3rdparty/python:future, =UnhydratedStruct, true)
              Computing Task(parse_address_family, Dir(path=3rdparty/python), =AddressFamily, true)
                Throw(Failed to parse 3rdparty/python/BUILD:
Invalid requirement, parse error at "u''")
                  Traceback (most recent call last):
                    File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/engine/native.py", line 523, in extern_generator_send
                      res = c.from_value(func[0]).send(c.from_value(arg[0]))
                    File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/engine/build_files.py", line 61, in parse_address_family
                      address_mapper.parser))
                    File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/engine/mapper.py", line 55, in parse
                      raise MappingError('Failed to parse {}:\n{}'.format(filepath, e))
                  MappingError: Failed to parse 3rdparty/python/BUILD:
                  Invalid requirement, parse error at "u''"

[jsirois]

Noting that the linked log does not correspond to the pasted error.

The pasted error happens before any test runs during the build graph setup for the whole pants run of:

./pants.pex test.pytest pants-plugins/tests/python:: -- -v --duration=3

This would seem to point not to a flaky test, but a flaky bit of the production system which is perhaps more alarming. Re-labeling as a bug.

[stuhood]

The actual underlying exception stacktrace is hidden by the raised MappingError. Trying out a nice and dumb repro here: #7052.

[stuhood]

Seen again in master.

Related to the dumb repro attempt: it's possible that wrapping-and-re-raising as MappingError is no longer necessary, and is now just obfuscating errors like these. I'm going to let the CI run on #7052 run to completion, and see whether it fails any tests. If not, will clean it up to land.

[yaup]

This is happening to us as well, and seems to be more common in the recent weeks (3 in the last 7 days, which is why I started paying attention), though I don't have more specific statistics.

Have yet to find a pattern on which commands cause them (the last three that I saw were ./pants list --changed... and ./pants binary (twice)). The only pattern I've found so far is that these happen on our build boxes (which are ubuntu EC2 instances or containers) while I haven't seen any reports on local dev machines (which are osx)

[cosmicexplorer]

If this is something that can be reliably recognized in some way (we can do some hacky thing looking at the exception message), we could consider adding an option to either provide a specific error message saying "THIS IS A FLAKY ERROR, SEE #6861", or to automatically restart after dumping a lot of warnings (and only retry once, probably).

[stuhood]

Hm... have we had a recent repro of this one? Wondering whether #7379 was related somehow.

[yaup]

I don't think I've seen this recently, but I'm not sure how long it's been since I last saw this, probably a week or two?

edit: derp we haven't changed anything regarding pants versions on our side so I'm not sure why I thought anything is supposed to be different. Saw it again today but it is interesting that there were days where it happens a lot and days where it just doesn't.

[haneul]

Is it fixed in recent versions? I hit this issue in 0.14.0

[jsirois]

Seen on master on 5/21/2019 in 'Unit tests (Py2.7 PEX)' shard:

$ ./build-support/bin/ci.sh -2lp
[=== 00:00 CI BEGINS ===]
[=== 00:00 Setting interpreter constraints to ['CPython==2.7.*'] ===]
[=== 00:00 Running internal backend python tests ===]
timestamp: 2019-05-22T04:26:01.002407
Exception caught: (pants.build_graph.address_lookup_error.AddressLookupError)
  File "/home/travis/build/pantsbuild/pants/pants.pex/.bootstrap/pex/pex.py", line 352, in execute
    exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/travis/build/pantsbuild/pants/pants.pex/.bootstrap/pex/pex.py", line 284, in _wrap_coverage
    return runner(*args)
  File "/home/travis/build/pantsbuild/pants/pants.pex/.bootstrap/pex/pex.py", line 315, in _wrap_profiling
    return runner(*args)
  File "/home/travis/build/pantsbuild/pants/pants.pex/.bootstrap/pex/pex.py", line 397, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/travis/build/pantsbuild/pants/pants.pex/.bootstrap/pex/pex.py", line 495, in execute_entry
    return runner(entry_point)
  File "/home/travis/build/pantsbuild/pants/pants.pex/.bootstrap/pex/pex.py", line 510, in execute_pkg_resources
    return runner()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/pants_loader.py", line 85, in main
    PantsLoader.run()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/pants_loader.py", line 81, in run
    cls.load_and_execute(entrypoint)
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/pants_loader.py", line 74, in load_and_execute
    entrypoint_main()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/pants_exe.py", line 39, in main
    PantsRunner(exiter, start_time=start_time).run()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/pants_runner.py", line 88, in run
    return runner.run()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/local_pants_runner.py", line 231, in run
    self._run()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/local_pants_runner.py", line 292, in _run
    goal_runner_result = self._maybe_run_v1()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/local_pants_runner.py", line 259, in _maybe_run_v1
    return goal_runner_factory.create().run()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/goal_runner.py", line 126, in create
    goals, context = self._setup_context()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/goal_runner.py", line 96, in _setup_context
    self._root_dir
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/init/engine_initializer.py", line 226, in create_build_graph
    for _ in graph.inject_roots_closure(target_roots):
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/engine/legacy/graph.py", line 195, in inject_roots_closure
    for address in self._inject_specs(target_roots.specs):
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/engine/legacy/graph.py", line 247, in _inject_specs
    [specs])
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/engine/legacy/graph.py", line 214, in _resolve_context
    'Build graph construction failed: {} {}'.format(type(e).__name__, str(e))
Exception message: Build graph construction failed: ExecutionError 1 Exception encountered:
Computing Select(Specs(dependencies<Exactly(tuple)>=(DescendantAddresses(directory=u'pants-plugins/src/python'), DescendantAddresses(directory=u'pants-plugins/tests/python')), matcher<Exactly(SpecsMatcher)>=SpecsMatcher(tags<Exactly(tuple)>=(), exclude_patterns<Exactly(tuple)>=())), TransitiveHydratedTargets)
  Computing Task(transitive_hydrated_targets(), Specs(dependencies<Exactly(tuple)>=(DescendantAddresses(directory=u'pants-plugins/src/python'), DescendantAddresses(directory=u'pants-plugins/tests/python')), matcher<Exactly(SpecsMatcher)>=SpecsMatcher(tags<Exactly(tuple)>=(), exclude_patterns<Exactly(tuple)>=())), TransitiveHydratedTargets, true)
    Computing Task(transitive_hydrated_target(), pants-plugins/src/python/internal_backend/utilities:plugin, TransitiveHydratedTarget, true)
      Computing Task(transitive_hydrated_target(), 3rdparty/python/twitter/commons:twitter.common.collections, TransitiveHydratedTarget, true)
        Computing Task(hydrate_target(), 3rdparty/python/twitter/commons:twitter.common.collections, HydratedTarget, true)
          Computing Task(hydrate_struct(), 3rdparty/python/twitter/commons:twitter.common.collections, HydratedStruct, true)
            Computing Task(parse_address_family(), Dir(path=3rdparty/python/twitter/commons), AddressFamily, true)
              Throw(Failed to parse 3rdparty/python/twitter/commons/BUILD:
Invalid requirement, parse error at "u''")
                Traceback (most recent call last):
                  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/engine/native.py", line 428, in extern_generator_send
                    res = c.from_value(func[0]).send(c.from_value(arg[0]))
                  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/engine/build_files.py", line 61, in parse_address_family
                    address_mapper.parser))
                  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/engine/mapper.py", line 55, in parse
                    raise MappingError('Failed to parse {}:\n{}'.format(filepath, e))
                MappingError: Failed to parse 3rdparty/python/twitter/commons/BUILD:
                Invalid requirement, parse error at "u''"
Internal backend python test failure
The command "./build-support/bin/ci.sh -2lp" exited with 1.

[jsirois]

Seen in PR #7804 on 5/26/2019 in job: https://travis-ci.org/pantsbuild/pants/jobs/537577092

[Eric-Arellano]

@yaup @haneul could you both please check if your pants.ini has the entry plugins? You can check either by opening it up, or cat pants.ini | grep 'plugins'.

Twitter removed their plugins entry today and as a result saw a crazy amount of this particular failure happening. We're unsure if we never had seen the error before, but we at least had never seen it that severely. I'm wondering if a missing plugins is the cause of this all..your help checking would be highly appreciated.

[yaup]

@Eric-Arellano we don't have any plugins installed via the plugins directive right now (we have some custom ones we built that's added via backend_packages.

[wisechengyi]

another: https://gist.github.com/wisechengyi/b3edaa5d27db2209f78d1a22e1c800bd

[Eric-Arellano]

@yaup also, what Pants version are you using? Are you using the ./pants script we distribute here, and if so, how recently have you updated it? I'm trying to see if you've encountered this problem when running Pants with Python 3, as every reported usage so far is when running with Python 2.

Another way to check this is if you have ever seen the error without u"", but just "".

[yaup]

@Eric-Arellano We are using pants 1.14. I haven't updated the ./pants script since taking over about a year ago, so it should still be running on python 2.

I don't recall seeing just ""

[Eric-Arellano]

Thanks for the quick response! That means that, yes, you are running Pants with Python 2.

I suspect this won't be an issue once you upgrade to >= 1.15.0 and upgrade the ./pants script (see https://www.pantsbuild.org/notes-1.15.x.html#heading__101 for instructions on how to update). I'm going to try to reproduce this for Twitter's codebase right now, which now runs Pants with Python 3 and doesn't have the plugins entry in pants.ini, and will see if I can get this to reproduce.

[yaup]

Hmmm I see. I had 1.15 upgrades in the pipelines when I handed this work off recently, so I suppose we'll see how that goes. Thanks!

[Eric-Arellano]

As a temporary workaround, I believe that you can add plugins: [] to your GLOBAL section of pants.ini. I'm not sure if that will actually fix it, but it seems very likely and it won't hurt anything beyond an extra line.

[stuhood]

Hmm... no repros in a long while on this one (in CI at least).

[Eric-Arellano]

We haven't seen this in several months. It seems this was from Python 2, and possibly when you did not have plugins set in pants.ini. We are far past running Pants w/ Py2 so this isn't relevant anymore 🎉