Crash with configuration transition when .so in deps

[chronal-xnor]

Description of the problem:

When using a configuration transition with a C++ build rule, if there is a .so file in the transitive deps of the transitioned target, Bazel crashes with the following stack trace:

Stack trace

Internal error thrown during build. Printing stack trace: java.lang.RuntimeException: Unrecoverable error while evaluating node '//:foo-getter-opt BuildConfigurationValue.Key[d534253f7f366fdfbb590befef52e47f] false' (requested by nodes )
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:531)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:399)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.NullPointerException
	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:877)
	at com.google.devtools.build.lib.vfs.PathFragment.startsWith(PathFragment.java:296)
	at com.google.devtools.build.lib.rules.cpp.LibrariesToLinkCollector.addLinkerInputs(LibrariesToLinkCollector.java:248)
	at com.google.devtools.build.lib.rules.cpp.LibrariesToLinkCollector.collectLibrariesToLink(LibrariesToLinkCollector.java:203)
	at com.google.devtools.build.lib.rules.cpp.CppLinkActionBuilder.build(CppLinkActionBuilder.java:864)
	at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.createDynamicLinkAction(CcLinkingHelper.java:769)
	at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.createCcLinkActions(CcLinkingHelper.java:452)
	at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.link(CcLinkingHelper.java:362)
	at com.google.devtools.build.lib.rules.cpp.CcBinary.createTransitiveLinkingActions(CcBinary.java:833)
	at com.google.devtools.build.lib.rules.cpp.CcBinary.init(CcBinary.java:489)
	at com.google.devtools.build.lib.rules.cpp.CcBinary.create(CcBinary.java:265)
	at com.google.devtools.build.lib.rules.cpp.CcBinary.create(CcBinary.java:80)
	at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createRule(ConfiguredTargetFactory.java:338)
	at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createConfiguredTarget(ConfiguredTargetFactory.java:184)
	at com.google.devtools.build.lib.skyframe.SkyframeBuildView.createConfiguredTarget(SkyframeBuildView.java:767)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.createConfiguredTarget(ConfiguredTargetFunction.java:899)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(ConfiguredTargetFunction.java:344)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:454)
	... 7 more
INFO: Elapsed time: 1.475s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (14 packages loaded, 83 targets configured)
Internal error thrown during build. Printing stack trace: java.lang.RuntimeException: Unrecoverable error while evaluating node '//:foo-getter-opt BuildConfigurationValue.Key[d534253f7f366fdfbb590befef52e47f] false' (requested by nodes )
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:531)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:399)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.NullPointerException
	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:877)
	at com.google.devtools.build.lib.vfs.PathFragment.startsWith(PathFragment.java:296)
	at com.google.devtools.build.lib.rules.cpp.LibrariesToLinkCollector.addLinkerInputs(LibrariesToLinkCollector.java:248)
	at com.google.devtools.build.lib.rules.cpp.LibrariesToLinkCollector.collectLibrariesToLink(LibrariesToLinkCollector.java:203)
	at com.google.devtools.build.lib.rules.cpp.CppLinkActionBuilder.build(CppLinkActionBuilder.java:864)
	at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.createDynamicLinkAction(CcLinkingHelper.java:769)
	at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.createCcLinkActions(CcLinkingHelper.java:452)
	at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.link(CcLinkingHelper.java:362)
	at com.google.devtools.build.lib.rules.cpp.CcBinary.createTransitiveLinkingActions(CcBinary.java:833)
	at com.google.devtools.build.lib.rules.cpp.CcBinary.init(CcBinary.java:489)
	at com.google.devtools.build.lib.rules.cpp.CcBinary.create(CcBinary.java:265)
	at com.google.devtools.build.lib.rules.cpp.CcBinary.create(CcBinary.java:80)
	at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createRule(ConfiguredTargetFactory.java:338)
	at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createConfiguredTarget(ConfiguredTargetFactory.java:184)
	at com.google.devtools.build.lib.skyframe.SkyframeBuildView.createConfiguredTarget(SkyframeBuildView.java:767)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.createConfiguredTarget(ConfiguredTargetFunction.java:899)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(ConfiguredTargetFunction.java:344)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:454)
	... 7 more
java.lang.RuntimeException: Unrecoverable error while evaluating node '//:foo-getter-opt BuildConfigurationValue.Key[d534253f7f366fdfbb590befef52e47f] false' (requested by nodes )
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:531)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:399)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.NullPointerException
	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:877)
	at com.google.devtools.build.lib.vfs.PathFragment.startsWith(PathFragment.java:296)
	at com.google.devtools.build.lib.rules.cpp.LibrariesToLinkCollector.addLinkerInputs(LibrariesToLinkCollector.java:248)
	at com.google.devtools.build.lib.rules.cpp.LibrariesToLinkCollector.collectLibrariesToLink(LibrariesToLinkCollector.java:203)
	at com.google.devtools.build.lib.rules.cpp.CppLinkActionBuilder.build(CppLinkActionBuilder.java:864)
	at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.createDynamicLinkAction(CcLinkingHelper.java:769)
	at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.createCcLinkActions(CcLinkingHelper.java:452)
	at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.link(CcLinkingHelper.java:362)
	at com.google.devtools.build.lib.rules.cpp.CcBinary.createTransitiveLinkingActions(CcBinary.java:833)
	at com.google.devtools.build.lib.rules.cpp.CcBinary.init(CcBinary.java:489)
	at com.google.devtools.build.lib.rules.cpp.CcBinary.create(CcBinary.java:265)
	at com.google.devtools.build.lib.rules.cpp.CcBinary.create(CcBinary.java:80)
	at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createRule(ConfiguredTargetFactory.java:338)
	at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createConfiguredTarget(ConfiguredTargetFactory.java:184)
	at com.google.devtools.build.lib.skyframe.SkyframeBuildView.createConfiguredTarget(SkyframeBuildView.java:767)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.createConfiguredTarget(ConfiguredTargetFunction.java:899)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(ConfiguredTargetFunction.java:344)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:454)
FAILED: Build did NOT complete successfully (14 packages loaded, 83 targets configured)

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Apply the following patch to an empty directory (with patch -p1 < 0001-Repro-Bazel-crash.patch.txt) and run bazel build //:foo-getter-opt

0001-Repro-Bazel-crash.patch.txt

Basically, this is a seemingly straightforward use of configuration transitions that works fine unless a .so file appears in the srcs of something upstream of the transition. In this example, I transitioned on //command_line_option:compilation_mode, but when I encountered the problem I was transitioning on a user-defined build setting.

What operating system are you running Bazel on?

Ubuntu 16.04, x86_64

What's the output of bazel info release?

Discovered on 0.28.1, reproduced on 6392e5c

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

Clean clone of Bazel, bazel build //src:bazel using Bazel 0.28.1.

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

https://github.com/bazelbuild/bazel
00e9d16d69b191bdfdc32bfd6b2bed203e5af844
6392e5c09ce10476fce33c13678f1c445f27ff66

Have you found anything relevant by searching the web?

Different crash related to configuration transitions -- #8936

Any other information, logs, or outputs that you want to share?

Appears to be a failure of the precondition at

bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/LibrariesToLinkCollector.java

Line 245 in 0bb860d

libDir.startsWith(solibDir) || libDir.startsWith(toolchainLibrariesSolibDir),

In particular, here is the value of libDir and solibDir:

• libDir: bazel-out/k8-opt-ST-f87b14d253ada1bf84e725ab51e04286/bin/_solib_k8/_U_S_S_Cfoo-user___U
• solibDir: bazel-out/k8-fastbuild/bin/_solib_k8

Note that the solibDir is not a prefix of libDir, violating the tested precondition. This appears to be due to the alternative bazel-out subdirectory used when building with configuration transitions.

[hlopko]

A little background - currently when linking against dynamic libraries, we require all shared libraries to be put into a single _solib directory, so the program loader doesn't have to search through multiple directories (which is quadratic in the number of directories) when loading dynamic libraries.

Now with configuration transitions this breaks - the _solib directory is created per configuration. That's what the preconditions error tells us, your library is present in the different _solib directory than what we expected. Our options:

0. Change the preconditions check to be rule error = make blaze not crash, but fail the build with sane error message. This will still prevent people from linking against libraries from multiple configurations until we are ready to support the use case.
1. Remove the preconditions check. This will (I didn't try but I assume) make your cl work. It will also mean that we now have multiple _solib directories, and if that number goes out of hand, the binary startup time will suffer. This is mostly visible with cc_tests, each test is one such binary. If we have only a handful configurations in the build, this is not a problem. If the number of configurations increases, the quadratic factor kicks in.
2. Change our C++ Starlark API to allow creating symlinks from other _solib directories to the _solib of the current configuration. That is not an afternoon fix though and requires time investment.

I'll wait for feedback from gregce@ re number of configurations we expect in the build. Until then I'm afraid we have to default to the option 0.

[chronal-xnor]

Thanks for the rundown. If it helps, for my use case, the library in question is actually a prebuilt .so that's included in the build from remote repository. There will be a small number (low single digit) number of configurations per build and only one or two will have the prebuilt .so. Is there any workaround I can implement to make the build succeed until the proper fix is implemented?

[oquenchil]

This should have stopped crashing with: https://cs.opensource.google/bazel/bazel/+/a90c4084315a490c4fecc864d0b3435cc7b2bd64