IllegalStateException while building C++ on Windows

[irengrig]

I was able to reproduce the issue first reported here: envoyproxy/envoy#4516

0. On the Windows machine
1. clone envoy project: (currently HEAD is at 9d32e5c2a14cd9ab96b6e77fb04f7bd77b2c0d71)
   git clone https://github.com/envoyproxy/envoy.git
2. install Bazel 17.0.2 (last release)
3. cd envoy
   bazel build //source/exe:envoy-static
4. see in the output: (below), also please check the attached java.log, which contains the following exceptions (there are multiple exceptions for several targets):
   181002 16:43:21.375:WT 903 [com.google.devtools.build.lib.concurrent.AbstractQueueVisitor.maybeSaveUnhandledThrowable] Found critical error in queue visitor java.lang.RuntimeException: Unrecoverable error while evaluating node '@envoy_api//envoy/config/filter/http/transcoder/v2:transcoder_cc BuildConfigurationValue.Key[297ab5962734d5a7c3f893fa49bcd72c] false' (requested by nodes '//source/common/config:filter_json_lib BuildConfigurationValue.Key[297ab5962734d5a7c3f893fa49bcd72c] false') at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:497) at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:491) at com.google.devtools.build.lib.rules.cpp.LibrariesToLinkCollector.addDynamicInputLinkOptions(LibrariesToLinkCollector.java:290) at com.google.devtools.build.lib.rules.cpp.LibrariesToLinkCollector.addLinkerInputs(LibrariesToLinkCollector.java:258) at com.google.devtools.build.lib.rules.cpp.LibrariesToLinkCollector.collectLibrariesToLink(LibrariesToLinkCollector.java:203) at com.google.devtools.build.lib.rules.cpp.CppLinkActionBuilder.build(CppLinkActionBuilder.java:916) at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.createDynamicLibrary(CcLinkingHelper.java:923) at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.createCcLinkActions(CcLinkingHelper.java:713) at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.link(CcLinkingHelper.java:462) at com.google.devtools.build.lib.rules.cpp.CcLibrary.init(CcLibrary.java:286) at com.google.devtools.build.lib.rules.cpp.CcLibrary.create(CcLibrary.java:74) at com.google.devtools.build.lib.rules.cpp.CcLibrary.create(CcLibrary.java:55) at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createRule(ConfiguredTargetFactory.java:320) at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createConfiguredTarget(ConfiguredTargetFactory.java:205) at com.google.devtools.build.lib.skyframe.SkyframeBuildView.createConfiguredTarget(SkyframeBuildView.java:631) at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.createConfiguredTarget(ConfiguredTargetFunction.java:770) at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(ConfiguredTargetFunction.java:320) at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:420) ... 4 more

output in the command line:

FAILED: Build did NOT complete successfully (203 packages loaded)
java.lang.RuntimeException: Unrecoverable error while evaluating node '@envoy_api//envoy/config/filter/http/transcoder/v2:transcoder_cc BuildConfigurationValue.Key[297ab5962734d5a7c3f893fa49bcd72c] false' (requested by nodes '//source/common/config:filter_json_lib BuildConfigurationValue.Key[297ab5962734d5a7c3f893fa49bcd72c] false')
at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:497)
at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:491)
at com.google.devtools.build.lib.rules.cpp.LibrariesToLinkCollector.addDynamicInputLinkOptions(LibrariesToLinkCollector.java:290)
at com.google.devtools.build.lib.rules.cpp.LibrariesToLinkCollector.addLinkerInputs(LibrariesToLinkCollector.java:258)
at com.google.devtools.build.lib.rules.cpp.LibrariesToLinkCollector.collectLibrariesToLink(LibrariesToLinkCollector.java:203)
at com.google.devtools.build.lib.rules.cpp.CppLinkActionBuilder.build(CppLinkActionBuilder.java:916)
at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.createDynamicLibrary(CcLinkingHelper.java:923)
at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.createCcLinkActions(CcLinkingHelper.java:713)
at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.link(CcLinkingHelper.java:462)
at com.google.devtools.build.lib.rules.cpp.CcLibrary.init(CcLibrary.java:286)
at com.google.devtools.build.lib.rules.cpp.CcLibrary.create(CcLibrary.java:74)
at com.google.devtools.build.lib.rules.cpp.CcLibrary.create(CcLibrary.java:55)
at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createRule(ConfiguredTargetFactory.java:320)
at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createConfiguredTarget(ConfiguredTargetFactory.java:205)
at com.google.devtools.build.lib.skyframe.SkyframeBuildView.createConfiguredTarget(SkyframeBuildView.java:631)
at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.createConfiguredTarget(ConfiguredTargetFunction.java:770)
at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(ConfiguredTargetFunction.java:320)
at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:420)
... 4 more
java.lang.RuntimeException: Unrecoverable error while evaluating node '@envoy_api//envoy/config/filter/http/transcoder/v2:transcoder_cc BuildConfigurationValue.Key[297ab5962734d5a7c3f893fa49bcd72c] false' (requested by nodes '//source/common/config:filter_json_lib BuildConfigurationValue.Key[297ab5962734d5a7c3f893fa49bcd72c] false')
at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:497)
at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:491)
at com.google.devtools.build.lib.rules.cpp.LibrariesToLinkCollector.addDynamicInputLinkOptions(LibrariesToLinkCollector.java:290)
at com.google.devtools.build.lib.rules.cpp.LibrariesToLinkCollector.addLinkerInputs(LibrariesToLinkCollector.java:258)
at com.google.devtools.build.lib.rules.cpp.LibrariesToLinkCollector.collectLibrariesToLink(LibrariesToLinkCollector.java:203)
at com.google.devtools.build.lib.rules.cpp.CppLinkActionBuilder.build(CppLinkActionBuilder.java:916)
at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.createDynamicLibrary(CcLinkingHelper.java:923)
at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.createCcLinkActions(CcLinkingHelper.java:713)
at com.google.devtools.build.lib.rules.cpp.CcLinkingHelper.link(CcLinkingHelper.java:462)
at com.google.devtools.build.lib.rules.cpp.CcLibrary.init(CcLibrary.java:286)
at com.google.devtools.build.lib.rules.cpp.CcLibrary.create(CcLibrary.java:74)
at com.google.devtools.build.lib.rules.cpp.CcLibrary.create(CcLibrary.java:55)
at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createRule(ConfiguredTargetFactory.java:320)
at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createConfiguredTarget(ConfiguredTargetFactory.java:205)
at com.google.devtools.build.lib.skyframe.SkyframeBuildView.createConfiguredTarget(SkyframeBuildView.java:631)
at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.createConfiguredTarget(ConfiguredTargetFunction.java:770)
at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(ConfiguredTargetFunction.java:320)
at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:420)
... 4 more
FAILED: Build did NOT complete successfully (203 packages loaded)

[irengrig]

java.log

[scottmin0r]

These stack traces look similar to #6169 and #6171.

[irengrig]

Thank you @scottminor-google, they are indeed seem duplicates. (However I want people from C++ team to confirm this @mhlopko @oquenchil )

[meteorcloudy]

I believe this issue (along with #6169 and #6171) are caused by 4e42e17

I added this precondition to prevent passing a shared library (DLL file) to linker on Windows. Because on Windows, we need its interface library for linking.

However, I just found this was not true for mingw (http://www.mingw.org/wiki/sampledll)

The import library created by the "--out-implib" linker option is required iff (==if and only if) the DLL shall be interfaced from some C/C++ compiler other than the MinGW toolchain. The MinGW toolchain is perfectly happy to directly link against the created DLL.

So for #6169, we should disable this precondition check for mingw gcc.

For this issue and #6171, I'm not sure which toolchain is used, I guess it's still the native MSVC toolchain?
Then we need to figure out why it's trying to pass a shared library (DLL file) to the linker.
I'll investigate more when I have time today.

[irengrig]

@meteorcloudy I was using the Windows virtual machine, and did not pass any specific flags, so it must be MSVC toolchain yes. It is really easy to reproduce.

[meteorcloudy]

I have a fix for this issue and #6169, #6171. But after fixing it, the build stuck at 'fetching @envoy_deps`, I believe it's because the CMake build doesn't work well on Windows, but that should be another issue.

[meteorcloudy]

@irengrig I'll leave this to you to decide if it should be cherry-picked for 0.19.0 release.

[rongjiecomputer]

Can the fix be included in the next closest release? I want to try to enable native singlejar on Windows as soon as possible (see #2241), but it will require the Bazel executable that it building the binary to have the fix.

[irengrig]

Can the fix be included in the next closest release? I want to try to enable native singlejar on Windows as soon as possible (see #2241), but it will require the Bazel executable that it building the binary to have the fix.

Unfortunately, we can not include it, since 0.19.0 is already branched and it is not a regression (this is the policy...). Sorry.

[rongjiecomputer]

Too bad. I hope it can at least be included in 0.20.0 so that I can enable native singlejar in 0.21.0? (Not familiar with how Bazel release works).

[meteorcloudy]

@rongjiecomputer We'll have release every month, yes I think it will be included 0.20.0.

[scottmin0r]

I'm not sure I follow - the (rather trivial) example in #6171 builds using 0.16.1, but not 0.17.x. How is this not a regression? Seeing as 0.18.0 was released today, it is 2 months until 0.20.0. Why do we need to wait two months to finally be able to build with the latest release on Windows again?

[meteorcloudy]

@scottminor-google You are right! It is a regression in 0.17.0, so we indeed should cherry-pick it into 0.19.0.

[laurentlb]

If it's a regression, it should be cherry-picked into a patch release of 0.18.

(we could consider a patch on top of 0.17, but I hope we can avoid it)

[hlopko]

I advice to create both patch 0.18 release, and to cherrypick the fix into 0.19. I consider this issue serious enough since:

• we cannot build protobuf with one of 2 Bazel provided Windows C++ toolchains
• we cannot build a project that has a proto_library depending on another proto_library.

BUT.

Last known working version was (reportedly) 0.16.1, so if we want to be pedantic, the issue is too old to be considered a regression. Therefore I'm fine with applying this policy strictly, I don't want to create a precedent. I leave the decision to the release team, and John in particular.

[scottmin0r]

@mhlopko Thanks for your support - our team is blocked from upgrading past 0.16.1 by this issue so I definitely agree with the first part of your assessment.

if we want to be pedantic, the issue is too old to be considered a regression

This I don't understand - can you help clarify the policy here?

Related issues #6169 and #6171 were reported on 17 September, 3 days after the release of 0.17.1 and 28 days prior to the release of 0.18.0. The regression might seem older than it is because it did not gain any visibility until this issue was created on 02 October. Regardless, this was reported three days after a patch release, with almost a month until the next minor release.

When should regressions be reported to be considered worth fixing?

[laurentlb]

@meteorcloudy Which commits should be cherry-picked? Only 914b4ce?

[meteorcloudy]

@laurentlb Yes, only 914b4ce

[irengrig]

@meteorcloudy can I close it already?

[meteorcloudy]

Yes, this is fixed.