Sporadic "Command exited with code -1" on some Visual Studio 2017 builds

[FeodorFitsner]

Discussion: http://help.appveyor.com/discussions/problems/6465-vs-2017-image-was-building-fine-then-fails-ever-since-2-days-ago

[jagerman]

Still failing. I have no idea how to debug this. I've tried enabling RDP access on failure, re-running the build on the exact same appveyor image where it just failed, out of the same build directory, and it succeeds every time.

I tried the build in a loop on a local VM (with Windows 10 + VS2017), and 100/100 builds passed.

[jagerman]

(For now, I'm just ignoring the failures and waiting for the next image update).

[FeodorFitsner]

Can you do I repro solution in a public repository? I understand it's not happening with any project, but with only specific ones.

[jagerman]

Sorry, I don't follow: "I repro solution"

[jagerman]

More details/investigation/tracking in pybind/pybind11#792.

[FeodorFitsner]

Sorry, I meant do you have a public repository/build reproducing the issue.

[jagerman]

Ah, yes, the https://github.com/pybind/pybind11 repository exhibits it.

[jagerman]

Here's an interesting failure that didn't even make it to the pybind code before failing (cmake's initial does-the-compiler-work test failed): https://ci.appveyor.com/project/wjakob/pybind11/build/1.0.1691/job/etguctmxjas78jn2

[ThirteenAG]

I have this too, for example: https://ci.appveyor.com/project/ThirteenAG/widescreenfixespack/build/218
Usually rebuilding the same commit works, but "Command exited with code -1" definitely appears every once in a while.

[Neproify]

Same thing on https://ci.appveyor.com/project/Neproify/iv-network

[FeodorFitsner]

We've deployed Visual Studio 2017 update - version 15.1 (26403.07) to Visual Studio 2017 image. It looks like it fixed the issue with exit code -1.

Could you guys please check and let me know how that worked?

[jagerman]

When did the updated version get deployed? The last -1 failure I saw was around 3 hours ago (several builds since haven't had a failure).

[FeodorFitsner]

@jagerman about an hour ago. Actually, we were using your project for verifying the fix: https://ci.appveyor.com/project/FeodorFitsner/pybind11/history 😄

[jagerman]

Cool! :) I'll keep an eye on it and report back if I encounter any more.

[jagerman]

I just encountered another one:

https://ci.appveyor.com/project/wjakob/pybind11/build/1.0.1751/job/8ua0rcj9a8ie2ubr

This is the first one I've seen, so it seems greatly reduced but perhaps not entirely solved.

[FeodorFitsner]

OK. Let's try removing /logger:%MSBuildLogger% and see if that might be affecting the build.
Second, is there any way to get more detailed logging in case of error? If it exists with code -1 I bet there should be something in the detailed log - some error or warning.

[jagerman]

Back when this was occurring often—it is very rare now: this was the first failure I've seen since the update 11 days ago—I tried some builds without /logger and still got failures (this one, for example—the log is really long as I also had MSBuild verbosity set to diagnostic mode, but there doesn't appear to be anything useful in that diagnostic output).

I also haven't been able to find anything useful in logs; though I admit Windows is a bit foreign to me, so I'm not entirely sure where to look for this (or even which program's logs I should be looking for). Something in the Event Viewer? Is there anything helpful I could add to the build script to grab any sort of system errors?

I also don't know what exit code -1 means: is that a process being killed by the system for some reason? Or some internal MSBuild error? (If I try killing MSBuild through task manager, I get a 1 as exit code). And when an individual compilation fails, MSBuild recognizes that and compiler output appears.

[jagerman]

And a couple more: https://ci.appveyor.com/project/wjakob/pybind11/build/1.0.1760/job/bym09yatp2f7xb6e
https://ci.appveyor.com/project/jagerman/pybind11/build/1.0.683/job/tkqgq0c8rnfqvxwd

Until today, there hadn't been any such failures since this issue was closed AFAIK. Did something change in the VS 2017?

[FeodorFitsner]

Will take a look. This maybe some scheduling task or some built-in check for updates (Windows updates are disabled) interfering with the build process.

[Neproify]

Still happening on https://ci.appveyor.com/project/Neproify/iv-network

[FeodorFitsner]

Hopefully, that will be fixed with the next update as we disabled most of VS-related scheduled tasks.

[jagerman]

Seems to be working so far; all builds for the last couple days have been good.

[FeodorFitsner]

Cool, thanks for the update!