Spurious failure with stalled macOS builds (specifically the check i686-apple-darwin job)

[kennytm]

First seen on 2017 Aug 31st in:

• Fix end region emission order #44129 (comment) (check i686)
• Use hir::ItemLocalId instead of ast::NodeId in rustc::middle::region::CodeExtent. #44171 (comment) (check i686)
• Fix end region emission order #44129 (comment) (check i686 again)

Symptom:

The job just stopped in the middle with no output in 30 minutes without any reason. End of the log looks like:

[00:40:25] �[m�[m�[32m�[1m   Compiling�[m rustc_llvm v0.0.0 (file:///Users/travis/build/rust-lang/rust/src/librustc_llvm)
[00:40:31] �[m�[m�[32m�[1m   Compiling�[m flate2 v0.2.19
[00:40:34] �[m�[m�[32m�[1m   Compiling�[m rustc_errors v0.0.0 (file:///Users/travis/build/rust-lang/rust/src/librustc_errors)



No output has been received in the last 30m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

The build has been terminated

Status of Travis and MacStadium are both green on 2017 Aug 31st. Not sure if unreported upstream issue or something else.

[ssbarnea]

Well, I was hit by the same issue using brew upgrade .... failed on rust 1.20 ... seems like a recurring problem with rust.

[alexcrichton]

I talked with Travis yesterday and got:

both your mac pros are healthy, once has two VMs and the other has three
both our networking and SAN stuff is hitting some limits, so it may be one of those, unfortunately.

which makes me think it's probably upstream issues and we don't really have a way to work around :(

[kennytm]

Not sure if Travis CI is aware of this, but the problem is very likely in the connection to the log collector or RabbitMQ, not the VMs themselves (yeah not something we can fix).

I'm currently monitoring https://travis-ci.org/rust-lang/rust/jobs/272020017, and logs are being written out if you "Follow log" at the web interface, but opening it again the logs will be all gone.

---

🤔 The failing jobs are always the check jobs, not the dist jobs. The difference between them are xcode8.2 vs xcode7.3. I wonder if this can be fixed by upgrading the check jobs to xcode8.3 or xcode9. (I don't think we want to downgrade to xcode7.3.)

[kennytm]

Incident notice: https://www.traviscistatus.com/incidents/2f0443bbphld

We’re investigating an increased rate of internal restarts of macOS builds, resulting in longer boot times for both public and private repositories. This has resulted in an increased backlog for macOS builds at travis-ci.org

Sep 6, 09:00 UTC

---

Edit: Upstream acknowledged the logging issue:

In addition to longer boot times, users are experiencing an increase in errored builds due to log timeouts when running macOS builds. We are investigating networking issues and will update again as soon as we know more.

Sep 06, 2017 - 10:45 UTC