[4.2.0rc1] rules_apple fails with NoSuchElementException

[katre]

Downstream rules_apple tests for the 4.2.0rc1 release (#13558) are failing with NoSuchElementException in error handling code.

Error stack trace:

(16:44:12) FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=ConfiguredTargetKey{label=//test/starlark_tests/resources:bundle_library_ios_lib, config=BuildConfigurationValue.Key[3bc2cf8a763c114aaa88eca5ffcf097389b4a9682260ba2edc7b24fd980e62cf]}, actionIndex=3}' (requested by nodes 'ArtifactNestedSetKey{rawChildren=[File:[[<execution_root>]bazel-out/ios-arm64-min8.0-applebin_ios-ios_arm64-opt-ST-2c3b17108f53/bin]test/starlark_tests/resources/libbasic_bundle_lib.a, File:[[<execution_root>]bazel-out/ios-arm64-min8.0-applebin_ios-ios_arm64-opt-ST-2c3b17108f53/bin]test/starlark_tests/resources/libbundle_library_ios_lib.a, File:[[<execution_root>]bazel-out/ios-arm64-min8.0-applebin_ios-ios_arm64-opt-ST-2c3b17108f53/bin]test/starlark_tests/resources/libempty_strings_file_lib.a, File:[[<execution_root>]bazel-out/ios-arm64-min8.0-applebin_ios-ios_arm64-opt-ST-2c3b17108f53/bin]test/starlark_tests/resources/libios_localized_assets_lib.a, File:[[<execution_root>]bazel-out/ios-arm64-min8.0-applebin_ios-ios_arm64-opt-ST-2c3b17108f53/bin]test/starlark_tests/resources/libios_non_localized_assets_lib.a, File:[[<execution_root>]bazel-out/ios-arm64-min8.0-applebin_ios-ios_arm64-opt-ST-2c3b17108f53/bin]test/starlark_tests/resources/libnested_bundle_lib.a, File:[[<execution_root>]bazel-out/ios-arm64-min8.0-applebin_ios-ios_arm64-opt-ST-2c3b17108f53/bin]test/starlark_tests/resources/libobjc_main_lib.a, File:[[<execution_root>]bazel-out/ios-arm64-min8.0-applebin_ios-ios_arm64-opt-ST-2c3b17108f53/bin]test/starlark_tests/resources/libobjc_common_lib.a, File:[[<execution_root>]bazel-out/ios-arm64-min8.0-applebin_ios-ios_arm64-opt-ST-2c3b17108f53/bin]test/starlark_tests/resources/libsticker_pack_ios_lib.a, File:[[<execution_root>]bazel-out/ios-arm64-min8.0-applebin_ios-ios_arm64-opt-ST-2c3b17108f53/bin]test/starlark_tests/targets_under_test/ios/app-linker.objlist]}')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:563)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:398)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.util.NoSuchElementException
	at java.base/java.util.LinkedHashMap$LinkedHashIterator.nextNode(Unknown Source)
	at java.base/java.util.LinkedHashMap$LinkedEntryIterator.next(Unknown Source)
	at java.base/java.util.LinkedHashMap$LinkedEntryIterator.next(Unknown Source)
	at com.google.devtools.build.lib.runtime.UiStateTracker.describeActionProgress(UiStateTracker.java:713)
	at com.google.devtools.build.lib.runtime.UiStateTracker.describeAction(UiStateTracker.java:783)
	at com.google.devtools.build.lib.runtime.UiStateTracker.sampleOldestActions(UiStateTracker.java:948)
	at com.google.devtools.build.lib.runtime.UiStateTracker.printActionState(UiStateTracker.java:882)
	at com.google.devtools.build.lib.runtime.UiStateTracker.writeProgressBar(UiStateTracker.java:1257)
	at com.google.devtools.build.lib.runtime.UiEventHandler.addProgressBar(UiEventHandler.java:968)
	at com.google.devtools.build.lib.runtime.UiEventHandler.handleLocked(UiEventHandler.java:352)
	at com.google.devtools.build.lib.runtime.UiEventHandler.handleInternal(UiEventHandler.java:430)
	at com.google.devtools.build.lib.runtime.UiEventHandler.handle(UiEventHandler.java:458)
	at com.google.devtools.build.lib.events.Reporter.handle(Reporter.java:142)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.dumpRecordedOutErr(SkyframeActionExecutor.java:1629)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.dumpRecordedOutErr(SkyframeActionExecutor.java:1612)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$1800(SkyframeActionExecutor.java:130)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.actuallyCompleteAction(SkyframeActionExecutor.java:1094)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:1065)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:975)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:129)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:81)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:472)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:834)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:307)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:477)
	... 4 more
bazel build failed with exit code 37

More details: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2105#d0d81f5b-9c85-4db8-b1ac-dfe2e57ba30f

[katre]

Looks like there is a race condition in UiStateTracker.describeActionProgress.

Specifically, the code first checks if action.runningProgresses is empty, then grabs the first, but something is clearing the list between the lines.

This isn't deterministic, but should still be fixed. Passing back to @coeuvre to investigate.