Bazel build hangs for ts_library in GitHub Actions

[kyubisation]

🐞 bug report

Affected Rule

The issue might be caused by ts_library, but that is hard to say.

Is this a regression?

No

Description

Running bazel build on a medium/large Angular code base in GitHub Actions on ubuntu-latest (Ubuntu 18.04) seems to hang after a certain amount of rules. It works locally.
The affected rule seems to be ts_library, as it consistently hangs on building :unit_test_sources.
Looking at the run times of the steps (when splitting the build targets), it suddenly spikes after a certain amount of build targets (See run times on the right in https://github.com/sbb-design-systems/sbb-angular/runs/558310489?check_suite_focus=true). The target it hangs on is not consistent and changes, when changing the order of the build targets.

Build comman(s): bazelisk build src/... --build_tag_filters=-docs-package,-release-package (https://github.com/sbb-design-systems/sbb-angular/blob/refactor-bazel/.github/workflows/continuous-integration-workflow.yml)

First (full) attempt: https://github.com/sbb-design-systems/sbb-angular/runs/552091858?check_suite_focus=true
Split building of packages: https://github.com/sbb-design-systems/sbb-angular/runs/558191658?check_suite_focus=true
Successfully built, by removing a certain amount of build targets: https://github.com/sbb-design-systems/sbb-angular/runs/558488642?check_suite_focus=true
Last (fullish) attempt: https://github.com/sbb-design-systems/sbb-angular/runs/558646690?check_suite_focus=true

🔬 Minimal Reproduction

https://github.com/sbb-design-systems/sbb-angular/tree/refactor-bazel
(GitHub Actions: https://github.com/sbb-design-systems/sbb-angular/actions?query=branch%3Arefactor-bazel++)

Project .bazelrc: https://github.com/sbb-design-systems/sbb-angular/blob/refactor-bazel/.bazelrc
GitHub ~/.bazelrc: https://github.com/sbb-design-systems/sbb-angular/blob/refactor-bazel/.github/.bazelrc

🔥 Exception or Error

Full log: https://github.com/sbb-design-systems/sbb-angular/runs/561124271?check_suite_focus=true

Last section before it hangs:

[319 / 323] Compiling TypeScript (devmode) //src/public/usermenu:unit_test_sources; 16s worker ... (2 actions, 1 running)
INFO: From Assembling npm package src/public/schematics/npm_package:
[packager_require_patch.js] patching require for @build_bazel_rules_nodejs//internal/pkg_npm:packager
  cwd: /home/runner/.cache/bazel/_bazel_runner/ffb965d366b78d9d290bb95018fa337f/sandbox/linux-sandbox/372/execroot/sbb_angular
  RUNFILES: /home/runner/.cache/bazel/_bazel_runner/ffb965d366b78d9d290bb95018fa337f/sandbox/linux-sandbox/372/execroot/sbb_angular/bazel-out/host/bin/external/build_bazel_rules_nodejs/internal/pkg_npm/packager.sh.runfiles
  TARGET: @build_bazel_rules_nodejs//internal/pkg_npm:packager
  BIN_DIR: bazel-out/host/bin
  GEN_DIR: bazel-out/host/bin
  INSTALL_SOURCE_MAP_SUPPORT: false
  MODULE_ROOTS: [
  {
    "module_name": {},
    "module_root": "build_bazel_rules_nodejs/third_party/github.com/gjtorikian/isBinaryFile/src/index"
  }
]
  NODE_MODULES_ROOT: build_bazel_rules_nodejs/node_modules
  USER_WORKSPACE_NAME: sbb_angular

[packager_require_patch.js] try to resolve in runfiles /home/runner/.cache/bazel/_bazel_runner/ffb965d366b78d9d290bb95018fa337f/sandbox/linux-sandbox/372/execroot/sbb_angular/bazel-out/host/bin/external/build_bazel_rules_nodejs/internal/pkg_npm/packager.sh.runfiles/build_bazel_rules_nodejs/internal/pkg_npm/packager.js
[packager_require_patch.js] resolved file /home/runner/.cache/bazel/_bazel_runner/ffb965d366b78d9d290bb95018fa337f/sandbox/linux-sandbox/372/execroot/sbb_angular/bazel-out/host/bin/external/build_bazel_rules_nodejs/internal/pkg_npm/packager.sh.runfiles/build_bazel_rules_nodejs/internal/pkg_npm/packager.js
[packager_require_patch.js] resolved build_bazel_rules_nodejs/internal/pkg_npm/packager.js within runfiles to /home/runner/.cache/bazel/_bazel_runner/ffb965d366b78d9d290bb95018fa337f/sandbox/linux-sandbox/372/execroot/sbb_angular/bazel-out/host/bin/external/build_bazel_rules_nodejs/internal/pkg_npm/packager.sh.runfiles/build_bazel_rules_nodejs/internal/pkg_npm/packager.js from undefined
[packager_require_patch.js] try to resolve in runfiles /home/runner/.cache/bazel/_bazel_runner/ffb965d366b78d9d290bb95018fa337f/sandbox/linux-sandbox/372/execroot/sbb_angular/bazel-out/host/bin/external/build_bazel_rules_nodejs/internal/pkg_npm/packager.sh.runfiles/build_bazel_rules_nodejs/third_party/github.com/gjtorikian/isBinaryFile/src/index
[packager_require_patch.js] resolved file /home/runner/.cache/bazel/_bazel_runner/ffb965d366b78d9d290bb95018fa337f/sandbox/linux-sandbox/372/execroot/sbb_angular/bazel-out/host/bin/external/build_bazel_rules_nodejs/internal/pkg_npm/packager.sh.runfiles/build_bazel_rules_nodejs/third_party/github.com/gjtorikian/isBinaryFile/src/index
[packager_require_patch.js] try to resolve in runfiles /home/runner/.cache/bazel/_bazel_runner/ffb965d366b78d9d290bb95018fa337f/sandbox/linux-sandbox/372/execroot/sbb_angular/bazel-out/host/bin/external/build_bazel_rules_nodejs/internal/pkg_npm/packager.sh.runfiles/nodejs_linux_amd64/run_npm.sh.template
[packager_require_patch.js] resolved file /home/runner/.cache/bazel/_bazel_runner/ffb965d366b78d9d290bb95018fa337f/sandbox/linux-sandbox/372/execroot/sbb_angular/bazel-out/host/bin/external/build_bazel_rules_nodejs/internal/pkg_npm/packager.sh.runfiles/nodejs_linux_amd64/run_npm.sh.template
[packager_require_patch.js] resolved nodejs_linux_amd64/run_npm.sh.template within runfiles to /home/runner/.cache/bazel/_bazel_runner/ffb965d366b78d9d290bb95018fa337f/sandbox/linux-sandbox/372/execroot/sbb_angular/bazel-out/host/bin/external/build_bazel_rules_nodejs/internal/pkg_npm/packager.sh.runfiles/nodejs_linux_amd64/run_npm.sh.template from /home/runner/.cache/bazel/_bazel_runner/ffb965d366b78d9d290bb95018fa337f/sandbox/linux-sandbox/372/execroot/sbb_angular/bazel-out/host/bin/external/build_bazel_rules_nodejs/internal/pkg_npm/packager.sh.runfiles/build_bazel_rules_nodejs/internal/pkg_npm/packager.js
[364 / 371] Compiling TypeScript (devmode) //src/public/autocomplete:unit_test_sources; 31s worker ... (2 actions running)
[364 / 371] Compiling TypeScript (devmode) //src/public/autocomplete:unit_test_sources; 227s worker ... (2 actions running)
[364 / 371] Compiling TypeScript (devmode) //src/public/autocomplete:unit_test_sources; 328s worker ... (2 actions running)
[364 / 371] Compiling TypeScript (devmode) //src/public/autocomplete:unit_test_sources; 432s worker ... (2 actions running)
[364 / 371] Compiling TypeScript (devmode) //src/public/autocomplete:unit_test_sources; 548s worker ... (2 actions running)
[364 / 371] Compiling TypeScript (devmode) //src/public/autocomplete:unit_test_sources; 687s worker ... (2 actions running)
[364 / 371] Compiling TypeScript (devmode) //src/public/autocomplete:unit_test_sources; 842s worker ... (2 actions running)
[364 / 371] Compiling TypeScript (devmode) //src/public/autocomplete:unit_test_sources; 1027s worker ... (2 actions running)
[364 / 371] Compiling TypeScript (devmode) //src/public/autocomplete:unit_test_sources; 1234s worker ... (2 actions running)
[364 / 371] Compiling TypeScript (devmode) //src/public/autocomplete:unit_test_sources; 1496s worker ... (2 actions running)
[364 / 371] Compiling TypeScript (devmode) //src/public/autocomplete:unit_test_sources; 1827s worker ... (2 actions running)
[364 / 371] Compiling TypeScript (devmode) //src/public/autocomplete:unit_test_sources; 2227s worker ... (2 actions running)
[364 / 371] Compiling TypeScript (devmode) //src/public/autocomplete:unit_test_sources; 2618s worker ... (2 actions running)

Bazel caught terminate signal; shutting down.


Bazel caught terminate signal; shutting down.


Bazel caught terminate signal; shutting down.


Could not interrupt server: (14) Socket closed


Could not interrupt server: (14) Connect Failed


Could not interrupt server: (14) channel is in state TRANSIENT_FAILURE


Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/home/runner/.cache/bazel/_bazel_runner/ffb965d366b78d9d290bb95018fa337f/server/jvm.out')

##[error]The operation was canceled.

🌍 Your Environment

Operating System:

ubuntu-latest of https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners

Output of bazel version:

Bazelisk with .bazelversion 2.1.0 (https://github.com/sbb-design-systems/sbb-angular/blob/refactor-bazel/.bazelversion)

Rules_nodejs version:

(Please check that you have matching versions between WORKSPACE file and @bazel/* npm packages.)

  
1.5.0 (updated, originally 1.4.1)
  

Anything else relevant?

[kyubisation]

I have refactored the code and split up one huge module into smaller modules. This has helped a bit, so I assume a memory problem?
#1803 might be the problem/solution.

[alexeagle]

Yes I'm confident this is another report of #1803. I've been working on that one all day...

[kyubisation]

Thanks for the feedback/confirmation.

On another note; I noticed that you exchanged ng_module with ts_library (with use_angular_plugin). Is this the intended way to compile Angular projects, once ViewEngine compilation can be dropped (for libraries)?
Answered by https://github.com/bazelbuild/rules_nodejs/releases/tag/1.6.0