Compiler benchmarks are failing and not reporting any results

[Akirathan]

The following benchmarks are failing on org.graalvm.polyglot.PolyglotException: org.enso.editions.EditionResolutionError$CannotLoadEdition: Cannot load edition [2024.1.1-dev]: The edition was not found.:

• org.enso.compiler.benchmarks.inline.InlineCompilerBenchmark.longExpression
  • For example at https://github.com/enso-org/enso/actions/runs/8257237123/job/22587421302#step:8:1231
• org.enso.compiler.benchmarks.inline.InlineCompilerErrorBenchmark.expressionWithErrors
• org.enso.compiler.benchmarks.module.ManyErrorsBenchmark.manyErrors
• org.enso.compiler.benchmarks.module.ManyLocalVarsBenchmark.longMethodWithLotOfLocalVars
• org.enso.compiler.benchmarks.module.ManyNestedBlocksBenchmark.manyNestedBlocks
• org.enso.compiler.benchmarks.module.ManySmallMethodsBenchmark.manySmallMethods

The only way to tell if a benchmark is failing nowadays is to manually look at the output of the benchmark, and to notice there are no results for that benchmark uploaded in the artifacts. In the future, we should report the erroneous benchmarks somehow automatically.

The only compiler benchmark that succeeds is org.enso.compiler.benchmarks.module.ImportStandardLibrariesBenchmark.importStandardLibraries. No other benchmarks are visible on https://enso-org.github.io/engine-benchmark-results/engine-benchs.html

These benchmarks were introduced in #9158

[Akirathan]

Duplicate of #9256

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-03-13):

Progress: - Some general problems with benchmarks and their data collecting:

• Noticed that the compiler benchmarks have been failing since inception
  • Contributed a way to prevent that in the future in dry-run benchmarks exits when some benchmark fails #9397
  • Looking into and fixing the compiler benchmarks.
• Noticed that bench upload job is failing
  • Probably because the underlying job IDs have changed, will have a look into that tomorrow.
• Quality of life improvement of sbt reload in sbt graalVMVersionCheck does not fail during reload #9396 It should be finished by 2024-03-15.