bazel coverage generating conflicting files with bazel 5.0rc3

[ptarjan]

Description of the problem / feature request:

When trying to use bazel 5.0rc3 I'm seeing this error with our monorepo build:

(05:18:41) ERROR: file 'bazel/python/interpreter/container_py_runtime_pair/baseline_coverage.dat' is generated by these conflicting actions:
  | Label: //bazel/python/interpreter:container_py_runtime_pair
  | RuleClass: py_runtime_pair rule
  | Configuration: 600691a7f9d17ce0f14c6173af35efe5e71db5a8ec4a2a4fcd279b10553a5c30
  | Mnemonic: BaselineCoverage
  | Action key: ee6504712039128a28b0e61d43c28805c6556a4fc52e6a180ea70faa846acc1b, c4ad6d14fd016359eed85dd1130a3a6c09f6bbdddddd4bf639897ecfd16f7c2d
  | Progress message: Writing file bazel/python/interpreter/container_py_runtime_pair/baseline_coverage.dat
  | PrimaryInput: (null)
  | PrimaryOutput: File:[[<execution_root>]bazel-out/k8-fastbuild/testlogs]bazel/python/interpreter/container_py_runtime_pair/baseline_coverage.dat
  | Owner information: ToolchainDependencyConfiguredTargetKey{label=//bazel/python/interpreter:container_py_runtime_pair, config=BuildConfigurationValue.Key[600691a7f9d17ce0f14c6173af35efe5e71db5a8ec4a2a4fcd279b10553a5c30], executionPlatformLabel=@io_bazel_rules_docker//platforms:local_container_platform}, ConfiguredTargetKey{label=//bazel/python/interpreter:container_py_runtime_pair, config=BuildConfigurationValue.Key[600691a7f9d17ce0f14c6173af35efe5e71db5a8ec4a2a4fcd279b10553a5c30]}
  | MandatoryInputs: are equal
  | Outputs: are equal
  | (05:18:41) WARNING: errors encountered while analyzing target '//bazel/python/interpreter:container_py_runtime_pair': it will not be built
  | (05:18:41) ERROR: file 'external/local_config_cc/cc-compiler-k8/baseline_coverage.dat' is generated by these conflicting actions:
  | Label: @local_config_cc//:cc-compiler-k8
  | RuleClass: cc_toolchain rule
  | Configuration: 600691a7f9d17ce0f14c6173af35efe5e71db5a8ec4a2a4fcd279b10553a5c30
  | Mnemonic: BaselineCoverage
  | Action key: ee6504712039128a28b0e61d43c28805c6556a4fc52e6a180ea70faa846acc1b, c4ad6d14fd016359eed85dd1130a3a6c09f6bbdddddd4bf639897ecfd16f7c2d
  | Progress message: Writing file external/local_config_cc/cc-compiler-k8/baseline_coverage.dat
  | PrimaryInput: (null)
  | PrimaryOutput: File:[[<execution_root>]bazel-out/k8-fastbuild/testlogs]external/local_config_cc/cc-compiler-k8/baseline_coverage.dat
  | Owner information: ToolchainDependencyConfiguredTargetKey{label=@local_config_cc//:cc-compiler-k8, config=BuildConfigurationValue.Key[600691a7f9d17ce0f14c6173af35efe5e71db5a8ec4a2a4fcd279b10553a5c30], executionPlatformLabel=@io_bazel_rules_docker//platforms:local_container_platform}, ToolchainDependencyConfiguredTargetKey{label=@local_config_cc//:cc-compiler-k8, config=BuildConfigurationValue.Key[600691a7f9d17ce0f14c6173af35efe5e71db5a8ec4a2a4fcd279b10553a5c30], executionPlatformLabel=@local_config_platform//:host}
  | MandatoryInputs: are equal
  | Outputs: are equal

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Sadly I haven't isolated which rules are actually causing this. Maybe you can help me get you the right debugging information?

What operating system are you running Bazel on?

Linux

What's the output of bazel info release?

$ bazel info release
INFO: Invocation ID: e7ad3ee9-85ab-4ac0-8179-69dab199ad4a
release 5.0.0rc3

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

Replace this line with your answer.

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

Private repo

Have you found anything relevant by searching the web?

No

Any other information, logs, or outputs that you want to share?

Replace this line with your answer.

[ptarjan]

Incase it helps, here are a few snippets that might be applicable:

bazel/python/interpreter/BUILD.bazel

load("@rules_python//python:defs.bzl", "py_runtime", "py_runtime_pair")
...
py_runtime_pair(
    name = "container_py_runtime_pair",
    py2_runtime = None,
    py3_runtime = ":container_py3_runtime",
)

toolchain(
    name = "container_py_toolchain",
    exec_compatible_with = [
        "@io_bazel_rules_docker//platforms:run_in_container",
    ],
    toolchain = ":container_py_runtime_pair",
    toolchain_type = "@bazel_tools//tools/python:toolchain_type",
)

    http_archive(
        name = "rules_python",
        sha256 = "a30abdfc7126d497a7698c29c46ea9901c6392d6ed315171a6df5ce433aa4502",
        strip_prefix = "rules_python-0.6.0",
        url = "https://github.com/bazelbuild/rules_python/archive/0.6.0.tar.gz",
    )

[meteorcloudy]

@ptarjan Can you confirm this doesn't happen with Bazel 4.2? It would be very helpful if you can construct a minimal reproduce case for this error.

/cc @alexjski Any idea on this?

[ptarjan]

Correct. This does not happen with 4.2. I’ll try for a repro case or if there is other logs or parts of my build files they would be helpful please point me at those.

…

On Fri, Jan 7, 2022 at 12:00 AM Yun Peng ***@***.***> wrote: @ptarjan <https://github.com/ptarjan> Can you confirm this doesn't happen with Bazel 4.2? It would be very helpful if you can construct a minimal reproduce case for this error. /cc @alexjski <https://github.com/alexjski> Any idea on this? — Reply to this email directly, view it on GitHub <#14521 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAJZT7XLNMHIVPYYLUYZ2DUU22VLANCNFSM5LM2P7VA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Wyverald]

Also cc @c-mita because "coverage". This might be another release blocker.

[alexjski]

if there is other logs or parts of my build files they would be helpful please
point me at those.

I tried briefly to reproduce that and I am missing //:container_py3_runtime -- would you mind sharing that? As @meteorcloudy said, if you could share a minimal repro (and all that participates in it) which runs into the issue, that would be great.

[ptarjan]

Oh sorry, that is:

py_runtime(
    name = "container_py3_runtime",
    interpreter_path = "/usr/bin/python",
    python_version = "PY3",
)

I'm working on a small repro but I haven't yet found the right combination of targets which make this happen. If I do:

bazel build //...

it always happens, but if I do:

bazel build @local_config_cc//... //bazel/python/interpreter/...

it doesn't.

[ptarjan]

Incase it helps, we also have this in our .bazelrc:

build --instrumentation_filter=^//
build --collect_code_coverage=true

(so that all build operations don't invalidate the analysis cache between coverage runs)

[ptarjan]

I'm playing around with the targets and I was able to get it spitting out a different message that is using only the local_config_cc.

(23:58:40) ERROR: file 'external/local_config_cc/cc-compiler-k8/baseline_coverage.dat' is generated by these conflicting actions:
Label: @local_config_cc//:cc-compiler-k8
RuleClass: cc_toolchain rule
Configuration: 600691a7f9d17ce0f14c6173af35efe5e71db5a8ec4a2a4fcd279b10553a5c30
Mnemonic: BaselineCoverage
Action key: c4ad6d14fd016359eed85dd1130a3a6c09f6bbdddddd4bf639897ecfd16f7c2d, ee6504712039128a28b0e61d43c28805c6556a4fc52e6a180ea70faa846acc1b
Progress message: Writing file external/local_config_cc/cc-compiler-k8/baseline_coverage.dat
PrimaryInput: (null)
PrimaryOutput: File:[[<execution_root>]bazel-out/k8-fastbuild/testlogs]external/local_config_cc/cc-compiler-k8/baseline_coverage.dat
Owner information: ConfiguredTargetKey{label=@local_config_cc//:cc-compiler-k8, config=BuildConfigurationValue.Key[600691a7f9d17ce0f14c6173af35efe5e71db5a8ec4a2a4fcd279b10553a5c30]}, ToolchainDependencyConfiguredTargetKey{label=@local_config_cc//:cc-compiler-k8, config=BuildConfigurationValue.Key[600691a7f9d17ce0f14c6173af35efe5e71db5a8ec4a2a4fcd279b10553a5c30], executionPlatformLabel=@io_bazel_rules_docker//platforms:local_container_platform}
MandatoryInputs: are equal
Outputs: are equal
(23:58:40) WARNING: errors encountered while analyzing target '@local_config_cc//:cc-compiler-k8': it will not be built
(23:58:40) ERROR: file 'external/local_config_cc/cc-compiler-k8/baseline_coverage.dat' is generated by these conflicting actions:
Label: @local_config_cc//:cc-compiler-k8
RuleClass: cc_toolchain rule
Configuration: 600691a7f9d17ce0f14c6173af35efe5e71db5a8ec4a2a4fcd279b10553a5c30
Mnemonic: BaselineCoverage
Action key: c4ad6d14fd016359eed85dd1130a3a6c09f6bbdddddd4bf639897ecfd16f7c2d, ee6504712039128a28b0e61d43c28805c6556a4fc52e6a180ea70faa846acc1b
Progress message: Writing file external/local_config_cc/cc-compiler-k8/baseline_coverage.dat
PrimaryInput: (null)
PrimaryOutput: File:[[<execution_root>]bazel-out/k8-fastbuild/testlogs]external/local_config_cc/cc-compiler-k8/baseline_coverage.dat
Owner information: ToolchainDependencyConfiguredTargetKey{label=@local_config_cc//:cc-compiler-k8, config=BuildConfigurationValue.Key[600691a7f9d17ce0f14c6173af35efe5e71db5a8ec4a2a4fcd279b10553a5c30], executionPlatformLabel=@local_config_platform//:host}, ToolchainDependencyConfiguredTargetKey{label=@local_config_cc//:cc-compiler-k8, config=BuildConfigurationValue.Key[600691a7f9d17ce0f14c6173af35efe5e71db5a8ec4a2a4fcd279b10553a5c30], executionPlatformLabel=@io_bazel_rules_docker//platforms:local_container_platform}
MandatoryInputs: are equal
Outputs: are equal
(23:58:40) WARNING: errors encountered while analyzing target '@local_config_cc//:cc-compiler-k8': it will not be built

[ptarjan]

Ok! I think I have a repro:

http_archive(
    name = "io_bazel_rules_docker",
    sha256 = "db427107e4c36a3713c24daf5fcdf7719fe664e0c36e089b5b16d15cf331ce60",
    strip_prefix = "rules_docker-8a4f73fb29a64ba813087220b200f49a1ca10faa",
    # A few commits past the latest 0.22.0 release so that we pick up
    # https://github.com/bazelbuild/rules_docker/commit/8a4f73fb29a64ba813087220b200f49a1ca10faa
    # which is required to support rules_nodejs 5.0
    urls = ["https://github.com/bazelbuild/rules_docker/archive/8a4f73fb29a64ba813087220b200f49a1ca10faa.tar.gz"],
)

load("@io_bazel_rules_docker//python3:image.bzl", "py3_image")
py3_image(
    name = "bug",
    srcs = [
        "junk.py",
    ],
    main = "junk.py",
)

$ bazel coverage //:bug
INFO: Invocation ID: b30caf08-38bf-429e-8a60-0daf2b89b159
ERROR: file 'external/local_config_cc/cc-compiler-darwin/baseline_coverage.dat' is generated by these conflicting actions:
Label: @local_config_cc//:cc-compiler-darwin
RuleClass: cc_toolchain rule
Configuration: cf5b7d4d0177dc8ba2edf80ba498e217a4f96c06de433e7a62878e573ec3b06c
Mnemonic: BaselineCoverage
Action key: 86a677174951273313888d0fc96686861c8e521335c6d13b1d5afbbeb118d631, c3ddfe4b6b22ca2261c738af252b256f854f97b86e9d2f1570b865a05809c4a0
Progress message: Writing file external/local_config_cc/cc-compiler-darwin/baseline_coverage.dat
PrimaryInput: (null)
PrimaryOutput: File:[[<execution_root>]bazel-out/darwin-fastbuild/testlogs]external/local_config_cc/cc-compiler-darwin/baseline_coverage.dat
Owner information: ToolchainDependencyConfiguredTargetKey{label=@local_config_cc//:cc-compiler-darwin, config=BuildConfigurationValue.Key[cf5b7d4d0177dc8ba2edf80ba498e217a4f96c06de433e7a62878e573ec3b06c], executionPlatformLabel=@io_bazel_rules_docker//platforms:local_container_platform}, ConfiguredTargetKey{label=@local_config_cc//:cc-compiler-darwin, config=BuildConfigurationValue.Key[cf5b7d4d0177dc8ba2edf80ba498e217a4f96c06de433e7a62878e573ec3b06c]}
MandatoryInputs: are equal
Outputs: are equal
ERROR: com.google.devtools.build.lib.skyframe.ArtifactConflictFinder$ConflictException: com.google.devtools.build.lib.actions.MutableActionGraph$ActionConflictException: for external/local_config_cc/cc-compiler-darwin/baseline_coverage.dat, previous action: action 'Writing file external/local_config_cc/cc-compiler-darwin/baseline_coverage.dat', attempted action: action 'Writing file external/local_config_cc/cc-compiler-darwin/baseline_coverage.dat'
INFO: Elapsed time: 0.339s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)

[meteorcloudy]

Thanks! @alexjski can you look into reproducing this?

[alexjski]

Ok! I think I have a repro

Thank you so much @ptarjan! I created a complete repro in my fork of Bazel which has prerequisites available.

I looked into the issue a little bit, what we have here is 2 actions which do differ on the action key:

• Their inputs/outputs and internal key are exactly the same for both of the action.
• The actions differ by the execution platform which is included in the key and makes them different.
  1. PlatformInfo(@local_config_platform//:host, constraints=<[@platforms//cpu:x86_64, @platforms//os:linux]>)
  2. PlatformInfo(@io_bazel_rules_docker//platforms:local_container_platform, constraints=<parent: <[@platforms//cpu:x86_64, @platforms//os:linux]>, [@io_bazel_rules_docker//platforms:run_in_container]>)
• We create 2 ConfiguredTargetKeys for label: @local_config_cc//:cc-compiler-k8, configuration, each of them will create one of the actions, which differ on the execution platform only: BuildConfigurationKey[9d030419d543d9e9cb548173163ba53f9507a4866d8c6457fe843e38ad0fedb9]:
  1. ConfiguredTargetKey -- null executionPlatformLabel
  2. ToolchainDependencyConfiguredTargetKey -- with executionPlatformLabel: @io_bazel_rules_docker//platforms:local_container_platform

If execution platform is important for what the action does (and the logic computing the key suggests so), then I would argue that the output path of the artifact should reflect that (like it does reflect e.g. different target CPU), if not, maybe the key should just be the same.

Adding @katre since he is the expert on execution platforms.

[Wyverald]

Nice! Do we know what caused this to only show up in 5.0.0rc's?

[alexjski]

Nice! Do we know what caused this to only show up in 5.0.0rc's?

I think ToolchainDependencyConfiguredTargetKey did not exist at verision 4, so this is a new feature.

[katre]

The original intention behind having the execution platform be part of the action key is that in remote execution cases, different execution platforms may be sent to different remove executors (depending on the exec_properties of the platform definition), and so this should be distinguished as separate actions. However, there are also cases like this, where the actions are clearly identical.

As for ConfiguredTargetKey, only the ToolchainDependencyConfiguredTargetKey can set the executionPlatformLabel field: the base class should not ever set this. However, it is possible that some instances of ToolchainDependencyConfiguredTargetKey would still leave this null.

I am slightly surprised that these have the same output directory: I would have expected the ToolchainDependencyConfiguredTargetKey to be via an execution transition, which would cause a hash of the execution platform to be present in the output directory. Can you provide a full dependency path from the top-level to that target? The right command would be blaze cquery --flags 'somepath(//top-level:target, config(//final-dependency, final-dependency-config-hash))'.

[alexjski]

I am slightly surprised that these have the same output directory

I think they have the same output directory because the configuration is the same.

[oquenchil]

@c-mita