Link rules that rely on cc_toolchain don't correctly resolve to an execution platform

[ebracho]

What version of rules_go are you using?

0.17.0

What version of gazelle are you using?

0.17.0

What version of Bazel are you using?

0.23.2

Does this issue reproduce with the latest releases of all the above?

Yes (also tested with rules_go v0.18.1)

What operating system and processor architecture are you using?

linux x86_64

Any other potentially useful information about your toolchain?

We are using remote build execution with our own custom platforms and cc toolchains (generated by rbe_autoconfig). We have three registered platforms, two of which we also have a compatible cc toolchain for (@bazel_tools//tools/cpp:toolchain_type), and one that doesn't have any kind of cc toolchain support.

What did you do?

Modified one of our go_test targets (which has a transient cgo library dependency) to only be compatible with the non-cc-toolchain platform

BUILD.bazel

go_test(
    name = "go_default_test",

    # //bazel/platforms:service_ubuntu is a constraint value that only matches
    # the non-cc-toolchain platform. 
    exec_compatible_with = [ "//bazel/platforms:service_ubuntu" ],

    deps = [
        # ... (transient cgo dependency)
    ],
    # ...
)

And ran bazel test --config=remote :go_default_test

What did you expect to see?

Toolchain resolution failure which prevents the action from executing.

What did you see instead?

GoLink action was executed on the non-cc-toolchain-supporting platform

ERROR: /home/eddie/analytics/go/src/arb/dqs/BUILD.bazel:30:1: GoLink go/src/arb/dqs/linux_amd64/go_default_test failed (Exit 1)
external/go_sdk/pkg/tool/linux_amd64/link: running /usr/bin/gcc failed: fork/exec /usr/bin/gcc: no such file or directory

Side note: the reason we were looking into this is that we want to execute our go tests on a platform that more closely resembles our production environments (which don't have gcc). However, as far as I understand there isn't a way to specify exec_compatible_with constraint values for specific rules generated by go_test, so the actual test execution action must share the same constraints as the compile/archive/link actions.

This could be a use case worth considering when addressing #1957

[jayconrod]

So if I understand correctly, you want to execute the Go compile and link actions on one platform and the test actions on another platform (the non-cc-toolchain platform), right?

I'm not sure how to express that in Starlark rules. @katre do you know?

[katre]

There are two pieces here:

1. @jayconrod's question: no, every action from the same target will have the same execution platform. The workaround for this, if really needed, is to decompose the rule into two separate rules, and use a macro to create both.
2. @ebracho's problem with cc toolchain selection: Currently, cc toolchain selection is not using execution platforms and toolchain resolution. We're working to fix this and a lot of work is underway. See incompatible_enable_cc_toolchain_resolution: Turn on toolchain resolution for cc rules bazelbuild/bazel#7260 for status.

[jayconrod]

Thanks @katre!

I'll close this issue, since I don't think this can be addressed in rules_go. @ebracho, you may want to chat with the RBE support folks to see if they have a recommended solution for this. This doesn't seem like a Go-specific problem, and they may have encountered this for another language, too.

[andyleap]

(Coworker of @ebracho here)
@jayconrod: The issue is not really cc toolchain selection. We want to build images using a "builder" image, that has gcc and other dependencies installed, and then run the tests in the image that would actually be deployed. We've been burned by builder image having things (like timezone info) that accidentally got removed from the production image during an upgrade, so the ideal fix is to run our tests in the production image. As I understand, this is a rules_go thing, as it would require breaking go_test out into a macro that creates 2 targets, with configurable exec environments.

[jayconrod]

This seems almost like cross-compilation. Can you select the different platforms with --platform and --host_platform when testing?

I'd rather not break go_test into multiple rules. That seems more complicated than necessary.

[andyleap]

It's kinda like cross-compilation, but bazel doesn't see it as that (as far as I can tell). The build process for the test doesn't say [for host], so I don't see that messing with host platform would affect it. While I do agree that breaking it into multiple rules is more complicated than necessary for the average case, it's something that we do need, and that we are currently working around via a custom test + a macro that marks the go_test as manual. This isn't the ideal state, so I'd love to get some of this stuff better supported via rules_go.

Maybe even just something like go_test is the default, and stays as the current system, but add a go_test_binary that doesn't run as a test, but generates the same test binary as go_test does, so those of us with special needs can use it without having to do seemingly hacky things?

[ebracho]

Thanks for looking into this guys. I haven't fully wrapped my head around the interactions between cgo rules, cc_* rules and toolchain resolution, but based on @katre's response this appears to be a known feature request for Bazel rather than a rules_go issue.

@jayconrod to clarify, I was trying to make it so the binary generated by go_test executes on a separate platform. If I understand correctly, link actions must execute on a gcc platform when a cgo module is involved (since the go link tool would rely on gcc). @andyleap came up with a nice solution which was to write a macro that tags go_test targets as manual and create a new test rule that depends on the test binary it outputs.

[jayconrod]

What I meant earlier about cross-compilation: this is similar to the case where you want to build on platform A and test on platform B. In this case, A and B are the same operating system / architecture, but B lacks toolchains. I haven't kept up with recent developments in Bazel toolchains, so I'm not sure if it's currently possible to express that. I hope so.

Again, I don't think this is a Go-specific issue. I expect you'd have the same problem with any language that depends on the C/C++ toolchain. Or any situation where tests need to run in an environment where the language's own toolchain is not present.

About implementing a new go_test_binary rule, sorry, I have very limited bandwidth to work on this in rules_go, and I don't think the general-purpose value justifies the ongoing maintenance costs. It seems like the macro solution should work well enough. You may also be able to write go_test_binary on your own with the toolchain API.