go: support //go:embed directives with embedsrcs attribute

[jayconrod]

go_library, go_binary, and go_test now support the //go:embed
directive in Go 1.16. When building a package containing a //go:embed
directive, they'll match patterns against embedable files to generate
an embedcfg file to pass to the compiler.

Embeddable files must be listed in the new embedsrcs attribute.

The GoSource provider now has an embedsrcs field to support this.

Fixes #2775

[jayconrod]

@bazelbuild/go-maintainers Please comment if you have any thoughts on this. I'm happy to answer any questions, too. This is a good template for adding new attributes and understanding how the rules work.

[tux21b]

Any updates yet? I am waiting eagerly for this exact feature and I was hoping it would make the Go 1.16 release.

[nicolai86]

any pointers on how to try this out before it's released would be greatly appreciated too :)

[achew22]

@nicolai86 you can, as always load from any path you want with a git_repository. You should be able to refer to Jay's branch and try it out. Please do report back if you have any issues with the new embeds stuff.

[nicolai86]

so I've replaced my rules_go with Jay's fork, setup embedsrcs, and it works really great. Much better as compared to go_embed_data directive!

simple change:

git_repository(
    name = "io_bazel_rules_go",
    branch = "embedsrcs",
    remote = "https://github.com/jayconrod/rules_go.git",
)

# http_archive(
#     name = "io_bazel_rules_go",
#     sha256 = "7904dbecbaffd068651916dce77ff3437679f9d20e1a7956bff43826e7645fcc",
#     urls = [
#         "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.25.1/rules_go-v0.25.1.tar.gz",
#         "https://github.com/bazelbuild/rules_go/releases/download/v0.25.1/rules_go-v0.25.1.tar.gz",
#     ],
# )

and straight forward transformation of go_embed_data to embedsrcs:

-go_embed_data(
-    name = "data",
-    srcs = glob([
-        "**/*.tpl",
-    ]),
-    package = "templates",
-    var = "Data",
-    visibility = ["//visibility:private"],
-)

 go_library(
     name = "go_default_library",
     srcs = [
         "doc.go",
-        ":data",  # keep
     ],
+    embedsrcs = glob([
+        "**/*.tpl",
+    ]),
     importpath = "git.randschau.ca/randschau/ledger/cmd/server/templates",
     visibility = ["//visibility:public"],
 )

smooth sailing with 1.16. <3

[tux21b]

Another question: we currently have a target :ui which runs react-scripts. react-scripts comes with a webpack config that generates a variable number of files. The files mostly look like "chunk.[sha256].js" or something like that (containing a hash so that the files can be cached indentinitely). As far as I know, Bazel requires static input / output files and therefore our :ui rule generates a "ui.tar" file containing the variable number of files.

Later, we use a custom go_bindata rule to extract the tar archive and generate an ui_bindata.go which is then compiled into the binary.

How would one archive a similar result with the new emedsrcs attribute? Is there a way to pass a "variable collection of files"? Should we also support passing tar achives? Embedding the tar archive directly would be possible, but has several disadvantages. Embedding a zip might work however (since zip also implements fs.FS) so that could be a possible workaround.

[jayconrod]

@nicolai86 Thanks for trying it out! Glad it works.

@tux21b Not sure if this is an option for you, but rules can declare output directories using ctx.actions.declare_directory. The directory needs to have a fixed name, but the action that produces it can put anything in there. You should be able to embed the contents of generated directories using a //go:embed directive and embedsrcs.

@achew22 I'm still working on generating embedsrcs in Gazelle, but I'll go ahead and merge this to make it easier to try out. Let me know if you have any other comments; happy to fix in a follow-up PR.

[rsippl]

@jayconrod embedsrcs works fine for me, too, thanks! One issue, though: I'm embedding a directory with an arbitrary number of files. If one of them changes, or a file is added, these changes don't get picked up by bazel. They do get picked up after a "bazel clean", and also when running via plain old "go run". Is this behavior desired, or am I doing something wrong?

[jayconrod]

@rsippl Sounds like something isn't going right. Files in embedsrcs are inputs, so if one of them changes, it should cause the package to be rebuilt. Could you open an issue with more detail?

[tommyknows]

@rsippl I've had something similar before (though not with this feature in particular), but I wasn't using a glob. Instead, I targeted the directory directly, e.g. //my:dir. What I should've done was glob(["//my:dir/**"]). With that, everything worked as expected.

[rsippl]

Files in embedsrcs are inputs, so if one of them changes, it should cause the package to be rebuilt.

@jayconrod this is true for files (e.g. embedsrcs = ["web/index.html"]), but not for directories (e.g. embedsrcs = ["web"]). A glob syntax as mentioned by tommyknows doesn't seem to be supported. How do you specify directories in embedsrcs?

[rsippl]

@jayconrod turns out I was wrong, embedsrcs = glob(["web/**"]) does works fine, and changes do get picked up this way. It would be better if the embedsrcs = ["web"] syntax didn't work at all (if web is a dir), so one is forced to use a glob, otherwise it's a bit confusing. But I guess this is the way Bazel works.

[jayconrod]

If you're embedding static files in a directory, you do need to specify them individually or with a glob. Bazel won't let you only name the directory. That's a design decision by Bazel; rules_go can't do anything about that.

If you have a rule that generates a directory (declared with ctx.actions.declare_directory), you should be able to embed that directory by name. glob won't match anything inside it, since glob is evaluated before Bazel knows what files are going to be in that directory.