embed: document go.mod exclusion

[codefromthecrypt]

We have a use case for serving example go projects via go:embed. This is particularly important as alternatives to go:embed have less maintenance now that they are obviated. The problems is that the file go.mod is special cased and prevents serving a directory that contains it.

@opennota did research here and found this behavior to be intentional, notable "stopping at module boundaries" #41191 (comment)

When serving static assets, module boundaries may not be important and in fact be a surprising limitation, as surprising as files outside the directory or via symbolic links might be. Right now, the latter constraints like symbolic links are documented, but go.mod, and why it doesn't work is not entirely obvious.

Ideally, there could be a way to allow go projects to serve go projects as static assets via go:embed. Until then, it would be handy to change the documentation to note this surprising constraint explicitly, as it would save folks time and head scratching.

Ex a line like this "Matches for a directory containing "go.mod" is invalid and fail to build".

What version of Go are you using (go version)?

1.16.2

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

amd64 darwin

What did you do?

//go:embed /path/to/goproject/*
var exfs embed.FS

What did you expect to see?

I expected to have go build not fail

What did you see instead?

An error like "pattern X: cannot embed directory Y in different module"

[Merovius]

Note that the primary distribution mechanism of modules are zip-files, which will only include the files that are considered part of the module. As I understand it, the zip you download from the module proxy (as opposed to using the repo) won't contain the nested module. So, module boundaries are very important - you can't embed a file that isn't there.

A workaround for your case is to name the file differently - e.g. go.mod_ - and rename it after the fact by wrapping the embed.FS.

[earthboundkid]

Name rewriting seems to come up a fair amount as a workaround for edge case problems with embedding. Someone should write a module to wrap an io.FS in a name rewriter. I volunteer to click the star button on Github and then link to it in comments when someone brings up an issue solved by rewriting, when someone else creates it.

[cagedmantis]

/cc @rsc

[codefromthecrypt]

one problem with the rename go.mod to go.mod_ approach is that running go mod tidy in the root, ends up importing whatever is in go.mod_. I'm not sure how to work around this, but it is annoying.

It would be nice to have a clean way to allow go:embed to opt-out of interpreting go.mod...

[Merovius]

It would be nice to have a clean way to allow go:embed to opt-out of interpreting go.mod...

That can't work. It would mean your module can't be build from the zip-file uploaded to the mirror. Third-party embedding solutions don't have this problem only because they generate code which is compiled in - so they effectively create a copy (but in Go syntax) of the files from the nested module in the current module, so it's not a problem if the nested module isn't distributed.

This really can't be solved on the //go:embed side, as far as I can tell. It might be possible to add a way to distribute subdirectories with a go.mod without interpreting them as a module - I don't know. But as long as the files are not part of the module, //go:embed can't use them, full stop.

[codefromthecrypt]

great explanation, @Merovius. I didn't quite get it last time, but I do now, both the what and the why. awesome!

[crhntr]

I ran into this and decided to use go:generate to tar the embedded module and then include the tape-archived file. It worked. https://github.com/crhntr/disaster-recovery-acceptance-tests/blob/embed-fixtures/fixtures/fixtures.go#L22-L28

My solution looked something like this:

repo/
  cmd/
    test-app/
      go.mod
      go.sum
      main.go
  fixtures/
  fixtures.go
  go.mod
  go.sum

In fixtures.go I have something like:

var (
	//go:embed fixtures
	fixtures embed.FS
)

//go:generate tar cf fixtures/test-app.tar ./cmd/test-app

I then commit the fixtures directory.

If there was a zip or txtar FS implementation I would use that instead but tar works. For my use case, I have a helper function to write the tar to the filesystem in a tmp dir for each test that needs the test app.

[earthboundkid]

For the record, there are zip and txtar fs.FS implementations.

[wagslane]

My use-case is that I'm building a server (in Go ofc) that serves Go projects as downloadable zip files. I have a workspace structure like this:

go.work
go.mod
main.go
projects/
  project-1/
    go.mod
    main.go

I want the top-level main.go to be able to embed nested project directories so I can zip them up at runtime and serve them over the wire. The reason it's convenient to keep the nested file named go.mod is because the go.work file at the top of the repo makes it easy to make updates to those nested projects without breaking tooling.

Are there any simple solutions for what I'm trying to do here? It would be so nice if there was a directive that allowed nested modules to be embedded (esp because they aren't a part of the package embedding them, they are essentially flat files)

Edit: My workaround currently is to have a build step where I zip up the projects dir into a projects.zip and commit that tho source control. Then I embed the zip file as a slice of bytes and implement a virtual file system on top of that... It kinda sucks but it works. I'd love to see this changed!

[gopherbot]

Change https://go.dev/cl/518858 mentions this issue: embed: document go.mod exclusion

[oktalz]

it really is interesting, you can embed any kind of code (python, rust, c#, ...)
but you can't embed go code, and to be precise, you can embed go code, but not if you use go.mod

my solution (and it is a bad solution) is to have a makefile that

• rename go.mod to go.mod.embed (then internally while serving i rename it back) - then you can embed it
• do a go build with embedded go files
• rename go.mod.embed back to go.mod
  purely because only go code projects are making problems (to repeat again, all other languages you can embed)

ziping it seems awkward although a bit easier to to.

newly added gonew tool is also not a good solution (but it is a good option) because does not support air-gapped environment

[ianlancetaylor]

While I'm sure the documentation could be improved, this is already documented. The docs already say "Patterns must not match files outside the package's module". If a subdirectory has a go.mod file, then it is by definition in a different module.

[francois76]

In go 1.18, the prefix "all:" has been added for the embed directive, to include dotFiles (for instance). However, go.mod directories are still not included. Maybe the "all:" prefix could also include these files too.

[seankhliao]

as explained above, modules inherently cannot contain other modules.

[fgm]

One thing which makes it weird is that AIUI the contents of testdata directories are not compiled so they should not be considered modules, but they still are.

/
/go.mod
/demo.go
/testdata
/testdata/go.mod
/testdata/other.go

In this layout, testdata is skipped by ./... patterns like go list ./... but if demo.go tries to embed testdata/ it is still considered as another module although it is not handled as code.

[Merovius]

"Not being compiled" is not the same as "not being part of the module". They are still used in tests and tests can be run on downloaded modules, so testdata needs to be included in the module zip.

And yes, you can put additional files, outside of the module, into the tree and the regular go command might not know they don't belong into the module. But, again, the important question here is what is or is not included in the module zip file.

[gopherbot]

Change https://go.dev/cl/633815 mentions this issue: embed: document exclusions more explicitly

[codefromthecrypt]

the last referenced CL will close this out to my satisfaction. thanks!