-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
embed: document go.mod exclusion #45197
Comments
Note that the primary distribution mechanism of modules are zip-files, which will only include the files that are considered part of the module. As I understand it, the zip you download from the module proxy (as opposed to using the repo) won't contain the nested module. So, module boundaries are very important - you can't embed a file that isn't there. A workaround for your case is to name the file differently - e.g. |
Name rewriting seems to come up a fair amount as a workaround for edge case problems with embedding. Someone should write a module to wrap an io.FS in a name rewriter. I volunteer to click the star button on Github and then link to it in comments when someone brings up an issue solved by rewriting, when someone else creates it. |
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it. go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step. Signed-off-by: Adrian Cole <adrian@tetrate.io>
/cc @rsc |
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it. go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step. Signed-off-by: Adrian Cole <adrian@tetrate.io>
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it. go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step. Signed-off-by: Adrian Cole <adrian@tetrate.io>
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it. go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it. go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step. Signed-off-by: Adrian Cole <adrian@tetrate.io>
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it. go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step. Signed-off-by: Adrian Cole <adrian@tetrate.io>
one problem with the rename It would be nice to have a clean way to allow go:embed to opt-out of interpreting go.mod... |
That can't work. It would mean your module can't be build from the zip-file uploaded to the mirror. Third-party embedding solutions don't have this problem only because they generate code which is compiled in - so they effectively create a copy (but in Go syntax) of the files from the nested module in the current module, so it's not a problem if the nested module isn't distributed. This really can't be solved on the |
great explanation, @Merovius. I didn't quite get it last time, but I do now, both the what and the why. awesome! |
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it. go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step. Signed-off-by: Adrian Cole <adrian@tetrate.io>
…129) This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it. go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step. Co-authored-by: Takeshi Yoneda <takeshi@tetrate.io> Signed-off-by: Adrian Cole <adrian@tetrate.io>
I ran into this and decided to use go:generate to tar the embedded module and then include the tape-archived file. It worked. https://github.com/crhntr/disaster-recovery-acceptance-tests/blob/embed-fixtures/fixtures/fixtures.go#L22-L28 My solution looked something like this:
In fixtures.go I have something like: var (
//go:embed fixtures
fixtures embed.FS
)
//go:generate tar cf fixtures/test-app.tar ./cmd/test-app I then commit the fixtures directory. If there was a zip or txtar FS implementation I would use that instead but tar works. For my use case, I have a helper function to write the tar to the filesystem in a tmp dir for each test that needs the test app. |
My use-case is that I'm building a server (in Go ofc) that serves Go projects as downloadable zip files. I have a workspace structure like this:
I want the top-level main.go to be able to embed nested project directories so I can zip them up at runtime and serve them over the wire. The reason it's convenient to keep the nested file named Are there any simple solutions for what I'm trying to do here? It would be so nice if there was a directive that allowed nested modules to be embedded (esp because they aren't a part of the package embedding them, they are essentially flat files) Edit: My workaround currently is to have a build step where I zip up the |
Fixes: golang#45197 Change-Id: I61dff71ba02bfafd054408d73b8a15db745d0198
Change https://go.dev/cl/518858 mentions this issue: |
it really is interesting, you can embed any kind of code (python, rust, c#, ...) my solution (and it is a bad solution) is to have a makefile that
ziping it seems awkward although a bit easier to to. newly added gonew tool is also not a good solution (but it is a good option) because does not support air-gapped environment |
While I'm sure the documentation could be improved, this is already documented. The docs already say "Patterns must not match files outside the package's module". If a subdirectory has a go.mod file, then it is by definition in a different module. |
Fixes: golang#45197 Change-Id: I61dff71ba02bfafd054408d73b8a15db745d0198
Fixes: golang#45197 Change-Id: I61dff71ba02bfafd054408d73b8a15db745d0198
In go 1.18, the prefix "all:" has been added for the embed directive, to include dotFiles (for instance). However, go.mod directories are still not included. Maybe the "all:" prefix could also include these files too. |
as explained above, modules inherently cannot contain other modules. |
One thing which makes it weird is that AIUI the contents of
In this layout, |
"Not being compiled" is not the same as "not being part of the module". They are still used in tests and tests can be run on downloaded modules, so And yes, you can put additional files, outside of the module, into the tree and the regular |
Change https://go.dev/cl/633815 mentions this issue: |
the last referenced CL will close this out to my satisfaction. thanks! |
We have a use case for serving example go projects via
go:embed
. This is particularly important as alternatives togo:embed
have less maintenance now that they are obviated. The problems is that the filego.mod
is special cased and prevents serving a directory that contains it.@opennota did research here and found this behavior to be intentional, notable "stopping at module boundaries" #41191 (comment)
When serving static assets, module boundaries may not be important and in fact be a surprising limitation, as surprising as files outside the directory or via symbolic links might be. Right now, the latter constraints like symbolic links are documented, but go.mod, and why it doesn't work is not entirely obvious.
Ideally, there could be a way to allow go projects to serve go projects as static assets via
go:embed
. Until then, it would be handy to change the documentation to note this surprising constraint explicitly, as it would save folks time and head scratching.Ex a line like this "Matches for a directory containing "go.mod" is invalid and fail to build".
What version of Go are you using (
go version
)?1.16.2
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env
)?amd64 darwin
What did you do?
What did you expect to see?
I expected to have
go build
not failWhat did you see instead?
An error like "pattern X: cannot embed directory Y in different module"
The text was updated successfully, but these errors were encountered: