Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

binary publishing is currently failing #258

Closed
codeboten opened this issue Dec 21, 2022 · 5 comments
Closed

binary publishing is currently failing #258

codeboten opened this issue Dec 21, 2022 · 5 comments

Comments

@codeboten
Copy link
Contributor

The last step before publishing a build is currently failing. See https://github.com/open-telemetry/opentelemetry-collector-releases/actions/runs/3743728445 for an example failure.

#8 [stage-1 2/3] COPY --chmod=755 otelcol-contrib /otelcol-contrib
#8 ERROR: failed to copy files: userspace copy failed: write /var/lib/docker/overlay2/c4hvwfbqrbn9zo6rgn18g2wyh/merged/otelcol-contrib: no space left on device
------
 > [stage-1 2/3] COPY --chmod=755 otelcol-contrib /otelcol-contrib:
------
ERROR: failed to solve: failed to copy files: userspace copy failed: write /var/lib/docker/overlay2/c4hvwfbqrbn9zo6rgn18g2wyh/merged/otelcol-contrib: no space left on device
Learn more at https://goreleaser.com/errors/docker-build
Error: The process '/opt/hostedtoolcache/goreleaser-action/1.13.1/x64/goreleaser' failed with exit code 1
@codeboten
Copy link
Contributor Author

As per @mx-psi's investigation:

What we know for certain so far

  • the builder works correctly locally (I haven't tried to do a full goreleaser run though)
  • on the CI, we run out of disk space (seemingly on the whole runner, since it even fails to recover the logs after that error)
  • binaries have not increased dramatically in size (roughly ~2MB bigger than previous release, in line with previous bump)
  • there have been no interesting changes on the tooling versions we use (QEMU, dockerx, goreleaser, Go, cosign have all the same versions)

Current most likely root causes in my mind

  • the natural increase in binary size has made it so that we fill up the Github runner 14GB disk just by creating artifacts
  • something changed on the runner environment (not sure what?) that made it so that we use more disk space on the goreleaser build

Useful next steps

  • Running goreleaser locally on the v0.68.0 tag and seeing if something goes wrong
  • Having a way to track the disk space on the Github runner while pushing the artifacts?

@codeboten
Copy link
Contributor Author

A suggestion if the issue is that the build is reaching size constraints: https://goreleaser.com/customization/partial/

@mx-psi
Copy link
Member

mx-psi commented Dec 21, 2022

I removed the v0.68.0 tag from this repository, we will re-tag once this is fixed

@codeboten
Copy link
Contributor Author

Splitting the build would also address #44

@codeboten
Copy link
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants