Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Provide a tiny tar utility for use in tests #11183

Open
aiuto opened this issue Apr 22, 2020 · 3 comments
Open

Provide a tiny tar utility for use in tests #11183

aiuto opened this issue Apr 22, 2020 · 3 comments
Labels
area-EngProd Bazel CI, infrastructure, bootstrapping, release, and distribution tooling P3 We're not considering working on this, but happy to review a PR. (No assignee) team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website type: feature request

Comments

@aiuto
Copy link
Contributor

aiuto commented Apr 22, 2020

Proposal: Create a rule(s) that can create a zip/tar archives from a set of sources. This would not be available at Bazel runtime as part of @tools (even though the source might go in //tools for lack of a better place). The capabilities would be to bundle up a list of sources and rename some files along the way. It would be stripped down version of the rules from pkg_tar.

Need:

  1. Some workspace tests often require building a tar archive as part of the test. These currently depend on pkg_tar from @tools/build_defs/pkg, which is going away (move rules_pkg out of Bazel #8857 ). https://github.com/bazelbuild/bazel/search?q=bazel_tools/tools/build_defs/pkg&unscoped_q=pkg_tar
    Rewriting them to depend on the rules_pkg distribution is possible, but introduces an external dependency buried in code. The package checksum ends up appearing in Java test code that is writing a WORKSPACE file. This is extremely ugly.
  2. Building the Bazel distribution makes frequent use of pkg_tar to bundle up some things. We currently depend on bazelbuild/rules_pkg for this. It looks like we could use the mini tool to eliminate an upstream dependency.

As an alternative, we could minimize chance of drift between rules_pkg in the build and tests by reusing the imported repo.

  • re-export the rules_pkg.tar.gz that we import into the Bazel WORKSPACE. (maybe an http_file duplicating the http_archive)
  • tests which need to import @rules_pkg into the WORKSPACE files they write will add a dep on this tarball.
@aiuto aiuto mentioned this issue Apr 22, 2020
Closed
@aiuto aiuto added the area-EngProd Bazel CI, infrastructure, bootstrapping, release, and distribution tooling label Apr 22, 2020
@aiuto aiuto mentioned this issue Jun 8, 2020
Closed
@philwo philwo added the team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website label Jun 15, 2020
@aiuto aiuto mentioned this issue Oct 28, 2020
Closed
@philwo philwo added P3 We're not considering working on this, but happy to review a PR. (No assignee) type: feature request and removed untriaged labels Dec 8, 2020
@aiuto aiuto changed the title Provide a tiny zip/tar utility for use in tests and packaging. Provide a tiny zip/tar utility for use in tests Apr 14, 2021
bazel-io pushed a commit that referenced this issue Apr 30, 2021
@aiuto @copybara-github
Remove xz compression from Bazel's build in pkg_tar.
19a4e6c 
- clean up Python 2/3 migration hacks
- #11183

RELNOTES:
Dropped fragile xz support from built in pkg_tar. Users requiring xz
compression should switch to bazlebuild/rules_pkg.
PiperOrigin-RevId: 371394411
@aiuto
Copy link
Contributor Author

aiuto commented Jun 3, 2021

My thinking as of late is to remove all the extra features from the internal pkg_tar and let it become the mini-tar.
Users of the esoteric ones will have to do the easy migration to rules_pkg

bazel-io pushed a commit that referenced this issue Jun 7, 2021
@aiuto @copybara-github
Remove more unneeded features from @bazel_tools/build_defs/pkg/pkg.bzl
0954af6 
Specifically:
- Remove the dependency on six
- Switch to pure python3
- Remove a capability to push raw data from a python program into a tar file. The capability was in an internal library, which no one should be using.

See: #11183

RELNOTES: None
PiperOrigin-RevId: 377908363
@steple steple mentioned this issue Mar 2, 2022
Closed
aiuto added a commit to aiuto/bazel that referenced this issue Mar 2, 2022
@aiuto
Create mini_tar and use it for our blackbox tests.
4c576b6 
bazelbuild#11183
@aiuto aiuto mentioned this issue Mar 2, 2022
Closed
@aiuto aiuto self-assigned this Mar 2, 2022
@aiuto
Copy link
Contributor Author

aiuto commented Mar 2, 2022

I picked this up again for now. Rough plan:

  • PR 1: create mini_tar using the remains of pkg_tar. Same build_tar, but few attributes.
  • migrate google off of bazel's pkg_tar to rules_pkg.pkg_tar
  • PR 2: drop more attributes from pkg_tar
  • PR 3: delete pkg.bzl
  • PR 4: consolidate archive.py into build_tar.py
  • PR 5: get rid of the shell tests.

It is very back burner, so it may take a few months.

@aiuto aiuto changed the title Provide a tiny zip/tar utility for use in tests Provide a tiny tar utility for use in tests Mar 6, 2022
@aiuto aiuto mentioned this issue Mar 15, 2022
Closed
bazel-io pushed a commit that referenced this issue Mar 26, 2022
@aiuto @copybara-github
Remove owners attribute from bazel legacy pkg_tar.
99ac541 
Also did some mini README cleanup (even though it should all be removed)

Advances: #11183

We can get to the mini-tar by slowly removing featuers from pkg_tar.

RELNOTES: None
PiperOrigin-RevId: 437462604
copybara-service bot pushed a commit that referenced this issue May 25, 2022
@aiuto @copybara-github
Create mini_tar() for use in our blackbox tests.
601fd1f 
This PR uses selected parts of tools/build_defs/pkg.  It is not the way I would have wrote it, but it is based on stuff that has been working for years, so I am disinclined to rewrite it totally. It is solely intended for use bazel's tests, and not distributed as part of the product.

It also contains some vestigial features from the old code that are only half enabled here - like adding symlinks. I want to keep that around in case I need it for the blackbox tests. If not, I can fully delete or enable it after I make that assessment.

Part of #11183

Next PR: Use it in src/test/java/com/google/devtools/build/lib/blackbox/tests/workspace.

Closes #15028.

PiperOrigin-RevId: 450908743
Change-Id: I7c393ab264924ef475e65321bdcd817b02cb0819
@github-actions GitHub Actions
Copy link

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

@github-actions github-actions bot added the stale Issues or PRs that are stale (no activity for 30 days) label Nov 25, 2023
@github-actions github-actions bot removed the stale Issues or PRs that are stale (no activity for 30 days) label Jan 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-EngProd Bazel CI, infrastructure, bootstrapping, release, and distribution tooling P3 We're not considering working on this, but happy to review a PR. (No assignee) team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website type: feature request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@philwo @aiuto