Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

introduce the zig fetch subcommand and symlink support in zig packages #17363

Merged
merged 6 commits into from
Oct 3, 2023

Conversation

andrewrk
Copy link
Member

@andrewrk andrewrk commented Oct 2, 2023

zig fetch [options] <url>
zig fetch [options] <path>

Fetches a package which is found at <url> or <path> into the global cache directory, printing the package hash to stdout.

Closes #16972
Related to #14280

Additionally, this branch:

Merge Checklist

  • support symlinks in Package/git.zig
  • more kinds of symlink metadata in std.tar eagerly awaiting @vesim987's upcoming PR for this
  • handle Diagnostics from std.tar

@andrewrk
Copy link
Member Author

andrewrk commented Oct 2, 2023

cc @ianprime0509 - would you be willing to add the git.zig logic to add symlink support here?

@ianprime0509
Copy link
Sponsor Contributor

Awesome! Sure, I'll take a closer look at this after work and send a patch to add support for this in Git.

There are no functional changes in this commit.
zig fetch [options] <url>
zig fetch [options] <path>

Fetches a package which is found at <url> or <path> into the global
cache directory, printing the package hash to stdout.

Closes #16972
Related to #14280

Additionally, this commit:

* Adds uncompressed .tar support to package fetching
* Introduces symlink support to package fetching
* Package: use std.tar diagnostics to give detailed error messages
* std.tar: add diagnostic for unsupported file type
@andrewrk
Copy link
Member Author

andrewrk commented Oct 3, 2023

more kinds of symlink metadata in std.tar

eagerly awaiting @vesim987's upcoming PR for this

@andrewrk
Copy link
Member Author

andrewrk commented Oct 3, 2023

Note that zig fetch on a directory will be a bit useless until #14311 is resolved, because the package will potentially include files from version control which would not make it into a tarball.

@ianprime0509
Copy link
Sponsor Contributor

@andrewrk not sure what would be most convenient for you (maybe I should just send a follow-up PR once this is merged), but here's a patch to integrate this with git.zig as well: https://gist.github.com/ianprime0509/f097b73a77e0c9412fe4a11824bd169d

This is able to fetch the dependency in this PR: kristoff-it/orca-ui-hello-zig#1

⬢[ian@toolbox orca-ui-hello-zig]$ ~/src/zig/build/stage4/bin/zig build
Fetch Packages... orca [9MiB] /var/home/ian/src/tmp/orca-ui-hello-zig/build.zig.zon:8:20: error: dependency is missing hash field
            .url = "git+https://git.handmade.network/hmn/orca.git#c4a1ad52a07bae2e6710c38f41a3c64ed8f93136",
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
note: expected .hash = "1220a913fdaac3962e4331bd71d729b70da9ecfa2a959d62d6c491f18b4619cdca61",

Sample when the underlying filesystem doesn't support symlinks:

⬢[ian@toolbox orca-ui-hello-zig]$ ZIG_GLOBAL_CACHE_DIR=$HOME/src/tmp/fat.mnt ~/src/zig/build/stage4/bin/zig build
Fetch Packages... orca [8MiB] Checkout... /var/home/ian/src/tmp/orca-ui-hello-zig/build.zig.zon:8:20: error: unable to unpack packfile
            .url = "git+https://git.handmade.network/hmn/orca.git#c4a1ad52a07bae2e6710c38f41a3c64ed8f93136",
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
note: unable to create symlink from 'src/ext/wasm3/platforms/android/app/src/main/cpp/m3' to '../../../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/arduino/lib/wasm3' to '../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/bluepill/lib/wasm3/src' to '../../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/esp32-idf-wasi/main/wasm3' to '../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/esp32-pio/lib/wasm3/src' to '../../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/esp8266/lib/wasm3' to '../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/fomu/src/wasm3' to '../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/hifive1/lib/wasm3' to '../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/particle/lib/wasm3/src' to '../../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/wm_w600/wasm3' to '../../../source': AccessDenied
note: unable to create symlink from 'tests/files/data/jail/dir_escape' to '..': AccessDenied
note: unable to create symlink from 'tests/files/data/jail/file_escape' to '../regular.txt': AccessDenied
note: unable to create symlink from 'tests/files/data/symlink' to 'regular.txt': AccessDenied

@andrewrk
Copy link
Member Author

andrewrk commented Oct 3, 2023

If you're fine with it, I can cherry-pick from your zig fork into this branch

@ianprime0509
Copy link
Sponsor Contributor

That sounds like an easier idea, I just pushed it up here: https://github.com/ianprime0509/zig/tree/tar-symlinks

@andrewrk
Copy link
Member Author

andrewrk commented Oct 3, 2023

Commit looks great 👍

@andrewrk andrewrk merged commit df4853a into master Oct 3, 2023
10 checks passed
@andrewrk andrewrk deleted the tar-symlinks branch October 3, 2023 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

build.zig.zon improve workflow to update dependency hash std.tar: handle tar files with symbolic links
2 participants