add inclusion/exclusion directives to build.zig.zon for deciding which files belong the package

[andrewrk]

Extracted from #14265.

The motivation for doing this is mirrors (#14291). Idea being that if you fetch a tarball, perhaps it does not contain any superfluous files, while the same package fetched via git protocol (#14295) does have some extra files that are not intended to be part of the package. Both URLs should work, and their hashes should be identical. This means some files should be excluded from the package.

I could see this going one of two ways:

...with exclusions:

.{
    // ...
    .excluded_files = .{
        ".gitignore",
        "/zig-cache",
        "build-*",
    },
    // ...
}

...or with inclusions:

.{
    // ...
    .included_files = .{
        "src/*.zig",
        "build.zig.zon",
    },
    // ...
}

Two open questions:

• should it be via inclusions, via exclusions, or either one is allowed?
• what should the default be, if no rules are present in the manifest file?

Either way, here is how fetching a package works:

1. Fetch URL from any mirror into a directory of files in a temp directory.
2. Apply inclusion/exclusion rules by deleting files from the file system.
3. Hash each file in the full tree independently.
4. Produce final hash from the hashes of each file, sorted by name (related: zig build package hashes are missing path names and executable bit for each file #14308).
5. Rename temp directory into the global package cache into a directory named after the hash.

The guarantee should be upheld that if the hash matches, the file system should match exactly. This means that, since empty directories are not included in the hash, empty directories should be removed by step 2, along with the inclusion/exclusion rules.

[deflock]

It's time for .zigignore

[ifreund]

It seems to me that the answers to these open questions that are most likely to guide users towards creating packages without unintended files are:

• Only allow inclusions
• if there are no rules, produce an empty package or perhaps just print an error.

[silversquirl]

I think @ifreund's suggestion matches Zig's design philosophy of "make the Right Way easy and the Wrong Way hard" (represented in the Zen as "Communicate intent precisely", "Edge cases matter" and "Only one obvious way to do things")

I do wonder whether being able to exclude things might still be useful, eg. for a directory containing things you want, with one file or subdirectory that you don't. However, I can't come up with an example that wouldn't be better solved by reorganizing the project, so that may not actually be a problem.

[andrewrk]

Additional GitHub search keywords for this issue: blacklist, whitelist, include, exclude, ignore

[andrewrk]

One reason to make it based on inclusion rules rather than exclusion rules would be that if some method of file delivery happens to add extra files, these will not get picked up and screw up the hash. For example, maybe somebody tries to make a mirror, but they accidentally add .DS_Store, or doing a fetch from version control has something unexpected in there.

If it were based on exclusion rules, then it would make sense to allow exclusion rules at the depender's side of things along with the URL and hash specification. We can avoid this and keep things simpler with inclusion rules.

[andrewrk]

Landed in f7bc55c.

Follow-up issue:

• package fetching: elide errors from unpacking that relate to excluded file paths #17460

This is currently implemented as a required field for the root package, but the rest of the dependency tree is allowed to omit the paths field; all files are considered to be included.