package hash differs between operating systems due to file system case sensitivity #18089

mitchellh · 2023-11-23T06:03:22Z

Zig Version

0.12.0-dev.1647+325e0f5f0

Steps to Reproduce and Observed Behavior

.{
    .name = "test",
    .version = "0.1.0",
    .paths = .{""},
    .dependencies = .{
        .iterm2_themes = .{
            .url = "https://github.com/mbadolato/iTerm2-Color-Schemes/archive/aed2a8f0565a51913e8ddc1731f25bc71978a1e0.tar.gz",
        },
    },
}

On macOS: 12201fff83e91996f5c6abf51c8d0a54630926d81564f75681e1972575058d6476fc
On Linux: 12202a04dfdfc563f0c74a61559a549c194405f0ef562e5c20e32066bde0bbcd1020

Expected Behavior

Hashes should match. I'm not sure what is causing the problem yet.

The text was updated successfully, but these errors were encountered:

andrewrk · 2023-11-23T07:23:37Z

First step to troubleshooting this will be to run zig fetch --debug-hash https://github.com/mbadolato/iTerm2-Color-Schemes/archive/aed2a8f0565a51913e8ddc1731f25bc71978a1e0.tar.gz on each OS, and then diff the two outputted text files.

oldendick · 2023-11-23T08:47:30Z

diff -u iterm-debug-hash-macos.txt iterm-debug-hash-linux.txt 
--- iterm-debug-hash-macos.txt	2023-11-23 02:23:17
+++ iterm-debug-hash-linux.txt	2023-11-23 02:24:46
@@ -576,6 +576,8 @@
 file: b12ae24abcc570ee232cb3f3b52e329d7031128a7ad062c9590fd897da01d827: alacritty/catppuccin-mocha.yml
 file: e9b349dc1f185b091aedeef71918d4686e11fef7c97fa7f1999da607126aa3f9: alacritty/coffee_theme.yml
 file: 4b3227494faae53ae31d79a47808f0bd99eed99727417794a008383fbf2e8665: alacritty/cyberpunk.yml
+file: a2cb474b347a6adf5a3aeb8cbc0a0ba2ea0b1f7774188634ff531433434e57e8: alacritty/darkermatrix.yml
+file: 522281b9703073d6cdfa91048151cfec3607aa25224d007e669b05af05463206: alacritty/darkmatrix.yml
 file: 17b6b72f117e5ab584ee10e4117d154ccbb88b7385f0c5eccbfc067dd72c08c4: alacritty/deep.yml
 file: d8c24d86c8102b03864fd75e5b109fd62503af27325a6afa545b5cef18fbddc4: alacritty/duckbones.yml
 file: bc74eda647449e0effb0600193648a9d8c6c92123053c140e5f3f3c66d594963: alacritty/flexoki-dark.yml
@@ -596,6 +598,7 @@
 file: f9460bee42703abee47a4315cd5267caccc105087282aea2e2df7a1052cbd701: alacritty/jubi.yml
 file: 25165b9ed2bfdb6594aa12d2cd758177280a44891eaabeae6aa9768a5e497460: alacritty/kanagawabones.yml
 file: f2ded6bdd2ee86ae91b8669a7f7fa1960f7e30869e06ac828f5aca9ae03cc38e: alacritty/lovelace.yml
+file: 05b5ad88a0a4309806776f106bfcaea4fb3a6af2e6af477a80b8fc1e4a2186c7: alacritty/matrix.yml
 file: 688838a8c20bd2a504d7c97f5a5964a69c2fe483c4962a54baf82bef70af3d63: alacritty/midnight-in-mojave.yml
 file: 703030abe167129b2a306680b7592b771cf7e444c6ea0ea1f6be82e9866e3a44: alacritty/neobones_dark.yml
 file: b16fd79bb0d599fd24bd70a0c17e5f38a2f2e5002b36ad05eb14a1f22b57956c: alacritty/neobones_light.yml
@@ -8001,6 +8004,8 @@
 file: 723825d2f244f3a7e2a6e89dd3560573c74bf1863677d29017a8c7a30631ba70: vscode/catppuccin-mocha.json
 file: 080dce3c0e2db8deff9e2188c61eccc319adf70f0b651a622f5d4870b3363d82: vscode/coffee_theme.json
 file: 7fae2360d810859ce6e439d81dfa335e5c56e39b3893892f60404f09ee177eda: vscode/cyberpunk.json
+file: 107b278672c8062fbdc99395e088f02f7cf0947a352d6846831e80fe574c71c7: vscode/darkermatrix.json
+file: 1a59da5a1ed1bc8bf396307f000ffa25b59902a20b29cdff956c3110fcbf411f: vscode/darkmatrix.json
 file: e28045bf63466ca7d64678543614f69c1cd0dec68100c67713e470f4e13e31f0: vscode/deep.json
 file: d1dfa07bb7ba6b7ac65f4f99d55d16f5f34a926455dee4f0c10014428e6b2347: vscode/duckbones.json
 file: 388bb8062225896879405b359decb62981cb5c0babee070a51744d5d72669b65: vscode/flexoki-dark.json
@@ -8021,6 +8026,7 @@
 file: 9cdf866d0c174010b652c46ce86a91e790bde54d0e6ef1346c3f92b123ceead1: vscode/jubi.json
 file: 717e9e69d64962908337c827b0c2c1f21fcffefc27f87516fd8d7642dcc425c7: vscode/kanagawabones.json
 file: 788f321a66832d07097642d22d3e49b9c0d346f1b167c8c8f86591144dd1f832: vscode/lovelace.json
+file: 415d3b9aa10d1e175ae72c7966be0b609446b094201267c49ebd492674192a63: vscode/matrix.json
 file: 4c955ab8ea102dfddef6345c572620abcbe330fe94cee74426510a9510961064: vscode/midnight-in-mojave.json
 file: d654d67d2e224296a347f3ec70b960d7976c0537309deb27032875db054ddd0f: vscode/neobones_dark.json
 file: 0ddb80da587ca8d107609f8c4d8b59e84238317f01b4583f0cf8bd209df55b48: vscode/neobones_light.json
@@ -9289,4 +9295,4 @@
 file: 0ff2794fecfc090fe98c4e879bb86b1b2ab578055828ba9a513d15cbcaea62c7: xrdb/zenburned.xrdb
 file: ef008b8cb4f23378482040df8bffbc4ae30d77472992805a00287b9bd05a31aa: xrdb/zenwritten_dark.xrdb
 file: a9948901a3da22b27ab71f132a2f8aec8b573199efda01deb048e86c04b406e3: xrdb/zenwritten_light.xrdb
-12201fff83e91996f5c6abf51c8d0a54630926d81564f75681e1972575058d6476fc
+12202a04dfdfc563f0c74a61559a549c194405f0ef562e5c20e32066bde0bbcd1020

The mac side seems to show 6 fewer files. Upon looking at the tar, these files are included in the tar twice with different case sensitivity in the filename.

i.e.:

tar ztvf aed2a8f0565a51913e8ddc1731f25bc71978a1e0.tar.gz | egrep -i "alacritty/darkermatrix.yml|alacritty/darkmatrix.yml|alacritty/matrix.yml|vscode/darkermatrix.json|vscode/darkmatrix.json|vscode/matrix.json" | sort -k5
-rw-rw-r--  0 root   root      554 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/alacritty/Matrix.yml
-rw-rw-r--  0 root   root      554 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/alacritty/matrix.yml
-rw-rw-r--  0 root   root      558 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/alacritty/Darkmatrix.yml
-rw-rw-r--  0 root   root      558 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/alacritty/darkmatrix.yml
-rw-rw-r--  0 root   root      560 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/alacritty/Darkermatrix.yml
-rw-rw-r--  0 root   root      560 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/alacritty/darkermatrix.yml
-rw-rw-r--  0 root   root      933 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/vscode/Darkermatrix.json
-rw-rw-r--  0 root   root      933 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/vscode/Darkmatrix.json
-rw-rw-r--  0 root   root      933 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/vscode/Matrix.json
-rw-rw-r--  0 root   root      933 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/vscode/darkermatrix.json
-rw-rw-r--  0 root   root      933 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/vscode/darkmatrix.json
-rw-rw-r--  0 root   root      933 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/vscode/matrix.json

der-teufel-programming · 2023-11-23T09:11:25Z

I've found the same comparing Windows and Linux. Windows gives the same hash as MacOS and the missing files are the same as listed above

slonik-az · 2023-11-23T11:03:37Z

Should one allow OS specific hashes in zon dependency description? Otherwise one will be fighting these corner cases forever.

der-teufel-programming · 2023-11-23T11:47:36Z

This would be problematic if the files with the same case insensitive name, but different case sensitive name actually contained different data

marler8997 · 2023-11-23T14:10:41Z

Oooof...differences in case-sensitivity of file names strikes again.

My suggestion would be for Zig to detect and by default reject archives that aren't usable on all supported platforms. So in this case you'd get:

$ zig build
error: dependency `iterm2_themes` contains the following filename conflicts:

alacritty/Darkermatrix.yml
alacritty/darkermatrix.yml

alacritty/Darkmatrix.yml
alacritty/darkmatrix.yml

...

This solution might be "good enough" and will encourage developers to create archives that are more portable. That being said...if we do want to support these types of archives (not saying we should), we have some options. We could allow projects to opt-in to allowing these conflicts, i.e.

.{
    // ...
        .iterm2_themes = .{
            .url = "https://github.com/mbadolato/iTerm2-Color-Schemes/archive/aed2a8f0565a51913e8ddc1731f25bc71978a1e0.tar.gz",
            .allow_filename_casing_conflicts = true,
        }
}

Note this would mean the package becomes usable when building on a system where zig's package cache is on a case-insensitive filesystem. This would break cross-compilation to/from particular targets which is unfortunate. A better option could be to provide some sort of way to modify how the archive is extracted to resolve the filename conflicts, i.e.

.{
    // ...
        .iterm2_themes = .{
            .url = "https://github.com/mbadolato/iTerm2-Color-Schemes/archive/aed2a8f0565a51913e8ddc1731f25bc71978a1e0.tar.gz",
            .renames = &.{
                &.{ "alacritty/Darkmatrix.yml", "alacritty/darkmatrix2.yml" },
            },
        }
}

This means no longer having a requirement on a case-sensitive filesystem and the dependency is uniform across all platforms.

This being said, I think it's reasonable to start with simply rejecting these types of archives and allow authors to repackage them in a more portable form, after all they'd be the best ones to know the correct changes. Enabling new exotic cross-compilation can be used as a motivator for these kinds of changes. We can think about whether we'd like to make it easy to patch archives separately.

P.S. It seems weird that Windows/macOS, the 2 big OS's backed by large companies happen to be the problem children that don't use case-sensitive filesystems by default. I expect this from Windows but macOS?

xdBronch · 2023-11-23T14:21:19Z

once #14597 is solved one could conditionally allow packages with conflicts like these on platforms where its allowed

mitchellh · 2023-11-23T16:21:00Z

I've also made an upstream PR to fix the source repo: mbadolato/iTerm2-Color-Schemes#425

We probably should still figure out the proper behavior for Zig though in these cases.

nektro · 2023-11-23T18:44:25Z

the tar unpacker should error on windows/macos that there's name conflicts when creating the files.

andrewrk · 2023-11-23T19:40:57Z

The first step is to implement errors in the tar unpacking code when a tar file cannot be extracted correctly due to the file system being incompatible with the tar file. This is already the case with, for example, symlinks on Windows with default settings. It also needs to be implemented in the git fetching code.

Just like with symlinks, errors with these particular files should be ignored if they are not included by the package manifest (i.e. filtered with the paths field).

Just like with symlinks, this will cause URLs to work on some operating systems but fail to be fetched on others.

What @marler8997 is describing is useful functionality, but I think it should be a separate linting feature. Here's a laundry list of features such a linter could provide:

zig build --lint - does everything that zig build --fetch already does and also...

report when any unfiltered symlinks are used
report when any unfiltered case sensitivity name collisions occur
report how many files and bytes are wasted due to being fetched but then filtered
report any new versions of dependencies that are detected as being available
report use of dependencies with versions less than 1.0.0
report any occurrence of redundant dependencies (same package included multiple times due to different incompatible versions)

Fail with error if file already exists. File is not silently overwritten but an error is raised. Fixes: ziglang#18089

Like in issue ziglang#18089, this tar contains, same file name in two case sensitive name version. Unpack should fail on case insensitive file systems and succeed on case sensitive. $ tar tvf 18089.tar 18089/ 18089/alacritty/ 18089/alacritty/darkermatrix.yml 18089/alacritty/Darkermatrix.yml

andrewrk · 2024-02-26T20:23:31Z

In order to close this issue:

The first step is to implement errors in the tar unpacking code when a tar file cannot be extracted correctly due to the file system being incompatible with the tar file.
It also needs to be implemented in the git fetching code.
Just like with symlinks, errors with these particular files should be ignored if they are not included by the package manifest (i.e. filtered with the paths field).

Fail with error if file already exists. File is not silently overwritten but an error is raised. Fixes: ziglang#18089

Like in issue ziglang#18089, this tar contains, same file name in two case sensitive name version. Unpack should fail on case insensitive file systems and succeed on case sensitive. $ tar tvf 18089.tar 18089/ 18089/alacritty/ 18089/alacritty/darkermatrix.yml 18089/alacritty/Darkermatrix.yml

mitchellh added the bug Observed behavior contradicts documented or intended behavior label Nov 23, 2023

mitchellh mentioned this issue Nov 23, 2023

Get Ghostty on the Zig Package Manager #16672

Closed

6 tasks

mitchellh changed the title ~~build: hash of tar.gz differs between macOS and Linux~~ package manager: hash of tar.gz differs between macOS and Linux Nov 23, 2023

andrewrk added this to the 0.12.0 milestone Nov 23, 2023

andrewrk added the contributor friendly This issue is limited in scope and/or knowledge of Zig internals. label Nov 23, 2023

andrewrk added the zig build system std.Build, the build runner, `zig build` subcommand, package management label Nov 23, 2023

andrewrk changed the title ~~package manager: hash of tar.gz differs between macOS and Linux~~ package hash differs between operating systems due to file system case sensitivity Nov 23, 2023

jacwil mentioned this issue Jan 29, 2024

Proposal for consistent dependency package behavior across platforms #18728

Closed

amusingimpala75 mentioned this issue Feb 2, 2024

Hash mismatch for ziglyph rockorager/libvaxis#1

Closed

schmee mentioned this issue Feb 10, 2024

Local package hash different between Windows and WSL Linux #18891

Closed

ianic added a commit to ianic/zig that referenced this issue Feb 24, 2024

std.tar don't overwrite existing file

b84301c

Fail with error if file already exists. File is not silently overwritten but an error is raised. Fixes: ziglang#18089

ianic mentioned this issue Feb 25, 2024

std.tar don't overwrite files on unpack #19081

Merged

andrewrk closed this as completed in #19081 Feb 26, 2024

andrewrk reopened this Feb 26, 2024

ianic mentioned this issue Mar 16, 2024

package: skip unpack errors on paths excluded by manifest #19324

Closed

RossComputerGuy pushed a commit to ExpidusOS/zig that referenced this issue Mar 20, 2024

std.tar don't overwrite existing file

948c3df

Fail with error if file already exists. File is not silently overwritten but an error is raised. Fixes: ziglang#18089

ianic mentioned this issue Mar 30, 2024

thanks ianprime0509/pathological-packages#1

Closed

ianic mentioned this issue Mar 31, 2024

package manager: filter unpack errors on paths excluded by manifest #19500

Merged

andrewrk closed this as completed in #19500 Apr 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

package hash differs between operating systems due to file system case sensitivity #18089

package hash differs between operating systems due to file system case sensitivity #18089

mitchellh commented Nov 23, 2023

andrewrk commented Nov 23, 2023

oldendick commented Nov 23, 2023

der-teufel-programming commented Nov 23, 2023

slonik-az commented Nov 23, 2023 •

edited

Loading

der-teufel-programming commented Nov 23, 2023

marler8997 commented Nov 23, 2023 •

edited

Loading

xdBronch commented Nov 23, 2023

mitchellh commented Nov 23, 2023

nektro commented Nov 23, 2023

andrewrk commented Nov 23, 2023 •

edited

Loading

andrewrk commented Feb 26, 2024 •

edited

Loading

package hash differs between operating systems due to file system case sensitivity #18089

package hash differs between operating systems due to file system case sensitivity #18089

Comments

mitchellh commented Nov 23, 2023

Zig Version

Steps to Reproduce and Observed Behavior

Expected Behavior

andrewrk commented Nov 23, 2023

oldendick commented Nov 23, 2023

der-teufel-programming commented Nov 23, 2023

slonik-az commented Nov 23, 2023 • edited Loading

der-teufel-programming commented Nov 23, 2023

marler8997 commented Nov 23, 2023 • edited Loading

xdBronch commented Nov 23, 2023

mitchellh commented Nov 23, 2023

nektro commented Nov 23, 2023

andrewrk commented Nov 23, 2023 • edited Loading

andrewrk commented Feb 26, 2024 • edited Loading

slonik-az commented Nov 23, 2023 •

edited

Loading

marler8997 commented Nov 23, 2023 •

edited

Loading

andrewrk commented Nov 23, 2023 •

edited

Loading

andrewrk commented Feb 26, 2024 •

edited

Loading