consider git metadata on 'create'

[refack]

Refs: https://mobile.twitter.com/izs/status/1115332210381299712

When checking out a git repository in Windows, some FS metadata is lost due to mismatch between the WIN32 and POSIX APIs. Specifically the file mode (but also uid, gid, and some of the time stamps).

refael@refaelux MINGW64 /d/code/node (mode-test)
$ ls -la r.sh
-rw-r--r-- 1 refael 197609 0 Apr  8 15:57 r.sh
^^^^^^^^^^

But git still stores this info

refael@refaelux MINGW64 /d/code/node (mode-test)
$ git ls-files r.sh --stage
100755 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0       r.sh
^^^^^^

The code at https://github.com/npm/node-tar/blob/d7268f65123780031e58120f55f28b87ae0c9608/lib/write-entry.js#L331-L334
Could consider this info to "enrich" the created tarball.

[refack]

P.S. the most interesting use case is losing the +x from script files when doing npm publish from a worktree checked outed on Windows.

[boneskull]

This bit Mocha yesterday when we published from Windows for the first time. Someone using npm install mocha --no-bin-links found that the executable files no longer had the executable flag. My workaround was to re-publish on macOS.

[isaacs]

If anyone wants to help move this along, the next step is to research and document how git stores this. Is there a specific file in the .git folder we could check for this info?

(Calling git cli in a child process is completely out of the question.)

[refack]

Big 🔫s - https://github.com/isomorphic-git/isomorphic-git
heat-seeking-🚀 - https://github.com/sbp/gin/blob/master/gin

Seems like .git/index is a fairly simple binary stream. Pulling a snippet out of gin

    with open(.git/index, "rb") as f:

        def read(format):
            # "All binary numbers are in network byte order."
            # Hence "!" = network order, big endian
            format = "! " + format
            bytes = f.read(struct.calcsize(format))
            return struct.unpack(format, bytes)[0]

        index = collections.OrderedDict()

        # 4-byte signature, b"DIRC"
        index["signature"] = f.read(4).decode("ascii")
        check(index["signature"] == "DIRC", "Not a Git index file")

        # 4-byte version number
        index["version"] = read("I")
        check(index["version"] in {2, 3},
            "Unsupported version: %s" % index["version"])

        # 32-bit number of index entries, i.e. 4-byte
        index["entries"] = read("I")

        yield index

        for n in range(index["entries"]):
            entry = collections.OrderedDict()
            entry["entry"] = n + 1
            entry["ctime_seconds"] = read("I")
            entry["ctime_nanoseconds"] = read("I")
            entry["mtime_seconds"] = read("I")
            entry["mtime_nanoseconds"] = read("I")
            entry["dev"] = read("I")
            entry["ino"] = read("I")
            # 4-bit object type, 3-bit unused, 9-bit unix permission
            entry["mode"] = read("I")
            if pretty:
                entry["mode"] = "%06o" % entry["mode"]
            entry["uid"] = read("I")
            entry["gid"] = read("I")
            entry["size"] = read("I")
            entry["sha1"] = binascii.hexlify(f.read(20)).decode("ascii")
            entry["flags"] = read("H")
...

then just need to seek till the end of the entry.
The magic sauce is python's struct

[refack]

read("I") -> buf.readInt32LE()
read("H") -> buf.readInt16LE()

[isaacs]

Working JS hacky example, could definitely do with a cleanup, but this seems like it works: https://gist.github.com/isaacs/55ad9e3338912c008804f69dbe855ed1

[boneskull]

@isaacs I know you won't be interested in pulling it in to this project, but Kaitai provides a declarative format for parsing binary blobs which can then generate code. I've used it before, and it seems suitable for something like this (in fact, a definition file might already exist for the git index format). the docs are a bit incomplete, but still might be fun to play with if you have nothing better to do.

advice, if you do look at it: stay away from the js implementation of the tool; it's incomplete, just use the reference (scala?) implementation

[boneskull]

imo the project would be better if it just output some sort of object code format that could then be piped into custom code generators, b/c the generated JS looks like what you'd get if you asked an career java dev to sit down and write JS

[boneskull]

here's the description file I created to parse multimeter output over serial

[isaacs]

@boneskull yeah... I'm probably not gonna use that.

Re: #210 (comment), I think npm probably ought to chmod the bins on install (or better yet, just update their mode at unpack time before they're written to disk), even if --no-bin-links is set. That might be hairy to get into v6, but I can make that change in Pacote v10 for npm 7 pretty easily.

[isaacs]

Ok, so, npm v7 will make the Mocha-published-from-windows-installed-without-binlinks a total non-issue.

Having looked into this git stuff, I'm kind of uncomfortable with the added complexity and fs ops required to treat the git dir cache as a source of truth. It was a fun little hack to learn how to parse it, but it feels like a wart in node-tar.

• git-archive already exists if you want to put git info into a tarball.
• the facts "on the ground" of the file system might not match what's in the git index and that may be what you want a lot of the time!
• the solution to the original npm footgun is now solved in an npm library (albeit behind a breaking change, but this would've likely been, anyway).
• .git folders can appear anywhere within the tree, the path parsing is relative to the .git folder rather than the root of the archive, etc, etc. It's all solveable, but it adds a lot of stuff.

So I'm gonna close this issue for now.

If anyone is really desperate for this kind of functionality (informing a tarball's header info based on a git index), you can get at the entry stats on the way into the archive by adding a filter option to tar.c. (That's how pacote is setting the executable bit on bins now.)