"twine upload" usually fails to upload .asc files

[warner]

On the most recent Foolscap release, I signed the sdist tarballs as usual, and tried to use twine to upload everything:

% python setup.py sdist --formats=zip,gztar bdist_wheel
% ls dist
foolscap-0.9.1-py2-none-any.whl     foolscap-0.9.1.tar.gz           foolscap-0.9.1.zip
% (gpg sign them all)
% ls dist
foolscap-0.9.1-py2-none-any.whl     foolscap-0.9.1.tar.gz           foolscap-0.9.1.zip
foolscap-0.9.1-py2-none-any.whl.asc foolscap-0.9.1.tar.gz.asc       foolscap-0.9.1.zip.asc
% python setup.py register
% twine upload dist/*

Twine uploaded the tar/zip/whl files, but ignored the .asc signatures, and the resulting pypi page doesn't show them either.

After some digging, I found that twine/upload.py upload() will only use pre-signed .asc files if the command was run like cd dist; twine upload *. It won't use them if it was run as cd dist; twine upload ./* or twine upload dist/*. The problem seems to be that the signatures dictionary is indexed by the basename of the signature files, while the lookup key is using the full (original) filename of the tarball/etc with ".asc" appended.

I think it might be simpler and safer to have the code just check for a neighboring .asc file inside the upload loop, something like:

for filename in uploads:
    package = PackageFile.from_filename(filename, comment)
    maybe_sig = package.signed_filename + ".asc"
    if os.path.exists(maybe_sig):
        package.gpg_signature = (os.path.basename(maybe_sig), sigdata)
    ...

I'll write up a patch for this. I started to look for a way of adding a test, but the code that looks for signatures happens deep enough in upload() that it'd need a oversized mock "Repository" class to exercise the .asc check without actually uploading anything. I'm not sure what the best way to approach the test would be.

[sigmavirus24]

Hey @warner, could you provide the output of twine --version and what operating system you're using? I'm curious if the glob is being interpreted by twine or the command line. We have handling in there for Windows users and others using things like tox to automate their release strategies. I'd like some more information so I know how to better debug this.

Thanks for reporting this

[warner]

It's on OS-X (10.10.5), using python 2.7.10 (from homebrew), so the shell is responsible for expanding the glob. I'm pretty sure I was using:

twine version 1.6.1 (pkginfo: 1.2.1, requests: 2.7.0, setuptools: 18.3.2)

While debugging it, I added a return just before the repository.upload(package), to avoid doing a real upload, and looked inside the repository object to see what files it knew about.

As best I could figure, the problem was a combination of a couple of things in 1.6.1, when I ran it with a command that expands to twine upload dist/foolscap-0.9.1.tar.gz dist/foolscap-0.9.1.tar.gz.asc:

upload() computes the list of signature files with basename(), so this will get foolscap-0.9.1.tar.gz.asc instead of dist/foolscap-0.9.1.tar.gz.asc:

    # Determine if the user has passed in pre-signed distributions
    signatures = dict(
        (os.path.basename(d), d) for d in dists if d.endswith(".asc")
    )

The package object is created here, where filename is the full (non-basenamed) dist/foolscap-0.9.1.tar.gz:

    for filename in uploads:
        package = PackageFile.from_filename(filename, comment)

and PackageFile.from_filename() and __init__ just do:

        self.signed_filename = self.filename + '.asc'

And then upload() tries to see if the expected signed_filename is in signatures:

        signed_name = package.signed_filename
        if signed_name in signatures:
            with open(signatures[signed_name], "rb") as gpg:
                package.gpg_signature = (signed_name, gpg.read())

but by that point it's comparing the path-bearing package.signed_filename against the basenamed keys of signatures, and they're not going to match.

Thanks for digging into this!

[meejah]

With twine 1.5.0, I (just) successfully made another upload of txtorcon that included signatures properly. The command-lines used are in the my Makefile: https://github.com/meejah/txtorcon/blob/master/Makefile#L101

I have not tried with other Twine versions.

[sigmavirus24]

This probably has something to do with the refactor that I did pre-1.6.0. Now that @warner has provided the debugging information, I'll try to release 1.6.2 tonight.

Thanks all!

[sigmavirus24]

@warner I released 1.6.2 a few seconds ago. Cheers!

[warner]

great, thanks!