Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nuget push binary caching fails for python #23053

Closed
anders-wind opened this issue Feb 11, 2022 · 18 comments
Closed

Nuget push binary caching fails for python #23053

anders-wind opened this issue Feb 11, 2022 · 18 comments
Assignees
@BillyONeal @JonLiu1993
Labels
requires:discussion

Comments

@anders-wind
Copy link
Contributor

Describe the bug
I'm trying to use the github nuget binary caching integration and while it works for most of our package it fails for python (dependency of pybind11)

Pushing python3_x64-linux.3.10.1-vcpkgc87aa8919988de817c495b22160d65c4accc42bdcfffbd763e4566758c070e28.nupkg to 'https://nuget.pkg.github.com/my-org'...
  PUT https://nuget.pkg.github.com/my-org/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/my-org/'. The request will now be retried.
Error while copying content to a stream.
  Unable to read data from the transport connection: The socket has been shut down.
  The socket has been shut down
  PUT https://nuget.pkg.github.com/my-org/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/my-org/'. The request will now be retried.
Error while copying content to a stream.
  Unable to read data from the transport connection: The socket has been shut down.
  The socket has been shut down
  PUT https://nuget.pkg.github.com/my-org/
Error while copying content to a stream.
  Unable to read data from the transport connection: The socket has been shut down.
  The socket has been shut down

I ran cmake with: -DVCPKG_INSTALL_OPTIONS=--debug

Here's our manifest

{
    "name": "epex",
    "version-semver": "0.0.1",
    "dependencies": [
        "nlohmann-json",
        "frozen",
        "pybind11",
        {
            "name": "abseil",
            "features": [
                "cxx17"
            ],
            "version>=": "20211102.1"
        },
        {
            "name": "fmt",
            "version>=": "8.1.1"
        },
        "spdlog",
        "gtest",
        "benchmark"
    ],
    "builtin-baseline": "c0d667ad6bb2d68890f33ded8b2e5bc182ab01fb"
}

and I have the following in my github action:

VCPKG_BINARY_SOURCES: 'default;nuget,GitHub,readwrite'

Again it works for most packages, but not for python3.

Environment

  • OS: Linux ubuntu-latest
  • Compiler: clang-10

To Reproduce
setup nuget credentials in github actions and configure vcpkg to use it.
add pybind11 as vcpkg dependency in manifest.
run cmake and see cache pushing fails.

Expected behavior
python3 dependency would be correctly pushed to the binary cache

Failure logs

Building package python3[core]:x64-linux...
-- Installing port from location: /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/versioning_/versions/python3/c9b705608afc0485c03effb5aaa439f027b20af4
[DEBUG] 139823004247872: popen(/usr/local/bin/cmake "-DALL_FEATURES=deprecated-win7-support;" -DCURRENT_PORT_DIR=/home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/versioning_/versions/python3/c9b705608afc0485c03effb5aaa439f027b20af4 -D_HOST_TRIPLET=x64-linux -DFEATURES=core -DPORT=python3 -DVCPKG_USE_HEAD_VERSION=0 -D_VCPKG_DOWNLOAD_TOOL=BUILT_IN -D_VCPKG_EDITABLE=0 -D_VCPKG_NO_DOWNLOADS=0 -DCMD=BUILD -DDOWNLOADS=/home/runner/work/EPEXcpp/EPEXcpp/vcpkg/downloads -DTARGET_TRIPLET=x64-linux -DTARGET_TRIPLET_FILE=/home/runner/work/EPEXcpp/EPEXcpp/vcpkg/triplets/x64-linux.cmake -DVCPKG_BASE_VERSION=2022-02-01 -DVCPKG_CONCURRENCY=3 -DVCPKG_PLATFORM_TOOLSET=external -DGIT=/usr/bin/git -DVCPKG_ROOT_DIR=/home/runner/work/EPEXcpp/EPEXcpp/vcpkg -DPACKAGES_DIR=/home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages -DBUILDTREES_DIR=/home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees -D_VCPKG_INSTALLED_DIR=/home/runner/work/EPEXcpp/EPEXcpp/build/temp.linux-x86_64-3.8/vcpkg_installed -DDOWNLOADS=/home/runner/work/EPEXcpp/EPEXcpp/vcpkg/downloads -DVCPKG_MANIFEST_INSTALL=OFF -P /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/scripts/ports.cmake 2>&1)
-- Downloading https://github.com/python/cpython/archive/v3.10.1.tar.gz -> python-cpython-v3.10.1.tar.gz...
-- Extracting source /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/downloads/python-cpython-v3.10.1.tar.gz
-- Applying patch 0001-static-library.patch
-- Applying patch 0002-use-vcpkg-zlib.patch
-- Applying patch 0003-devendor-external-dependencies.patch
-- Applying patch 0004-dont-copy-vcruntime.patch
-- Applying patch 0005-only-build-required-projects.patch
-- Using source at /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/python3/src/v3.10.1-7ec7e9e4c8.clean
-- Getting CMake variables for x64-linux-dbg
-- Getting CMake variables for x64-linux-rel
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Installing x64-linux-dbg
-- Building x64-linux-rel
-- Installing x64-linux-rel
-- Installing: /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages/python3_x64-linux/share/python3/copyright
-- Fixing pkgconfig file: /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages/python3_x64-linux/lib/pkgconfig/python-3.10-embed.pc
-- Fixing pkgconfig file: /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages/python3_x64-linux/lib/pkgconfig/python-3.10.pc
-- Fixing pkgconfig file: /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages/python3_x64-linux/debug/lib/pkgconfig/python-3.10-embed.pc
-- Fixing pkgconfig file: /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages/python3_x64-linux/debug/lib/pkgconfig/python-3.10.pc
-- Installing: /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages/python3_x64-linux/share/python3/usage
-- Installing cmake wrappers
[DEBUG] 139823004247872: cmd_execute_and_stream_data() returned 0 after 237497821 us
-- Performing post-build validation
-- Performing post-build validation done
[DEBUG] system(cd /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages/python3_x64-linux && zip --quiet -y -r /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/python3/x64-linux.zip *)
[DEBUG] cmd_execute() returned 0 after 14252988 us
Stored binary cache: /home/runner/.cache/vcpkg/archives/c8/c87aa8919988de817c495b22160d65c4accc42bdcfffbd763e4566758c070e28.zip
[DEBUG] 139823004247872: popen(/usr/bin/mono /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/downloads/tools/nuget-5.11.0-linux/nuget.exe pack /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/python3/x64-linux.nuspec -OutputDirectory /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees -NoDefaultExcludes -ForceEnglishOutput -NonInteractive 2>&1)
[DEBUG] 139823004247872: cmd_execute_and_stream_data() returned 0 after 16391572 us
Attempting to build package from 'x64-linux.nuspec'.
WARNING: NU5103: The folder 'lib/pkgconfig/python-3.10-embed.pc' under 'lib' is not recognized as a valid framework name or a supported culture identifier. Rename it to a valid framework name or culture identifier.
WARNING: NU5103: The folder 'lib/pkgconfig/python-3.10.pc' under 'lib' is not recognized as a valid framework name or a supported culture identifier. Rename it to a valid framework name or culture identifier.
WARNING: NU5103: The folder 'lib/python3.10/enum.py' under 'lib' is not recognized as a valid framework name or a supported culture identifier. Rename it to a valid framework name or culture identifier.
WARNING: NU5103: The folder 'lib/python3.10/turtle.py' under 'lib' is not recognized as a valid framework name or a supported culture identifier. Rename it to a valid framework name or culture identifier.
WARNING: NU5103: The folder 'lib/python3.10/xdrlib.py' under 'lib' is not recognized as a valid framework name or a supported culture identifier. Rename it to a valid framework name or culture identifier.
... (a million lines like the ones above)
...
...
...
WARNING: NU5123: The file 'debug/lib/python3.10/site-packages/pip/_vendor/urllib3/contrib/_securetransport/__pycache__/__init__.cpython-310.pyc' path, name, or both are too long. Your package might not work without long file path support. Please shorten the file path or file name.
WARNING: NU5128: Some target frameworks declared in the dependencies group of the nuspec and the lib/ref folder do not have exact matches in the other location. Consult the list of actions below:
- Add a dependency group for pkgconfig0.0 to the nuspec
- Add a dependency group for python3.100.0 to the nuspec
Successfully created package '/home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/python3_x64-linux.3.10.1-vcpkgc87aa8919988de817c495b22160d65c4accc42bdcfffbd763e4566758c070e28.nupkg'.
Uploading binaries for python3:x64-linux to NuGet source GitHub.
[DEBUG] 139823004247872: popen(/usr/bin/mono /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/downloads/tools/nuget-5.11.0-linux/nuget.exe push /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/python3_x64-linux.3.10.1-vcpkgc87aa8919988de817c495b22160d65c4accc42bdcfffbd763e4566758c070e28.nupkg -ForceEnglishOutput -Timeout 100 -Source GitHub -NonInteractive 2>&1)
[DEBUG] 139823004247872: cmd_execute_and_stream_data() returned 1 after  3021775 us
WARNING: No API Key was provided and no API Key could be found for 'https://nuget.pkg.github.com/twig-energy'. To save an API Key for a source use the 'setApiKey' command.
Pushing python3_x64-linux.3.10.1-vcpkgc87aa8919988de817c495b22160d65c4accc42bdcfffbd763e4566758c070e28.nupkg to 'https://nuget.pkg.github.com/twig-energy'...
  PUT https://nuget.pkg.github.com/twig-energy/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/twig-energy/'. The request will now be retried.
Error while copying content to a stream.
  Unable to read data from the transport connection: The socket has been shut down.
  The socket has been shut down
  PUT https://nuget.pkg.github.com/twig-energy/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/twig-energy/'. The request will now be retried.
Error while copying content to a stream.
  Unable to read data from the transport connection: The socket has been shut down.
  The socket has been shut down
  PUT https://nuget.pkg.github.com/twig-energy/
Error while copying content to a stream.
  Unable to read data from the transport connection: The socket has been shut down.
  The socket has been shut down
Pushing NuGet to GitHub failed. Use --debug for more information.
Installing package python3[core]:x64-linux...
Elapsed time for package python3:x64-linux: 4.568 min
Starting package 17/18: pybind11:x64-linux
Installing package pybind11[core]:x64-linux...
Elapsed time for package pybind11:x64-linux: 13.81 ms
Starting package 18/18: spdlog:x64-linux
Installing package spdlog[core]:x64-linux...
Elapsed time for package spdlog:x64-linux: 26.78 ms

Total elapsed time: 4.888 min
@anders-wind
Copy link
Contributor Author

Screenshot 2022-02-11 at 14-50-41 Slack Casper twig energy
And we should have plenty of space etc

@mathisloge
Copy link
Contributor

I think this might be a thing: To save an API Key for a source use the 'setApiKey' command.

https://docs.microsoft.com/de-de/nuget/reference/cli-reference/cli-ref-setapikey

      - name: "Setup NuGet apikey"
        shell: "bash"
        run: >
          `./vcpkg/vcpkg fetch nuget | tail -n 1`
          setapikey "<NUGET_REGISTRY_PAT>" -Source "<NUGET_REGISTRY>"

@anders-wind
Copy link
Contributor Author

But why would it be required for pushing python3 but not the 17 other packages which are successfully pushed?

@JackBoosY
Copy link
Contributor

cc @BillyONeal

@BillyONeal
Copy link
Member

For helping investigation: python3 is 106MB, and ~35MB zipped.
image

Unfortunately it is blowing up inside the command

/usr/bin/mono /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/downloads/tools/nuget-5.11.0-linux/nuget.exe push /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/python3_x64-linux.3.10.1-vcpkgc87aa8919988de817c495b22160d65c4accc42bdcfffbd763e4566758c070e28.nupkg -ForceEnglishOutput -Timeout 100 -Source GitHub -NonInteractive 2>&1

meaning our options for direct diagnosis are limited.

@anders-wind
Copy link
Contributor Author

Is there anything I can do to provide more input? I am not sure what the requires:discussion label entails

@anders-wind
Copy link
Contributor Author

NuGet/Home#9775 (comment)
^seems relevant, apparently using api key is more stable than username/password. Is this something we could get for vcpkg?

@mathisloge
Copy link
Contributor

https://docs.microsoft.com/de-de/nuget/reference/cli-reference/cli-ref-setapikey

      - name: "Setup NuGet apikey"
        shell: "bash"
        run: >
          `./vcpkg/vcpkg fetch nuget | tail -n 1`
          setapikey "<NUGET_REGISTRY_PAT>" -Source "<NUGET_REGISTRY>"

@anders-wind have you tried this?

@BillyONeal
Copy link
Member

Related, maybe: #22191

@JackBoosY
Copy link
Contributor

Hi guys, can anyone test whether #22191 works?

Thanks.

@anders-wind
Copy link
Contributor Author

Im trying out the setapikey setup. Im getting Your request could not be authenticated by the GitHub Packages service. Please ensure your access token is valid and has the appropriate scopes configured.

        mono `./vcpkg/vcpkg fetch nuget | tail -n 1` sources add -source "https://nuget.pkg.github.com/twig-energy/index.json" -name "GitHub";
        mono `./vcpkg/vcpkg fetch nuget | tail -n 1` setapikey "${{ secrets.PACKAGES_ACCESS_TOKEN }}" -source "https://nuget.pkg.github.com/twig-energy/index.json";

The secrets.PACKAGES_ACCESS_TOKEN is a personal access token from my account. I also tried with secrets.GITHUB_TOKEN

@mathisloge
Copy link
Contributor

If the packages are stored in a different repo, your PAT need to have the read and write packages persmissions.

If it is the same repo, github token works,but might fail if multiple repos are using binary caching if there isnt a global repo to store the build packages

@anders-wind
Copy link
Contributor Author

image
These are the permissions I gave the key.
The path I'm pointing to is just the organization url and not the entire repo url. Should I change that?

@mathisloge
Copy link
Contributor

Yeah, you need a repo as a location. I recommend just to create a new repo e.g. vcpkg-cache and use that repo everywhere. That works quite well with GHA.

Permissions are looking good.

@anders-wind
Copy link
Contributor Author

Okay ill try that :)

@anders-wind
Copy link
Contributor Author

anders-wind commented Feb 15, 2022 •
edited
Loading

Okay, I got the setapikey working - it actually enables consistent pushes, but for whatever reason it does not provide neccesary permissions to download from the registry. So I need to input both username/password and the apikey

        mono `./vcpkg/vcpkg fetch nuget | tail -n 1` sources add -source "https://nuget.pkg.github.com/twig-energy/index.json" -storepasswordincleartext -name "GitHub" -username "twig-energy" -password "${{ secrets.GITHUB_TOKEN }}"
        mono `./vcpkg/vcpkg fetch nuget | tail -n 1` setapikey "${{ secrets.PACKAGES_ACCESS_TOKEN }}" -source "https://nuget.pkg.github.com/twig-energy/index.json"

@mathisloge I tried adding a new repo but got a bunch of other errors so I returned to the org setup

@mathisloge
Copy link
Contributor

yeah username + password + apikey.
IIRC the first one is for github basic auth to get access to the package manifests and the second one for uploading

@anders-wind
Copy link
Contributor Author

I need to try if the GITHUB_TOKEN can also be used for the setapikey :) But yea hope the documentation will be updated - but this issue can be closed I guess :) Thanks for the help!

@LilyWangLL LilyWangLL assigned JonLiu1993 and unassigned JackBoosY Dec 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@BillyONeal BillyONeal

@JonLiu1993 JonLiu1993

Labels
requires:discussion
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@BillyONeal @anders-wind @mathisloge @JackBoosY @JonLiu1993