Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't install packages from private static repository #5548

Closed
3 tasks done
lecardozo opened this issue May 5, 2022 · 8 comments
Closed
3 tasks done

Can't install packages from private static repository #5548

lecardozo opened this issue May 5, 2022 · 8 comments
Labels
kind/bug Something isn't working as expected

Comments

@lecardozo
Copy link

  • I am on the latest Poetry version.
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).
  • OS version and name: macOS Monterey (12.2.1)
  • Poetry version: 1.1.13
  • Link of a Gist with the contents of your pyproject.toml file: pyproject.toml

Issue

Hey, everyone! :D

I'm trying to use Poetry as the dependency manager for one of our internal applications. Here we have an internal private PyPI repository that is essentially a static index (served on Github Pages) following the https://<DOMAIN>/python-package-server/<PACKAGE_NAME>/ URL pattern. For each package, we have a list of links to specific repositories where the artifacts live in. Here is an example of what a page looks like for a package in this repository. 👇

<!DOCTYPE html>
<html>
  <head>
      <title>Links for package</title>
  </head>
  <body>
    <h1>Links for package</h1>
    <!-- package-server-links-start -->
    <a href="git+https://github.com/<redacted>/package@2.0.0#egg=package-2.0.0">package-2.0.0</a><br/>
    <a href="git+https://github.com/<redacted>/package@1.1.0#egg=package-1.1.0">package-1.1.0</a><br/>
    <a href="git+https://github.com/<redacted>/package@1.0.0#egg=package-1.0.0">package-1.0.0</a><br/>
  </body>
</html>

After adding the private source to pyproject.toml, the resolver was unable to find matching versions for that package

Loading configuration file /Users/me/Library/Application Support/pypoetry/config.toml
Loading configuration file /Users/me/projects/my-project/poetry.toml
Adding repository private (https://<DOMAIN>/python-package-server)
Using virtualenv: /Users/me/projects/my-project/.venv
[urllib3.connectionpool] Starting new HTTPS connection (1): <DOMAIN>:443
[urllib3.connectionpool] https://<DOMAIN>:443 "GET /python-package-server/package/ HTTP/1.1" 304 0
private: 0 packages found for package * # <---- at least it tried :) 
private: 0 packages found for package * # <---- once again? maybe 
[urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/package/json/ HTTP/1.1" 404 1988
PyPI: No packages found for package *

I guess this is happening because this format doesn't seem to be fully PEP 503 complaint, although we never had issues with PIP or PDM before.

After checking for previous discussions, I ended up finding this PR which seems to be related but I'm not completely sure. I've tested that specific version (from 5517) to see if it would work on my use case, but, unfortunately, it didn't.

Any clues on what could be the issue here? 🤔

Thanks!
@lecardozo lecardozo added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels May 5, 2022
@abn
Copy link
Member

abn commented May 5, 2022

Hi @lecardozo #5517 is aimed at single page PEP 503 repositories only. This won't work as it is referring to VCS sources. The pip and PDM use cases for these sources are different to that of Poetry. You might have better luck either pointing them to release artifacts - ie. valid sdist fiels or wheels instead of a git repository.

@lecardozo
Copy link
Author

Thanks for the quick reply @abn ! 😃

The pip and PDM use cases for these sources are different to that of Poetry. You might have better luck either pointing them to release artifacts - ie. valid sdist fiels or wheels instead of a git repository.

I see. So the limitation is in the fact that Poetry can't handle links to a specific tag in a repository (to be cloned and installed), only direct links to artifacts (sdists or wheels). Is that correct? If so, is it something that poetry might support in the future?

@abn
Copy link
Member

abn commented May 5, 2022

Not sure if it is a limitation, but rather a lack of standards around this. Poetry's source support is based on the PEP 503 standards, which iirc, does not define the distribution of direct link or vcs sources.

Also, since Poetry will need to download (in your case clone) all the package versions separately and inspect the metadata to generate the universal lock file, such a mechanism of consuming packages might be rather inefficient. Additionally, this can also lead to a lot of ambuity. If a package distributed via such a mecahnism is locked, do we identify it as a VCS sourced dependency or a simple API repository sourced dependency? And what happens when a user consumes this lockfile? Does Poetry go over these again and inspect on their local machine? Or do we treat it as VCS dependency and manage it so?

As things are today, I doubt this will be something we will support. But maybe if there are enough folks out there who want something similar, you could potentially create a plugin.

@lecardozo
Copy link
Author

Thanks for the clarification!

If a package distributed via such a mecahnism is locked, do we identify it as a VCS sourced dependency or a simple API repository sourced dependency? And what happens when a user consumes this lockfile? Does Poetry go over these again and inspect on their local machine? Or do we treat it as VCS dependency and manage it so?

I tend to think that this should be identified as a VCS sourced dependency instead of Simple API (even though the API is the intermediate here). The API ends up being just a "shortcut" index, so the user doesn't need to explicitly pass the git dependency.

With the link-to-VCS API

[tool.poetry.dependencies]
python = "^3.8"
package = "2.0.0" # still comes from VCS, but has an intermediate API that links to the actual repo

Directly to VCS

[tool.poetry.dependencies]
python = "^3.8"
package = { git = "https://github.com/company/package-repo.git", tag = "2.0.0" } #  points directly to the repo

As things are today, I doubt this will be something we will support. But maybe if there are enough folks out there who want something similar, you could potentially create a plugin.

Yeah, that makes sense! In our case, linking explicitly to the VCS seems to be enough. Although this is a little less convenient from the user's point-of-view, this seems a better choice. 😃

@abn
Copy link
Member

abn commented May 6, 2022

@lecardozo there is general interest in supporting something similar to "find-links" in pip. Might be a good discussion topic. Although what you have is more similar to a constraints file.

But all of this seems to be a way to avoid having a proper package index. Having one will also save developers wheel build times too among other things.

@lecardozo
Copy link
Author

lecardozo commented May 6, 2022 •
edited
Loading

@lecardozo there is general interest in supporting something similar to "find-links" in pip.

It seems like my use case falls between the find-links and the PEP 503 index approach. I guess a find-links functionality would probably work for this case but doesn't seem ideal.

But all of this seems to be a way to avoid having a proper package index. Having one will also save developers wheel build times too among other things.

Agreed. Although I feel that, in some cases, setting up a proper private package index might be prohibitive/not worth the effort.

@abn
Copy link
Member

abn commented May 6, 2022 •
edited
Loading

GitHub packages supporting Python packages might be helpful in your case. Alternatively, I could also recommend that you use a github release action, similar to

poetry/.github/workflows/release.yml

Lines 36 to 42 in 0d418fc

- name: Create Release
uses: ncipollo/release-action@v1
with:
artifacts: "dist/*"
token: ${{ secrets.GITHUB_TOKEN }}
draft: false
prerelease: steps.check-version.outputs.prerelease == 'true'

And then in your html file link to the release assets, like in https://github.com/python-poetry/poetry/releases/download/1.2.0b1/poetry-1.2.0b1-py3-none-any.whl.

@mkniewallner mkniewallner removed the status/triage This issue needs to be triaged label Jun 11, 2022
@acrogenesis acrogenesis mentioned this issue Jun 3, 2023
Closed
@astariul astariul mentioned this issue Sep 14, 2023
Open
@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 1, 2024

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 1, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Something isn't working as expected
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@abn @mkniewallner @lecardozo