Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't use private repository in poetry add #233

Closed
ojii opened this issue Jun 19, 2018 · 22 comments · Fixed by #421
Closed

Can't use private repository in poetry add #233

ojii opened this issue Jun 19, 2018 · 22 comments · Fixed by #421

Comments

@ojii
Copy link
Contributor

ojii commented Jun 19, 2018

Poetry version: 0.10.3
Python Version: 3.6.5

(Note: I redacted URL of my private repository)

pyproject.toml:

[tool.poetry]
name = "project"
version = "0.1.0"
description = ""
authors = []

[[tool.poetry.source]]
name = "private"
url = "<url>"

[tool.poetry.dependencies]
python = "^3.6"

[tool.poetry.dev-dependencies]
pytest = "^3.5"

I ran the following two commands:

  • poetry config repositories.private <url>
  • poetry config http-basic.private <username>

Checking the config files on disk, the values are correct.

When I run poetry search <private package> or poetry add <private package> poetry cannot find the package. Here's the output of poetry -vvv add:

poetry -vvv add <private package>
                                  
[ValueError]       
Package [<private package>] not found.  
                                  
Exception trace:
 /usr/local/lib/python3.6/site-packages/poetry/_vendor/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /usr/local/lib/python3.6/site-packages/poetry/console/application.py in do_run() at line 87
   return super(Application, self).do_run(i, o)
 /usr/local/lib/python3.6/site-packages/poetry/_vendor/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /usr/local/lib/python3.6/site-packages/poetry/console/commands/command.py in run() at line 72
   return super(BaseCommand, self).run(i, o)
 /usr/local/lib/python3.6/site-packages/poetry/_vendor/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /usr/local/lib/python3.6/site-packages/poetry/_vendor/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /usr/local/lib/python3.6/site-packages/poetry/console/commands/add.py in handle() at line 65
   packages, allow_prereleases=self.option("allow-prereleases")
 /usr/local/lib/python3.6/site-packages/poetry/console/commands/init.py in _determine_requirements() at line 215
   requirement["name"], allow_prereleases=allow_prereleases
 /usr/local/lib/python3.6/site-packages/poetry/console/commands/init.py in _find_best_version_for_package() at line 245
   name, required_version, allow_prereleases=allow_prereleases
 /usr/local/lib/python3.6/site-packages/poetry/version/version_selector.py in find_best_candidate() at line 28
   package_name, constraint, allow_prereleases=allow_prereleases
 /usr/local/lib/python3.6/site-packages/poetry/repositories/pool.py in find_packages() at line 83
   name, constraint, extras=extras, allow_prereleases=allow_prereleases
 /usr/local/lib/python3.6/site-packages/poetry/repositories/pypi_repository.py in find_packages() at line 90
   info = self.get_package_info(name)
 /usr/local/lib/python3.6/site-packages/poetry/repositories/pypi_repository.py in get_package_info() at line 225
   name, lambda: self._get_package_info(name)
 /usr/local/lib/python3.6/site-packages/poetry/_vendor/cachy/repository.py in remember_forever() at line 174
   val = value(callback)
 /usr/local/lib/python3.6/site-packages/poetry/_vendor/cachy/helpers.py in value() at line 6
   return val()
 /usr/local/lib/python3.6/site-packages/poetry/repositories/pypi_repository.py in <lambda>() at line 225
   name, lambda: self._get_package_info(name)
 /usr/local/lib/python3.6/site-packages/poetry/repositories/pypi_repository.py in _get_package_info() at line 231
   raise ValueError("Package [{}] not found.".format(name))

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--python PYTHON] [--platform PLATFORM] [--allow-prereleases] [--dry-run] [--] <name> (<name>)...

We use pypicloud for our private repository.

@xsduan
Copy link

xsduan commented Jun 20, 2018

have you tried changing if you're using the /simple/ index (or some other equivalent), and then toggling the trailing slash? I've found that it's pretty finicky about which url goes where, and after running into your problem it just...fixed itself after a few different urls. (currently using /simple with no slash)

It'd be nice to have some attempted diagnostics (like wrong format vs 404/403 or something).

@sdispater
Copy link
Member

@ojii You might want to try the latest preview release (0.11.0a4), it should be fixed.

@ojii
Copy link
Contributor Author

ojii commented Jun 21, 2018

have you tried changing if you're using the /simple/ index (or some other equivalent), and then toggling the trailing slash?

I'm using the exact same URL that works fine with pip3.

@ojii You might want to try the latest preview release (0.11.0a4), it should be fixed.
@sdispater I've upgraded to Poetry 0.11.0-alpha.4 but still have the same issue.

@Gr1N
Copy link
Contributor

Gr1N commented Jul 14, 2018

Got same issue. Private repository configured step by step like described in https://poetry.eustace.io/docs/repositories/.

Updating dependencies
Resolving dependencies...
   1: fact: my-package is 0.1.0-alpha.0
   1: derived: my-package
   1: fact: my-package depends on my-private (*)
   1: selecting my-package (0.1.0-alpha.0)
   1: derived: my-private (*)
privatepypi: 0 packages found for my-private *
   1: fact: my-private doesn't exist
   1: conflict: my-private doesn't exist
   1: ! my-private (*) is satisfied by my-private (*)
   1: ! which is caused by "my-package depends on my-private (*)"
   1: ! thus: version solving failed
   1: Version solving took 0.440 seconds.
   1: Tried 1 solutions.

[SolverProblemError]
Because my-package depends on my-private (*) which doesn't exist, version solving failed.

Exception trace:
 /Users/ng/.pyenv/versions/3.6.5/lib/python3.6/site-packages/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /Users/ng/.pyenv/versions/3.6.5/lib/python3.6/site-packages/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /Users/ng/.pyenv/versions/3.6.5/lib/python3.6/site-packages/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /Users/ng/.pyenv/versions/3.6.5/lib/python3.6/site-packages/poetry/console/commands/command.py in run() at line 72
   return super(BaseCommand, self).run(i, o)
 /Users/ng/.pyenv/versions/3.6.5/lib/python3.6/site-packages/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /Users/ng/.pyenv/versions/3.6.5/lib/python3.6/site-packages/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /Users/ng/.pyenv/versions/3.6.5/lib/python3.6/site-packages/poetry/console/commands/install.py in handle() at line 50
   return installer.run()
 /Users/ng/.pyenv/versions/3.6.5/lib/python3.6/site-packages/poetry/installation/installer.py in run() at line 76
   self._do_install(local_repo)
 /Users/ng/.pyenv/versions/3.6.5/lib/python3.6/site-packages/poetry/installation/installer.py in _do_install() at line 158
   ops = solver.solve(use_latest=self._whitelist)
 /Users/ng/.pyenv/versions/3.6.5/lib/python3.6/site-packages/poetry/puzzle/solver.py in solve() at line 31
   packages, depths = self._solve(use_latest=use_latest)
 /Users/ng/.pyenv/versions/3.6.5/lib/python3.6/site-packages/poetry/puzzle/solver.py in _solve() at line 130
   raise SolverProblemError(e)

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

Interesting fact if in pyproject.toml define source with basic auth then everything work as expected (but I don't want to store credentials in repo).

[[tool.poetry.source]]
name = "privatepypi"
url = "https://username@password:party.jfrog.io/party/api/pypi/pypi/simple/"

As PyPI server we use Artifactory by JFrog.

And poetry version is 0.11.2.

@Gr1N
Copy link
Contributor

Gr1N commented Jul 14, 2018

So as I understand from code base when poetry tries to access private repository for search or install command authentication credentials which configured and stored in auth.toml not used.

And for example in case of publish command poetry reads auth.toml and uses credentials from it https://github.com/sdispater/poetry/blob/master/poetry/masonry/publishing/publisher.py#L66.

@Gr1N
Copy link
Contributor

Gr1N commented Jul 14, 2018

@sdispater am I right?

@ojii
Copy link
Contributor Author

ojii commented Jul 14, 2018

search (and probably install) seem to completely ignore custom repositories.

@Gr1N
Copy link
Contributor

Gr1N commented Jul 14, 2018

Can it be fixed soon?

@Gr1N
Copy link
Contributor

Gr1N commented Jul 15, 2018

I've created a PR #307 with fix.

@lepisma
Copy link

lepisma commented Jul 17, 2018

I am facing a similar issue (not able to add package from a private repository) though its not related to authentication.

  • OS version and name: 4.17.5-1-MANJARO
  • Poetry version: Poetry 0.11.2
  • Link of a Gist with the contents of your pyproject.toml file: toml

Trying to add -vvv gives this:

[IndexError]  
list index out of range    
                             
Exception trace:
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/_vendor/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/_vendor/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/console/commands/command.py in run() at line 72
   return super(BaseCommand, self).run(i, o)
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/_vendor/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/_vendor/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/console/commands/add.py in handle() at line 137
   status = installer.run()
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/installation/installer.py in run() at line 76
   self._do_install(local_repo)
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/installation/installer.py in _do_install() at line 158
   ops = solver.solve(use_latest=self._whitelist)
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/puzzle/solver.py in solve() at line 31
   packages, depths = self._solve(use_latest=use_latest)
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/puzzle/solver.py in _solve() at line 121
   self._package, self._provider, locked=locked, use_latest=use_latest
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/mixology/__init__.py in resolve_version() at line 7
   return solver.solve()
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/mixology/version_solver.py in solve() at line 79
   next = self._choose_package_version()
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/mixology/version_solver.py in _choose_package_version() at line 378
   version = self._provider.complete_package(version)
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/puzzle/provider.py in complete_package() at line 323
   package.name, package.version.text, extras=package.requires_extras
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/repositories/pool.py in package() at line 67
   package = repository.package(name, version, extras=extras)
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/repositories/legacy_repository.py in package() at line 235
   release_info = self.get_release_info(name, version)
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/repositories/pypi_repository.py in get_release_info() at line 239
   "{}:{}".format(name, version), lambda: self._get_release_info(name, version)
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/_vendor/cachy/repository.py in remember_forever() at line 174
   val = value(callback)
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/_vendor/cachy/helpers.py in value() at line 6
   return val()
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/repositories/pypi_repository.py in <lambda>() at line 239
   "{}:{}".format(name, version), lambda: self._get_release_info(name, version)
 /home/lepisma/.miniconda/lib/python3.6/site-packages/poetry/repositories/legacy_repository.py in _get_release_info() at line 297
   default_link = links[0]

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--python PYTHON] [--platform PLATFORM] [--allow-prereleases] [--dry-run] [--] <name> (<name>)...

I tried to look into the problem and found that the IndexError gets thrown when it looks for pytest in my private repo (which is a LegacyRepository) while looping over the pool. I assume when we don't find anything in a repository from the pool, we skip that repo and continue. This skipping is not capturing the IndexError (happening here).

As of now, I am adding a catch block for IndexError in pool.py and its working fine. Though @sdispater might want to take a look.

@Gr1N
Copy link
Contributor

Gr1N commented Jul 22, 2018

Any news on this? @sdispater?

sdispater pushed a commit that referenced this issue Jul 25, 2018
… legacy repo (#233) (#306)

* Adding support for HTTP Basic Auth when pulling from legacy repo

* added docs

* Adding unit test for http auth

* Adding mock to dev deps

* moved config reading logic to helper function, adapted test

* corrections

* rework pip_installer

* use source_reference as repo name when writing lock file and installing
sdispater pushed a commit that referenced this issue Jul 27, 2018
… legacy repo (#233) (#306)

* Adding support for HTTP Basic Auth when pulling from legacy repo

* added docs

* Adding unit test for http auth

* Adding mock to dev deps

* moved config reading logic to helper function, adapted test

* corrections

* rework pip_installer

* use source_reference as repo name when writing lock file and installing
@greatvovan
Copy link

greatvovan commented Dec 11, 2019

Why is this issue closed?

poetry add still does not work for packages in private repos. The repo is configured:

poetry  config --list
settings.virtualenvs.create = true
settings.virtualenvs.in-project = false
settings.virtualenvs.path = "/Users/xxx/Library/Caches/pypoetry/virtualenvs"
repositories.my.url = "http://pypi.my.domain.net/simple/"
$ poetry add mypackage==1.3.2
                                  
[PackageNotFound]  
Package [mypackage] not found.  

See also verbose output.

There is no authentication in my repo.

The path from repositories.my.url works well with pip install.

I was able to overcome this by adding the package manually into pyproject.toml as well as this section:

[[tool.poetry.source]]
name = "my"
url = "http://pypi.my.domain.net/simple/"

Then I deleted poetry.lock and issued poetry install. Now I got it in the 'lock' file, but this seems too cumbersome.

@sdispater should I open a new issue?

P.S. My username, package name and URL are consistently changed into other values.

$ poetry --version
Poetry 0.12.17

@greatvovan
Copy link

greatvovan commented Dec 11, 2019

With version 1.0.0b9 it shows this:

$ poetry add -vvv mypackage==1.3.2
Using virtualenv: /Users/xxx/Library/Caches/pypoetry/virtualenvs/peapi-7kyxBY5Q-py3.8
PyPI: 0 packages found for mypackage 1.3.2

[ValueError]
Could not find a matching version of package mypackage

Traceback (most recent call last):
  File "/Users/xxx/.poetry/lib/poetry/_vendor/py3.8/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/Users/xxx/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "/Users/xxx/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/Users/xxx/.poetry/lib/poetry/_vendor/py3.8/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/Users/xxx/.poetry/lib/poetry/console/commands/add.py", line 88, in handle
    requirements = self._determine_requirements(
  File "/Users/xxx/.poetry/lib/poetry/console/commands/init.py", line 314, in _determine_requirements
    name, _ = self._find_best_version_for_package(
  File "/Users/xxx/.poetry/lib/poetry/console/commands/init.py", line 338, in _find_best_version_for_package
    raise ValueError(

If I add [[tool.poetry.source]] in poetry.toml poetry add works. But isn't it supposed to work right away with all the configured repositories?

Edit: I don't think so because:

When you specify a dependency in pyproject.toml, Poetry first take the name of the package that you have requested and searches for it in any repository you have registered using the repositories key. If you have not registered any extra repositories, or it does not find a package with that name in the repositories you have specified, it falls back on PyPI.
(https://python-poetry.org/docs/basic-usage/#version-constraints)

@szafranek
Copy link

This issue should be reopened. The problem still happens in poetry 1.0.5.

@qiuwei
Copy link

qiuwei commented May 7, 2020

I have the same problem poetry 1.0.5.

@LSaldyt
Copy link

LSaldyt commented Jun 4, 2020

EDIT: I've fixed this, by adding /simple/ to the end of my config repo, and now the debug log actually checks my custom repo. A warning about a bad URL would be very helpful, though.

Same issue in 1.0.5. Debug output indicates poetry doesn't even look at the custom repo:

!poetry config --list
cache-dir = "/Users/X/Library/Caches/pypoetry"
repositories.X.url = "https://sonatype.services.X.com/repository/pypi-internal/"
virtualenvs.create = true
virtualenvs.in-project = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /Users/X/Library/Caches/pypoetry/virtualenvs
!poetry add my-package -vvv
Using virtualenv: /Users/X/Library/Caches/pypoetry/virtualenvs/X-BmgmJgwc-py3.7
PyPI: No packages found for my-package *

[ValueError]
Could not find a matching version of package my-package

Traceback (most recent call last):
  File "/Users/X/Library/Python/3.7/lib/python/site-packages/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/Users/X/Library/Python/3.7/lib/python/site-packages/clikit/api/command/command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "/Users/X/Library/Python/3.7/lib/python/site-packages/clikit/api/command/command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/Users/X/Library/Python/3.7/lib/python/site-packages/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/Users/X/Library/Python/3.7/lib/python/site-packages/poetry/console/commands/add.py", line 89, in handle
    packages, allow_prereleases=self.option('allow-prereleases')
  File "/Users/X/Library/Python/3.7/lib/python/site-packages/poetry/console/commands/init.py", line 302, in _determine_requirements
    requirement['name'], allow_prereleases=allow_prereleases
  File "/Users/X/Library/Python/3.7/lib/python/site-packages/poetry/console/commands/init.py", line 338, in _find_best_version_for_package
    'Could not find a matching version of package {}'.format(name)

!poetry --version
Poetry version 1.0.5

And here's the project config

[tool.poetry]
name = "X"
version = "0.1.0"
description = ""
authors = ["X"]

[[tool.poetry.source]]
name = 'X'
url  = 'https://sonatype.services.X.com/repository/pypi-internal/'
secondary = true

[tool.poetry.dependencies]
python = "^3.7"
Django = "<3.0"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

The weird part is that I had poetry downloading dependencies from our custom PyPI repository, but once I uploaded a package myself, it doesn't like it.

@kmcminn
Copy link

kmcminn commented Jun 5, 2020

still a source of confusion not mentioned in any docs I could find. @LSaldyt fix worked though.

@adrianabreu
Copy link

Got it working thanks to @LSaldyt /simple/ trick.

-vvv showed that it wasn't looking at all for my custom private repository. It was pretty hard to understand since I was copying and pasting the url from gitlab suggestion, can this be added explicitly to the docs?

@vduseev
Copy link

vduseev commented Jan 29, 2021

For those who get the "Could not find a matching version of package " when trying to install from private repo

$ poetry add my_package 

ValueError
Could not find a matching version of package my_package

Try to add /simple/ to the url of the repo which you have used in pyproject.toml:

[[tool.poetry.source]]
name = "myrepo"
url = "https://my.rul/simple/"

That solved the problem for me with AWS CodeArtifact repository. Curiously, during authentication for publishing while setting the poetry config repositories.myrepo $URL and poetry config http-basic.myrepo $USER $TOKEN I had to use CodeArtifact url WITHOUT /simple/ at the end.

@clanzett
Copy link

I am experiencing the same problem. Using the solution suggested by @vduseev works for getting dependencies from my private repo, but prevents me from uploading the version from this package to the same repo, probably because of the "/simple/" path of the repo url in the pyproject.toml file.

Could you please investigate further? Many thanks...

@clanzett
Copy link

clanzett commented Feb 22, 2021

A possible "fix" or workaround could be to bypass the url in the pyproject.toml by adding this code to the publishing/uploader.py file at line 110:
import re
r_simple = re.compile(r"/simple")
if r_simple.search(url):
url = url.removesuffix('simple')`

This removes the simple suffix from the url configured in the toml file. And yes I know, that this is very ugly....

dimbleby pushed a commit to dimbleby/poetry that referenced this issue Apr 21, 2022
* change type annotations to type hints using com2ann

* add missing type hints to core

* add type hints to tests

* add missing type hints to stanza

* add flake8-annotations to flake8 pre-commit hook
Copy link

github-actions bot commented Mar 2, 2024

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 2, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet