Current master falls back to PyPi when using private repo.

[voney]

Following on from #5651

The original issue was that poetry could not handle repositories that hand back CDN links for packages with authenticationbaked into the URL because it would still add the base repository index auth details to the request headers.

After trying the lastest version from git I get a new issue.

(.venv) [daniel@dv-xps-fedora Davis_state_consumer]$ poetry@git add --source myget 'redacted'
Using version ^2.0.6 for redacted

Updating dependencies
Resolving dependencies... (4.2s)

  HTTPError

  404 Client Error: Not Found for url: https://pypi.org/pypi/redacted_dependency/json/

  at ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/requests/models.py:960 in raise_for_status
      956│         elif 500 <= self.status_code < 600:
      957│             http_error_msg = u'%s Server Error: %s for url: %s' % (self.status_code, reason, self.url)
      958│ 
      959│         if http_error_msg:
    → 960│             raise HTTPError(http_error_msg, response=self)
      961│ 
      962│     def close(self):
      963│         """Releases the connection back to the pool. Once this method has been
      964│         called the underlying ``raw`` object must not be accessed again.

That's after adding the myget repo using:

poetry@git source add myget https://www.myget.org/F/redacted/python/

Originally posted by @voney in #5651 (comment)

[abn]

Following from #5651. Any chance you can provide the logs for the same command with -vvv?

[voney]

Sorry for the double tickets, I'll comment here going forward.

(.venv) [daniel@dv-xps-fedora my_project]$ poetry@git add --source myget 'redacted' -vvv
Loading configuration file /home/daniel/.config/pypoetry/config.toml
Loading configuration file /home/daniel/.config/pypoetry/auth.toml
Adding repository myget (https://www.myget.org/F/myrepo/python)
Using virtualenv: /home/daniel/myrepo/my_project/.venv
Project environment contains an empty path in sys_path, ignoring.
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for www.myget.org
[urllib3.connectionpool] Starting new HTTPS connection (1): www.myget.org:443
[urllib3.connectionpool] https://www.myget.org:443 "GET /F/myrepo/python/redacted/ HTTP/1.1" 200 1357
Source (myget): 11 packages found for redacted *
Using version ^2.0.6 for redacted

Updating dependencies
Resolving dependencies...
   1: fact: my-project is 0.1.0
   1: derived: my-project
   1: fact: my-project depends on redacted (^2.0.6)
   1: selecting my-project (0.1.0)
   1: derived: redacted (>=2.0.6,<3.0.0)
[urllib3.connectionpool] https://www.myget.org:443 "GET /F/myrepo/python/redacted/ HTTP/1.1" 200 1357
Source (myget): 1 packages found for redacted >=2.0.6,<3.0.0
   1: fact: redacted (2.0.6) depends on nose (*)
   1: fact: redacted (2.0.6) depends on azure-eventhub (>=5.0.0)
   1: fact: redacted (2.0.6) depends on redacted_dependency (*)
   1: selecting redacted (2.0.6)
   1: derived: redacted_dependency
   1: derived: azure-eventhub (>=5.0.0)
   1: derived: nose
[urllib3.connectionpool] https://www.myget.org:443 "GET /F/myrepo/python/redacted_dependency/ HTTP/1.1" 200 1030
Source (myget): 3 packages found for redacted_dependency *
Creating new session for pypi.org
[urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/redacted_dependency/json/ HTTP/1.1" 404 1980
   1: Version solving took 1.010 seconds.
   1: Tried 1 solutions.

  Stack trace:

  28  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cleo/application.py:330 in run
       328│ 
       329│             try:
     → 330│                 exit_code = self._run(io)
       331│             except Exception as e:
       332│                 if not self._catch_exceptions:

  27  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/console/application.py:182 in _run
       180│         self._load_plugins(io)
       181│ 
     → 182│         exit_code: int = super()._run(io)
       183│         return exit_code
       184│ 

  26  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cleo/application.py:425 in _run
       423│                 io.set_input(ArgvInput(argv))
       424│ 
     → 425│         exit_code = self._run_command(command, io)
       426│         self._running_command = None
       427│ 

  25  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cleo/application.py:467 in _run_command
       465│ 
       466│         if error is not None:
     → 467│             raise error
       468│ 
       469│         return event.exit_code

  24  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cleo/application.py:451 in _run_command
       449│ 
       450│             if event.command_should_run():
     → 451│                 exit_code = command.run(io)
       452│             else:
       453│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  23  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cleo/commands/base_command.py:118 in run
       116│         io.input.validate()
       117│ 
     → 118│         status_code = self.execute(io)
       119│ 
       120│         if status_code is None:

  22  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cleo/commands/command.py:85 in execute
        83│ 
        84│         try:
     →  85│             return self.handle()
        86│         except KeyboardInterrupt:
        87│             return 1

  21  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/console/commands/add.py:256 in handle
       254│         self._installer.whitelist([cast(str, r["name"]) for r in requirements])
       255│ 
     → 256│         status = self._installer.run()
       257│ 
       258│         if status == 0 and not self.option("dry-run"):

  20  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/installation/installer.py:112 in run
       110│         local_repo = Repository()
       111│ 
     → 112│         return self._do_install(local_repo)
       113│ 
       114│     def dry_run(self, dry_run: bool = True) -> Installer:

  19  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/installation/installer.py:245 in _do_install
       243│                 source_root=self._env.path.joinpath("src")
       244│             ):
     → 245│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       246│         else:
       247│             self._io.write_line("Installing dependencies from lock file")

  18  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/puzzle/solver.py:80 in solve
        78│         with self._provider.progress():
        79│             start = time.time()
     →  80│             packages, depths = self._solve(use_latest=use_latest)
        81│             end = time.time()
        82│ 

  17  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/puzzle/solver.py:149 in _solve
       147│ 
       148│         try:
     → 149│             result = resolve_version(
       150│                 self._package, self._provider, locked=locked, use_latest=use_latest
       151│             )

  16  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/mixology/__init__.py:24 in resolve_version
        22│     solver = VersionSolver(root, provider, locked=locked, use_latest=use_latest)
        23│ 
     →  24│     return solver.solve()
        25│ 

  15  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/mixology/version_solver.py:125 in solve
       123│             while next is not None:
       124│                 self._propagate(next)
     → 125│                 next = self._choose_package_version()
       126│ 
       127│             return self._result()

  14  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/mixology/version_solver.py:409 in _choose_package_version
       407│             dependency = unsatisfied[0]
       408│         else:
     → 409│             dependency = min(*unsatisfied, key=_get_min)
       410│ 
       411│         locked = self._get_locked(dependency)

  13  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/mixology/version_solver.py:401 in _get_min
       399│                 return (
       400│                     not dependency.marker.is_any(),
     → 401│                     len(self._dependency_cache.search_for(dependency)),
       402│                 )
       403│             except ValueError:

  12  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/mixology/version_solver.py:62 in _search_for
        60│         packages = self.cache.get(key)
        61│         if packages is None:
     →  62│             packages = self.provider.search_for(dependency)
        63│         else:
        64│             packages = [p for p in packages if dependency.constraint.allows(p.version)]

  11  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/puzzle/provider.py:220 in search_for
       218│             packages = self.search_for_url(dependency)
       219│         else:
     → 220│             packages = self._pool.find_packages(dependency)
       221│ 
       222│             packages.sort(

  10  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/repositories/pool.py:169 in find_packages
       167│         packages = []
       168│         for repo in self._repositories:
     → 169│             packages += repo.find_packages(dependency)
       170│ 
       171│         return packages

   9  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/repositories/pypi_repository.py:54 in find_packages
        52│ 
        53│         try:
     →  54│             info = self.get_package_info(dependency.name)
        55│         except PackageNotFound:
        56│             self._log(

   8  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/repositories/pypi_repository.py:155 in get_package_info
       153│             return self._get_package_info(name)
       154│ 
     → 155│         package_info: dict[str, Any] = self._cache.store("packages").remember_forever(
       156│             name, lambda: self._get_package_info(name)
       157│         )

   7  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cachy/repository.py:174 in remember_forever
       172│             return val
       173│ 
     → 174│         val = value(callback)
       175│ 
       176│         self.forever(key, val)

   6  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cachy/helpers.py:6 in value
         4│ def value(val):
         5│     if callable(val):
     →   6│         return val()
         7│ 
         8│     return val

   5  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/repositories/pypi_repository.py:156 in <lambda>
       154│ 
       155│         package_info: dict[str, Any] = self._cache.store("packages").remember_forever(
     → 156│             name, lambda: self._get_package_info(name)
       157│         )
       158│         return package_info

   4  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/repositories/pypi_repository.py:161 in _get_package_info
       159│ 
       160│     def _get_package_info(self, name: str) -> dict[str, Any]:
     → 161│         data = self._get(f"pypi/{name}/json")
       162│         if data is None:
       163│             raise PackageNotFound(f"Package [{name}] not found.")

   3  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/repositories/pypi_repository.py:246 in _get
       244│     def _get(self, endpoint: str) -> dict[str, Any] | None:
       245│         try:
     → 246│             json_response = self.session.get(self._base_url + endpoint)
       247│         except requests.exceptions.TooManyRedirects:
       248│             # Cache control redirect loop.

   2  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/utils/authenticator.py:235 in get
       233│ 
       234│     def get(self, url: str, **kwargs: Any) -> requests.Response:
     → 235│         return self.request("get", url, **kwargs)
       236│ 
       237│     def post(self, url: str, **kwargs: Any) -> requests.Response:

   1  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/utils/authenticator.py:221 in request
       219│                 if resp.status_code not in [502, 503, 504] or is_last_attempt:
       220│                     if raise_for_status:
     → 221│                         resp.raise_for_status()
       222│                     return resp
       223│ 

  HTTPError

  404 Client Error: Not Found for url: https://pypi.org/pypi/redacted_dependency/json/

  at ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/requests/models.py:960 in raise_for_status
      956│         elif 500 <= self.status_code < 600:
      957│             http_error_msg = u'%s Server Error: %s for url: %s' % (self.status_code, reason, self.url)
      958│ 
      959│         if http_error_msg:
    → 960│             raise HTTPError(http_error_msg, response=self)
      961│ 
      962│     def close(self):
      963│         """Releases the connection back to the pool. Once this method has been
      964│         called the underlying ``raw`` object must not be accessed again.

[abn]

Thanks for the logs @voney I will take a peek in the next few days. Seems for some reason instead of accepting a 404 as a "package not available in this source", we are raising the exception.

One the of the issues here is that we do not deal very well with transient deps. A mitigation in the interim might be to explicitly add redacted_dependency with source set to your project. But we should fix this propertly soon.

[voney]

Thans for the quick reply @abn I'll wait for version 1.2, I'm trying to get poetry used within my work and workarounds aren't going to fly.

[abn]

@voney can you see if #5698 fixes this?

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.