pypi auth issue when installing lock-file with package from private index

[croth1]

I am struggeling getting conda-lock with a private gitlab index to work:

I have setup my pypi repository following the instructions on the readme. poetry config repositories.foo gives me the expected output:

{'url': 'https://<myuser>:<my-acc>@gitlab.com/api/v4/groups/<my-grp>/-/packages/pypi/simple'}

with this I was able to create a lockfile with conda-lock lock -k explict.

The lockfile lists the package as:

# pip sagemaker @ https://files.pythonhosted.org/packages/f4/c7/f6ca830f41726081dd7baeb30c2195f25d9d573ac387f2419e751e21ffd7/sagemaker-2.75.1.tar.gz#sha256=30dfc5ed359a2ac42928c47b3f1c7455a9b5f8ef8f14427f1f98cb261df3a5ee
# pip <my-private-package>@ https://gitlab.com/api/v4/groups/<my-grp>/-/packages/pypi/files/58a41a9cc01317beaadef11808fc861f0513d0f575c3fe4abbdb08935f66986c/<my-private-package>-<version>-py3-none-any.whl#sha256=58a41a9cc01317beaadef11808fc861f0513d0f575c3fe4abbdb08935f66986c

In the lock file the url does not contain the authentication information.

If I try to create the environment from the lock file, i get an auth error:

INFO:root:Collecting sagemaker@ https://files.pythonhosted.org/packages/f4/c7/f6ca830f41726081dd7baeb30c2195f25d9d573ac387f2419e751e21ffd7/sagemaker-2.75.1.tar.gz#sha256=30dfc5ed359a2ac42928c47b3f1c7455a9b5f8ef8f14427f1f98cb261df3a5ee
INFO:root:  Using cached sagemaker-2.75.1.tar.gz (511 kB)
INFO:root:  Preparing metadata (setup.py): started
INFO:root:  Preparing metadata (setup.py): finished with status 'done'
INFO:root:Collecting <my-private-package>@ https://gitlab.com/api/v4/groups/<mygrp>/-/packages/pypi/files/58a41a9cc01317beaadef11808fc861f0513d0f575c3fe4abbdb08935f66986c/<my-private-package>-<version>-py3-none-any.whl#sha256=58a41a9cc01317beaadef11808fc861f0513d0f575c3fe4abbdb08935f66986c
INFO:root:User for [gitlab.com](http://gitlab.com/):
ERROR:root:ERROR: Exception:
ERROR:root:Traceback (most recent call last):
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
ERROR:root:    status = run_func(*args)
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
ERROR:root:    return func(self, options, args)
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 369, in run
ERROR:root:    requirement_set = resolver.resolve(
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
ERROR:root:    collected = self.factory.collect_root_requirements(root_reqs)
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 491, in collect_root_requirements
ERROR:root:    req = self._make_requirement_from_install_req(
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 453, in _make_requirement_from_install_req
ERROR:root:    cand = self._make_candidate_from_link(
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
ERROR:root:    self._link_candidate_cache[link] = LinkCandidate(
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 297, in __init__
ERROR:root:    super().__init__(
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 162, in __init__
ERROR:root:    self.dist = self._prepare()
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 231, in _prepare
ERROR:root:    dist = self._prepare_distribution()
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 308, in _prepare_distribution
ERROR:root:    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 438, in prepare_linked_requirement
ERROR:root:    return self._prepare_linked_requirement(req, parallel_builds)
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 483, in _prepare_linked_requirement
ERROR:root:    local_file = unpack_url(
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 165, in unpack_url
ERROR:root:    file = get_http_url(
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 106, in get_http_url
ERROR:root:    from_path, content_type = download(link, temp_dir.path)
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/network/download.py", line 134, in __call__
ERROR:root:    resp = _http_get_download(self._session, link)
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/network/download.py", line 117, in _http_get_download
ERROR:root:    resp = session.get(target_url, headers=HEADERS, stream=True)
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 600, in get
ERROR:root:    return self.request("GET", url, **kwargs)
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/network/session.py", line 518, in request
ERROR:root:    return super().request(method, url, *args, **kwargs)
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 587, in request
ERROR:root:    resp = self.send(prep, **send_kwargs)
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 708, in send
ERROR:root:    r = dispatch_hook("response", hooks, r, **kwargs)
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_vendor/requests/hooks.py", line 30, in dispatch_hook
ERROR:root:    _hook_data = hook(hook_data, **kwargs)
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/network/auth.py", line 270, in handle_401
ERROR:root:    username, password, save = self._prompt_for_password(parsed.netloc)
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/network/auth.py", line 233, in _prompt_for_password
ERROR:root:    username = ask_input(f"User for {netloc}: ")
ERROR:root:  File "/home/ubuntu/miniconda3/envs/foo123/lib/python3.9/site-packages/pip/_internal/utils/misc.py", line 204, in ask_input
ERROR:root:    return input(message)
ERROR:root:EOFError: EOF when reading a line
ERROR:root:
ERROR:root:ERROR conda.cli.main_run:execute(49): `conda run pip install --no-deps -r /tmp/tmp0upx01sq` failed. (See above for error)

I tried to play around with the --auth option but this one only seems to apply to private conda packages - pip entries start with a # and are automatically ignored.

Any clues what i am missing here? How is the authentication supposed to work? Should the auth information be in the pypi index url in the lockfile - and if so, how do I get it there?

[croth1]

I had a bit of time debugging this and in the end I traced the bug until here deep in the poetry code base: https://github.com/python-poetry/poetry/blob/1.1/poetry/repositories/legacy_repository.py#L118

url = self.clean_link(urlparse.urljoin(self._url, href))

self._url is an absolute url, with the auth information: https://<myuser>:<my-acc>@gitlab.com/api/v4/groups/<my-grp>/...., href is another absolute url, but without the token: https://gitlab.com/...

Since href is not relative, urljoin simply returns href and the user:token authentication part of the url is lost.

Not sure what to make out of that though. One of these seems most plausible:

• href is supposed to be a relative path and the gitlab pypi index is malformed
• poetry strips authentication information on purpose and uses other mechanisms to authenticate (https://python-poetry.org/docs/1.1/repositories/#configuring-credentials) - bit strange because if href was a relative path, the authentication information would not have been striped.
• there is a bug in the poetry logic and the auth information should not have been stripped.

Do you think it would make sense to address this within conda-lock? Something like --auth adding credentials also to pip packages?

---

From @neersighted on poetry discord:

Honestly I'm not sure what the state of credentials in dependency URLs is
The intention overall has been to discourage/prevent their usage, but I don't think that's been uniformly achieved
and in some cases it may be a breaking change
But the pyproject.toml and lock file are intended to be shareable for support and never contain anything more private than URLs/package names

My current hypothesis is that poetry losing the auth information is actually a bug (or at least unexpected behaviour) in the poetry code base, but it doesn't really show for them because poetry discourages adding authentication information to the repo url. I feel the easiest workaround in conda-lock would be to also follow poetries strategy to strip auth information (if not already stripped by poetry) and use --auth to bring them back.

[croth1]

I had some more time to play around, and removing '#' in this line together with the --auth parameter adds credentials back to the private pypi url, and conda-lock succeeds.

https://github.com/conda-incubator/conda-lock/blob/main/conda_lock/conda_lock.py#L800

I feel the cleanest way to deal with all of this is to extend stripping and re-adding auth also to poetry-solved packages. I might even have some time to work on that - but it would be nice to get a quick thumbs up/down before I start to make sure I don't miss something obvious before I invest the time :)

[maresb]

Thanks for looking into this! I see you're working with explicit lockfiles. Do things work any better with new-style lockfiles?

[maresb]

...I'd personally rather see the new-style lockfiles developed further, so I'm wondering if there's something missing there for your use case.

[croth1]

Good news and bad news: it seems that switching to the new-stye lockfiles works out of the box - however it seems, I tripped over a new problem: #261.

[croth1]

I finally had some time to look into this again. I am somewhat sure that skipping lines starting with # here when adding the auth back is preventing auth being added back to private pypi indices:

https://github.com/conda-incubator/conda-lock/blob/90b370a7b8f2b918820159a3c4f95ac0a31ac446/conda_lock/conda_lock.py#L906

No matter what lock-file is being used, it seems to get reformatted to the explicit lockfile style, which has pip entries as # pip ... - and they are simply skipped in the auth-adding logic.

https://github.com/conda-incubator/conda-lock/blob/90b370a7b8f2b918820159a3c4f95ac0a31ac446/conda_lock/conda_lock.py#L1443-L1450

https://github.com/conda-incubator/conda-lock/blob/90b370a7b8f2b918820159a3c4f95ac0a31ac446/conda_lock/conda_lock.py#L952-L958

Would you be fine with a PR that changes the logic such that lines starting with # pip are still processed so that auth can be added to packages from private pypi indices?

[maresb]

Would you be fine with a PR

Sounds reasonable to me. @mariusvniekerk, do you agree?

[kevinpauli]

FYI I'm having this exact issue. Glad to see there exists a reasonable solution, eagerly awaiting it!

[croth1]

@mariusvniekerk any thoughts on this one? I am still interested in fixing this.

[kevinpauli]

Just in case anyone is coming here because you are trying get reproducible environments with conda and python packages (including private repos), you might try the combo of conda+poetry rather than conda+pip. It is a little more work to set up but I think the benefits of poetry have made it worth the effort in my opinion. Here is a great post explaining how to do it.

[croth1]

@kevinpauli, I personally did not get this to work, at least not for pytorch. I tried satisfying pytorch with conda-forge::pytorch-gpu and torch from pypi, both pinned down to patch, and poetry still pulled in a wealth of packages for pytorch. I guess it only really works if the conda and pip packages have exactly the same dependencies with exactly the same names.