--pip-args not passed to pip installing setuptools

[neaxi]

I have to use custom/private pypi index as our network setup blocks access to https://pypi.org. Passing --pip-args="--index_url https://pypi.private.url" (aka pip -i) is not applied when setuptools and wheel are being installed. The issue is also visible with other args like --timeout 1, etc...

As this seems to be an issue out of scope of pip-tools I'm not sure it can be fixed/handled. I wanted to provide a writeup with observed behavior and workarounds I've tested for devs facing the same issues in future, because I've failed to find a simple answer/solution when searching through issues myself.

Environment Versions

1. OS Type: Windows 10.0.19044
2. Python version: $ python -V: Python 3.10.6
3. pip version: $ pip --version: pip 22.3.1
4. pip-tools version: $ pip-compile --version: pip-compile, version 6.12.0

Steps to replicate

1. create venv
2. update pip, install piptools
3. create pyproject.toml. nothing fancy, barebones setup

# pyproject.toml

[project]
name = "modulename"
version = "1.0.0"

dependencies = []
requires-python = ">=3.8"

[build-system]
requires      = ["setuptools>=61.0.0", "wheel"]
build-backend = "setuptools.build_meta"

4. try to compile pyproject.toml:
   pip-compile --verbose pyproject.toml --resolver=backtracking --pip-args="-i https://pypi.private.url'

My findings so far

1. click loads --pip-args succcessfuly -> no issues with CLI parsing
2. the variable pip_args is available until piptools\scripts\compile.py", line 483, metadata = project_wheel_metadata()
3. It seems the -i is not passed to \build\util.py", line 53, project_wheel_metadata() in any way and there is no interface to make it happen.
4. and that's why build\env.py", line 211, in install; _subprocess(cmd) executes without -i and fails

Expected result

Successful result obtained via Workaround

Creating venv isolated environment...
Installing packages in isolated environment... (setuptools>=61.0.0, wheel)
Getting build dependencies for wheel...
Installing packages in isolated environment... (wheel)
Getting metadata for wheel...
Using indexes:
  https://pypi.private.url
  Looking in indexes: https://pypi.private.url

Actual result

subprocess executes pip without -i, tries to reach pypi.org despite --pip-args="-i https://pypi.private.url" and obviously fails.

output + traceback:

Creating venv isolated environment...
Installing packages in isolated environment... (setuptools>=61.0.0, wheel)
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
...

Traceback (most recent call last):
  File "C:\Python38\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python38\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Python38\Scripts\pip-compile.exe\__main__.py", line 7, in <module>
  File "C:\Python38\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Python38\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Python38\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Python38\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Python38\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "C:\Python38\lib\site-packages\piptools\scripts\compile.py", line 483, in cli
    metadata = project_wheel_metadata(
  File "C:\Python38\lib\site-packages\build\util.py", line 53, in project_wheel_metadata
    env.install(builder.build_system_requires)
  File "C:\Python38\lib\site-packages\build\env.py", line 211, in install
    _subprocess(cmd)
  File "C:\Python38\lib\site-packages\build\env.py", line 76, in _subprocess
    raise e
  File "C:\Python38\lib\site-packages\build\env.py", line 73, in _subprocess
    subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  File "C:\Python38\lib\subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['C:\\Users\\USERNAME\\AppData\\Local\\Temp\\build-env-nltqvjdv\\Scripts\\python.exe', '-Im', 'pip', 'install', '--use-pep517', '--no-warn-script-location', '-r', 'C:\\Users\\USERNAME\\AppData\\Local\\Temp\\build-reqs-wthveb8h.txt']' returned non-zero exit status 1.

Workarounds

1. Use PIP_INDEX_URL environment variable

2. Force system global pip index in:
   Win: %HOME%/pip/pip.ini
   nix: $HOME/.config/pip/pip.conf

[global]
index_url = https://pypi.private.url

[search]
index_url = https://pypi.private.url

[iggyfisk]

Experiencing the same problem when trying to use a constraints file with pip-compile requirements.in --upgrade --resolver=backtracking --pip-args "-c constraints.txt".

My workaround has been (temporarily) adding -c constraints.txt to requirements.in

[atugushev]

--pip-args does not work in your case because pip-tools parses pyproject.toml metadata using build, not by pip. Pip passes some options to the isolated build environments (the mechanism could be changed in future though pypa/pip#9081), but pip-tools does not except --build-isolation. At first glance pip-tools could monkeypatch pip option environments (e.g., PIP_INDEX_URL) before calling build.utils.project_wheel_metadata() to fix the issue.

Setting PIP_INDEX_URL should definitely help. As another workaround, you could disable build isolation via pip-compile --no-build-isolation and manage the build environment manually.

[chrysle]

@atugushev Maybe we could introduce an own --index-url CLI option?

That would simplify things.

[atugushev]

Maybe we could introduce an own --index-url CLI option?

@chrysle pip-compile already has --index-url option:

$ pip-compile --help | grep -A1 -- --index-url
  -i, --index-url TEXT            Change index URL (defaults to
                                  https://pypi.org/simple)

However, the main issue is figuring out how to pass the index URL properly to build.utils.project_wheel_metadata().

[chrysle]

However, the main issue is figuring out how to pass the index URL properly to build.utils.project_wheel_metadata().

Wasn't there the golden rule that if StackOverflow doesn't know the answer, it's impossible? I think we should really set PIP_INDEX_URL then.

If you passed --index-url directly, but not via pip args, should that be used for the wheel metadata?