'file:' identifier in '--requirements' argument yields FileNotFoundError

[mfansler]

Description

Using a file: identifier when providing a --requirements|-r argument results in FileNotFoundError.

Expected behavior

The file should be found.

Previous versions worked fine. Alternatively, omitting the file: identifier works.

pip version

21.2.1

Python version

3.9.6

OS

MacOS 10.14

How to Reproduce

1. Create and activate Conda environment with:

   conda create -n foo -c conda-forge python=3.9 pip=21.2.1
   conda activate foo

2. Create requirements.txt:

   echo "numpy" > requirements.txt

3. Run install command:

   pip install -r file:requirements.txt

4. Error.

Output

(foo) bash-3.2$ pip install -r file:requirements.txt

ERROR: Exception:                                                                                                                                                                         
Traceback (most recent call last):                                                                                                                                                        
  File "/Users/mfansler/miniconda3/envs/foo/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 173, in _main                                                            
    status = self.run(options, args)                                                                                                                                                      
  File "/Users/mfansler/miniconda3/envs/foo/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 203, in wrapper                                                           
    return func(self, options, args)                                                                                                                                                      
  File "/Users/mfansler/miniconda3/envs/foo/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 286, in run                                                              
    reqs = self.get_requirements(args, options, finder, session)                                                                                                                          
  File "/Users/mfansler/miniconda3/envs/foo/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 384, in get_requirements                                                  
    for parsed_req in parse_requirements(                                                                                                                                                 
  File "/Users/mfansler/miniconda3/envs/foo/lib/python3.9/site-packages/pip/_internal/req/req_file.py", line 135, in parse_requirements                                                   
    for parsed_line in parser.parse(filename, constraint):                                                                                                                                
  File "/Users/mfansler/miniconda3/envs/foo/lib/python3.9/site-packages/pip/_internal/req/req_file.py", line 319, in parse                                                                
    yield from self._parse_and_recurse(filename, constraint)                                                                                                                              
  File "/Users/mfansler/miniconda3/envs/foo/lib/python3.9/site-packages/pip/_internal/req/req_file.py", line 324, in _parse_and_recurse                                                   
    for line in self._parse_file(filename, constraint):                                                                                                                                   
  File "/Users/mfansler/miniconda3/envs/foo/lib/python3.9/site-packages/pip/_internal/req/req_file.py", line 353, in _parse_file                                                          
    _, content = get_file_content(filename, self._session)                                                                                                                                
  File "/Users/mfansler/miniconda3/envs/foo/lib/python3.9/site-packages/pip/_internal/req/req_file.py", line 518, in get_file_content                                                     
    resp = session.get(url)                                                                                                                                                               
  File "/Users/mfansler/miniconda3/envs/foo/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 555, in get                                                               
    return self.request('GET', url, **kwargs)                                                                                                                                             
  File "/Users/mfansler/miniconda3/envs/foo/lib/python3.9/site-packages/pip/_internal/network/session.py", line 454, in request                                                           
    return super().request(method, url, *args, **kwargs)                                                                                                                                  
  File "/Users/mfansler/miniconda3/envs/foo/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 542, in request                                                           
    resp = self.send(prep, **send_kwargs)                                                                                                                                                 
  File "/Users/mfansler/miniconda3/envs/foo/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 697, in send                                                              
    r.content                                                                                                                                                                             
  File "/Users/mfansler/miniconda3/envs/foo/lib/python3.9/site-packages/pip/_vendor/requests/models.py", line 836, in content                                                             
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''                                                                                                                
  File "/Users/mfansler/miniconda3/envs/foo/lib/python3.9/site-packages/pip/_vendor/requests/models.py", line 769, in generate                                                            
    chunk = self.raw.read(chunk_size)                                                                                                                                                     
AttributeError: 'FileNotFoundError' object has no attribute 'read'

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

file:requirements.txt is not a valid URL. The standard (RFC 8089) mandates the URL must starts with file:// or file: + absolute path. This is neither, so pip treats it as a local relative path and correctly fails.

Feel free to submit a PR for better error messages, but this is not a bug.

[DiddiLeija]

Feel free to submit a PR for better error messages, but this is not a bug.

Hi @uranusjr. If you want (and if no one is working on it, of course), I can open a pull request for the next week. Just tell me where to go to adjust the error message.

I'm not an expert with Conda (I actually don't use it on my work), but this issue doesn't seems to be really hard.

[uranusjr]

I think the conda env part is not relevant; this is only caused by appending file: to the file name. This probably hit some edge case in the path/URL parser. Previously pip would parse file:requirements.txt as a URL with file: scheme and relative path requirements.txt; now it’s (correctly) recognising this is not a valid URL, and treating the entire string as a local filename.

I traced the code a bit and it seems like the error is originated from LocalFSAdapter (used to provide file: URL support), around here:

pip/src/pip/_internal/network/session.py

Lines 218 to 235 in 87aee20

	try:
	stats = os.stat(pathname)
	except OSError as exc:
	resp.status_code = 404
	resp.raw = exc
	else:
	modified = email.utils.formatdate(stats.st_mtime, usegmt=True)
	content_type = mimetypes.guess_type(pathname)[0] or "text/plain"
	resp.headers = CaseInsensitiveDict(
	{
	"Content-Type": content_type,
	"Content-Length": stats.st_size,
	"Last-Modified": modified,
	}
	)
	resp.raw = open(pathname, "rb")
	resp.close = resp.raw.close

This send() function is used to send network requests and return a response object. When there’s an OSError, a 404 response is emitted, but resp.raw is set to the exception object. Which is not correct since, according to requests documentation, raw should be a file-like object. So this can probably be fixed by changing resp.raw = exc to something like

error_message = f"{type(exc).__name__}: {exc}"
resp.raw = io.BytesIO(error_message.encode("utf8"))

Or you can try out things in traceback to format the error message and see what result you like best.

[DiddiLeija]

Ok. I'll take a look on it later. Thanks @uranusjr! It doesn't seems too hard, so I can do it on these days.

[mfansler]

@DiddiLeija @pradyunsg thanks for addressing this! <3