Uncaught exception in http.server request handling (<=3.10)

[kenballus]

Sending a null byte in an HTTP request path or URL triggers an uncaught ValueError in CPython 3.10 and below.

Reproduction Instructions

Open two terminals and do the following (requires python <=3.10; current 3.10 branch included):

Terminal 1:

$ ./python -m http.server --bind 127.0.0.1
Serving HTTP on 127.0.0.1 port 8000 (http://127.0.0.1:8000/) ...

Terminal 2:

$ printf 'GET \x00 HTTP/1.1\r\n\r\n' | nc localhost 8000

Terminal 1:

----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 56900)
Traceback (most recent call last):
  File "/home/bkallus/clones/cpython/Lib/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "/home/bkallus/clones/cpython/Lib/http/server.py", line 1304, in finish_request
    self.RequestHandlerClass(request, client_address, self,
  File "/home/bkallus/clones/cpython/Lib/http/server.py", line 668, in __init__
    super().__init__(*args, **kwargs)
  File "/home/bkallus/clones/cpython/Lib/socketserver.py", line 747, in __init__
    self.handle()
  File "/home/bkallus/clones/cpython/Lib/http/server.py", line 433, in handle
    self.handle_one_request()
  File "/home/bkallus/clones/cpython/Lib/http/server.py", line 421, in handle_one_request
    method()
  File "/home/bkallus/clones/cpython/Lib/http/server.py", line 672, in do_GET
    f = self.send_head()
  File "/home/bkallus/clones/cpython/Lib/http/server.py", line 727, in send_head
    f = open(path, 'rb')
ValueError: embedded null byte
----------------------------------------

This bug is fixed in python 3.11+, but I haven't yet figured out why. As far as I can tell, the relevant portions of Lib/http/server.py and Lib/socketserver.py remain unchanged. I think the issue boils down to catching only OSErrors, but not ValueErrors, even though open can raise ValueErrors. I would guess that this happened because the docs do not clearly state that open(path, 'rb') can raise a ValueError.

Environment:

Arch Linux, x86_64
Python 3.10.10+

[CCLDArjun]

did some debugging:

looks like translate_path() in send_head() in 3.10 returns the path with a null character at the end, whereas in 3.11, it doesn't. Since in translate_path():

cpython/Lib/http/server.py

Line 848 in 9a8ce95

path = posixpath.normpath(path)

is called. the behavior of posixpath.normpath varies in 3.10 and 3.11:

python3.11 -c "import pathlib; print(ord(pathlib.posixpath.normpath('\x00')))"
46

python3.10 -c "import pathlib; print(ord(pathlib.posixpath.normpath('\x00')))"
0

We error on this line in 3.10 because open() doesn't like null characters in the file path

cpython/Lib/http/server.py

Line 727 in 9a8ce95

f = open(path, 'rb')

So, should we catch the ValueError, like @kenballus suggested? Unsure whether either of the posixpath.normpath()s are broken, but it's suspicious that they're returning different values.

[iritkatriel]

3.10 is in security-fix only mode now, so this is out of date.

[kenballus]

Oops! Was going based on the chart on python.org and didn't notice that 3.10.11 came out.