From e3fa50d1c5d41f68f5b824788acbfedf8ccdb40b Mon Sep 17 00:00:00 2001 From: "Paul J. Dorn" Date: Wed, 31 Jul 2024 01:21:01 +0200 Subject: [PATCH 1/3] update docs --- docs/source/settings.rst | 118 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 111 insertions(+), 7 deletions(-) diff --git a/docs/source/settings.rst b/docs/source/settings.rst index 4e0c11877..a5bd51c58 100644 --- a/docs/source/settings.rst +++ b/docs/source/settings.rst @@ -210,7 +210,7 @@ H protocol s status B response length b response length or ``'-'`` (CLF format) -f referrer +f referer a user agent T request time in seconds M request time in milliseconds @@ -347,7 +347,7 @@ Format: https://docs.python.org/3/library/logging.config.html#logging.config.jso **Command line:** ``--log-syslog-to SYSLOG_ADDR`` -**Default:** ``'unix:///var/run/syslog'`` +**Default:** ``'udp://localhost:514'`` Address to send syslog messages. @@ -527,7 +527,7 @@ SSL certificate file SSL version to use (see stdlib ssl module's). -.. deprecated:: 20.2 +.. deprecated:: 21.0 The option is deprecated and it is currently ignored. Use :ref:`ssl-context` instead. ============= ============ @@ -569,7 +569,7 @@ Whether client certificate is required (see stdlib ssl module's) =========== =========================== --cert-reqs Description =========== =========================== -`0` no client verification +`0` no client veirifcation `1` ssl.CERT_OPTIONAL `2` ssl.CERT_REQUIRED =========== =========================== @@ -982,7 +982,7 @@ Following example shows a configuration file that sets the minimum TLS version t context.minimum_version = ssl.TLSVersion.TLSv1_3 return context -.. versionadded:: 20.2 +.. versionadded:: 21.0 Server Mechanics ---------------- @@ -1390,7 +1390,7 @@ Set a PasteDeploy global config variable in ``key=value`` form. The option can be specified multiple times. -The variables are passed to the the PasteDeploy entrypoint. Example:: +The variables are passed to the PasteDeploy entrypoint. Example:: $ gunicorn -b 127.0.0.1:8000 --paste development.ini --paste-global FOO=1 --paste-global BAR=2 @@ -1410,7 +1410,111 @@ Strip spaces present between the header name and the the ``:``. This is known to induce vulnerabilities and is not compliant with the HTTP/1.1 standard. See https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn. -Use with care and only if necessary. +Use with care and only if necessary. May be removed in a future version. + +.. versionadded:: 20.0.1 + +.. _permit-unconventional-http-method: + +``permit_unconventional_http_method`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Command line:** ``--permit-unconventional-http-method`` + +**Default:** ``False`` + +Permit HTTP methods not matching conventions, such as IANA registration guidelines + +This permits request methods of length less than 3 or more than 20, +methods with lowercase characters or methods containing the # character. +HTTP methods are case sensitive by definition, and merely uppercase by convention. + +This option is provided to diagnose backwards-incompatible changes. + +Use with care and only if necessary. May be removed in a future version. + +.. versionadded:: 22.0.0 + +.. _permit-unconventional-http-version: + +``permit_unconventional_http_version`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Command line:** ``--permit-unconventional-http-version`` + +**Default:** ``False`` + +Permit HTTP version not matching conventions of 2023 + +This disables the refusal of likely malformed request lines. +It is unusual to specify HTTP 1 versions other than 1.0 and 1.1. + +This option is provided to diagnose backwards-incompatible changes. +Use with care and only if necessary. May be removed in a future version. + +.. versionadded:: 22.0.0 + +.. _casefold-http-method: + +``casefold_http_method`` +~~~~~~~~~~~~~~~~~~~~~~~~ + +**Command line:** ``--casefold-http-method`` + +**Default:** ``False`` + +Transform received HTTP methods to uppercase + +HTTP methods are case sensitive by definition, and merely uppercase by convention. + +This option is provided because previous versions of gunicorn defaulted to this behaviour. + +Use with care and only if necessary. May be removed in a future version. + +.. versionadded:: 22.0.0 + +.. _header-map: + +``header_map`` +~~~~~~~~~~~~~~ + +**Command line:** ``--header-map`` + +**Default:** ``'drop'`` + +Configure how header field names are mapped into environ + +Headers containing underscores are permitted by RFC9110, +but gunicorn joining headers of different names into +the same environment variable will dangerously confuse applications as to which is which. + +The safe default ``drop`` is to silently drop headers that cannot be unambiguously mapped. +The value ``refuse`` will return an error if a request contains *any* such header. +The value ``dangerous`` matches the previous, not advisabble, behaviour of mapping different +header field names into the same environ name. + +Use with care and only if necessary and after considering if your problem could +instead be solved by specifically renaming or rewriting only the intended headers +on a proxy in front of Gunicorn. + +.. versionadded:: 22.0.0 + +.. _tolerate-dangerous-framing: + +``tolerate_dangerous_framing`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Command line:** ``--tolerate-dangerous-framing`` + +**Default:** ``False`` + +Process requests with both Transfer-Encoding and Content-Length + +This is known to induce vulnerabilities, but not strictly forbidden by RFC9112. + +Use with care and only if necessary. May be removed in a future version. + +.. versionadded:: 22.0.0 Server Socket ------------- From eda9d456d3dfe201617baabe09b7fa37a10ae8cc Mon Sep 17 00:00:00 2001 From: "Paul J. Dorn" Date: Thu, 30 May 2024 00:01:57 +0200 Subject: [PATCH 2/3] forbid lone CR/LF and NUL in headers New parser rule: refuse HTTP requests where a header field value contains characters that a) should never appear there in the first place, b) might have lead to incorrect treatment in a proxy in front, and c) might lead to unintended behaviour in applications. From RFC 9110 section 5.5: "Field values containing CR, LF, or NUL characters are invalid and dangerous, due to the varying ways that implementations might parse and interpret those characters; a recipient of CR, LF, or NUL within a field value MUST either reject the message or replace each of those characters with SP before further processing or forwarding of that message." --- gunicorn/config.py | 26 ++++++++++++++----- gunicorn/http/message.py | 7 +++++ .../invalid/invalid_field_value_01.http | 7 +++++ .../invalid/invalid_field_value_01.py | 5 ++++ .../valid/invalid_field_value_01_compat.http | 8 ++++++ .../valid/invalid_field_value_01_compat.py | 18 +++++++++++++ 6 files changed, 65 insertions(+), 6 deletions(-) create mode 100644 tests/requests/invalid/invalid_field_value_01.http create mode 100644 tests/requests/invalid/invalid_field_value_01.py create mode 100644 tests/requests/valid/invalid_field_value_01_compat.http create mode 100644 tests/requests/valid/invalid_field_value_01_compat.py diff --git a/gunicorn/config.py b/gunicorn/config.py index 144acaecc..a117a73e0 100644 --- a/gunicorn/config.py +++ b/gunicorn/config.py @@ -2254,7 +2254,7 @@ class StripHeaderSpaces(Setting): This is known to induce vulnerabilities and is not compliant with the HTTP/1.1 standard. See https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn. - Use with care and only if necessary. May be removed in a future version. + Use with care and only if necessary. Deprecated; scheduled for removal in 25.0.0 .. versionadded:: 20.0.1 """ @@ -2274,9 +2274,13 @@ class PermitUnconventionalHTTPMethod(Setting): methods with lowercase characters or methods containing the # character. HTTP methods are case sensitive by definition, and merely uppercase by convention. - This option is provided to diagnose backwards-incompatible changes. + If unset, Gunicorn will apply nonstandard restrictions and cause 400 response status + in cases where otherwise 501 status is expected. While this option does modify that + behaviour, it should not be depended upon to guarantee standards-compliant behaviour. + Rather, it is provided temporarily, to assist in diagnosing backwards-incompatible + changes around the incomplete application of those restrictions. - Use with care and only if necessary. May be removed in a future version. + Use with care and only if necessary. Temporary; scheduled for removal in 24.0.0 .. versionadded:: 22.0.0 """ @@ -2296,7 +2300,8 @@ class PermitUnconventionalHTTPVersion(Setting): It is unusual to specify HTTP 1 versions other than 1.0 and 1.1. This option is provided to diagnose backwards-incompatible changes. - Use with care and only if necessary. May be removed in a future version. + Use with care and only if necessary. Temporary; the precise effect of this option may + change in a future version, or it may be removed altogether. .. versionadded:: 22.0.0 """ @@ -2316,7 +2321,7 @@ class CasefoldHTTPMethod(Setting): This option is provided because previous versions of gunicorn defaulted to this behaviour. - Use with care and only if necessary. May be removed in a future version. + Use with care and only if necessary. Deprecated; scheduled for removal in 24.0.0 .. versionadded:: 22.0.0 """ @@ -2378,7 +2383,16 @@ class TolerateDangerousFraming(Setting): This is known to induce vulnerabilities, but not strictly forbidden by RFC9112. - Use with care and only if necessary. May be removed in a future version. + In any case, the connection is closed after the malformed request, + as it is unclear if and at which boundary additional requests start. + + Use with care and only if necessary. + Temporary; will be changed or removed in a future version. .. versionadded:: 22.0.0 + .. versionchanged: 22.1.0 + The newly added rejection of invalid and dangerous characters CR, LF and NUL in + header field values is also controlled with this setting. rfc9110 permits both + rejecting and SP-replacing. With this option set, Gunicorn passes the field value + unchanged. With this option unset, Gunicorn rejects the request. """ diff --git a/gunicorn/http/message.py b/gunicorn/http/message.py index 88ffa5a25..3f172a5a2 100644 --- a/gunicorn/http/message.py +++ b/gunicorn/http/message.py @@ -28,6 +28,7 @@ METHOD_BADCHAR_RE = re.compile("[a-z#]") # usually 1.0 or 1.1 - RFC9112 permits restricting to single-digit versions VERSION_RE = re.compile(r"HTTP/(\d)\.(\d)") +RFC9110_5_5_INVALID_AND_DANGEROUS = re.compile(r"[\0\r\n]") class Message(object): @@ -121,6 +122,12 @@ def parse_headers(self, data, from_trailer=False): value.append(curr.strip("\t ")) value = " ".join(value) + if RFC9110_5_5_INVALID_AND_DANGEROUS.search(value): + if not self.cfg.tolerate_dangerous_framing: + raise InvalidHeader(name) + # value = RFC9110_5_5_INVALID_AND_DANGEROUS.sub(" ", value) + self.force_close() + if header_length > self.limit_request_field_size > 0: raise LimitRequestHeaders("limit request headers fields size") diff --git a/tests/requests/invalid/invalid_field_value_01.http b/tests/requests/invalid/invalid_field_value_01.http new file mode 100644 index 000000000..0cae83069 --- /dev/null +++ b/tests/requests/invalid/invalid_field_value_01.http @@ -0,0 +1,7 @@ +GET / HTTP/1.1\r\n +Host: x\r\n +Newline: a\n +Content-Length: 26\r\n +GET / HTTP/1.1\n +Host: x\r\n +\r\n diff --git a/tests/requests/invalid/invalid_field_value_01.py b/tests/requests/invalid/invalid_field_value_01.py new file mode 100644 index 000000000..95b0581ae --- /dev/null +++ b/tests/requests/invalid/invalid_field_value_01.py @@ -0,0 +1,5 @@ +from gunicorn.config import Config +from gunicorn.http.errors import InvalidHeader + +cfg = Config() +request = InvalidHeader diff --git a/tests/requests/valid/invalid_field_value_01_compat.http b/tests/requests/valid/invalid_field_value_01_compat.http new file mode 100644 index 000000000..eab2b5f1f --- /dev/null +++ b/tests/requests/valid/invalid_field_value_01_compat.http @@ -0,0 +1,8 @@ +GET / HTTP/1.1\r\n +Host: x\r\n +Newline: a\n +Content-Length: 26\r\n +X-Forwarded-By: broken-proxy\r\n\r\n +GET / HTTP/1.1\n +Host: x\r\n +\r\n diff --git a/tests/requests/valid/invalid_field_value_01_compat.py b/tests/requests/valid/invalid_field_value_01_compat.py new file mode 100644 index 000000000..9621bc0f4 --- /dev/null +++ b/tests/requests/valid/invalid_field_value_01_compat.py @@ -0,0 +1,18 @@ +from gunicorn.config import Config + +cfg = Config() +cfg.set("tolerate_dangerous_framing", True) + +req1 = { + "method": "GET", + "uri": uri("/"), + "version": (1, 1), + "headers": [ + ("HOST", "x"), + ("NEWLINE", "a\nContent-Length: 26"), + ("X-FORWARDED-BY", "broken-proxy"), + ], + "body": b"" +} + +request = [req1] From 70a1e437b5119c9297c79ce1d599a9c8f62086dd Mon Sep 17 00:00:00 2001 From: "Paul J. Dorn" Date: Wed, 31 Jul 2024 17:39:13 +0200 Subject: [PATCH 3/3] forbid lone CR/LF and NUL in headers (docs) --- docs/source/settings.rst | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/docs/source/settings.rst b/docs/source/settings.rst index a5bd51c58..ec1d2b457 100644 --- a/docs/source/settings.rst +++ b/docs/source/settings.rst @@ -1410,7 +1410,7 @@ Strip spaces present between the header name and the the ``:``. This is known to induce vulnerabilities and is not compliant with the HTTP/1.1 standard. See https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn. -Use with care and only if necessary. May be removed in a future version. +Use with care and only if necessary. Deprecated; scheduled for removal in 25.0.0 .. versionadded:: 20.0.1 @@ -1429,9 +1429,13 @@ This permits request methods of length less than 3 or more than 20, methods with lowercase characters or methods containing the # character. HTTP methods are case sensitive by definition, and merely uppercase by convention. -This option is provided to diagnose backwards-incompatible changes. +If unset, Gunicorn will apply nonstandard restrictions and cause 400 response status +in cases where otherwise 501 status is expected. While this option does modify that +behaviour, it should not be depended upon to guarantee standards-compliant behaviour. +Rather, it is provided temporarily, to assist in diagnosing backwards-incompatible +changes around the incomplete application of those restrictions. -Use with care and only if necessary. May be removed in a future version. +Use with care and only if necessary. Temporary; scheduled for removal in 24.0.0 .. versionadded:: 22.0.0 @@ -1450,7 +1454,8 @@ This disables the refusal of likely malformed request lines. It is unusual to specify HTTP 1 versions other than 1.0 and 1.1. This option is provided to diagnose backwards-incompatible changes. -Use with care and only if necessary. May be removed in a future version. +Use with care and only if necessary. Temporary; the precise effect of this option may +change in a future version, or it may be removed altogether. .. versionadded:: 22.0.0 @@ -1469,7 +1474,7 @@ HTTP methods are case sensitive by definition, and merely uppercase by conventio This option is provided because previous versions of gunicorn defaulted to this behaviour. -Use with care and only if necessary. May be removed in a future version. +Use with care and only if necessary. Deprecated; scheduled for removal in 24.0.0 .. versionadded:: 22.0.0 @@ -1512,9 +1517,18 @@ Process requests with both Transfer-Encoding and Content-Length This is known to induce vulnerabilities, but not strictly forbidden by RFC9112. -Use with care and only if necessary. May be removed in a future version. +In any case, the connection is closed after the malformed request, +as it is unclear if and at which boundary additional requests start. + +Use with care and only if necessary. +Temporary; will be changed or removed in a future version. .. versionadded:: 22.0.0 +.. versionchanged: 22.1.0 + The newly added rejection of invalid and dangerous characters CR, LF and NUL in + header field values is also controlled with this setting. rfc9110 permits both + rejecting and SP-replacing. With this option set, Gunicorn passes the field value + unchanged. With this option unset, Gunicorn rejects the request. Server Socket -------------