Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix aws-sigv4 canonical request formatting fallibility #1656

Conversation

ysaito1001
Copy link
Contributor

This PR attempts to address issue 711 where signing a request can cause panic! if a CanonicalRequest contains invalid UTF-8 in the header value.

The PR intentionally starts with a simple solution that makes a given test pass before we commit ourselves to start refactoring CanonicalRequest to support a special formatting function, which will probably be more expensive to do so. If the solution described in this PR works, that is great but if not, we can certainly switch to the refactoring ideas mentioned in the open issue.

The rationale behind the code changes is to make validating UTF-8 in the header value be part of an invariant of CanonicalRequest. That is, by the time CanonicalRequest has been successfully constructed by CanonicalRequest::from, we can guarantee that the type always contains valid UTF-8 in the header value. That way, we don't need to wait to catch invalid UTF-8 in the header value until we call to_string on CanonicalRequest (here).
That being said, the proposed approach may be inefficient because of eager checking of valid UTF-8 in the header value within normalize_header_value, especially when a CanonicalRequest does not need to be signed (in which case I assume to_string on the CanonicalRequest does not need to be called in the execution path).

This PR comes with a test provided in the open issue as well as a small test for normalize_header_value to exercise the part that has been updated.

I would appreciate if the reviewers with more background knowledge could assess the code changes from different perspectives.

This commit addresses panic in the case of formatting a CanonicalRequest
containing invalid UTF-8 in the header value.

The commit aims for the least disturbance to the codebase in order to
pass a given failing test in the PR. We want to quickly determine if
this low-cost approach addresses the problem before we commit ourselves
to start refactoring CanonicalRequest to have a special format function.

Fixes smithy-lang#711
@ysaito1001 ysaito1001 requested a review from a team as a code owner August 23, 2022 04:57
@rcoh
Copy link
Collaborator

rcoh commented Aug 23, 2022

this looks great! do you mind making a few small improvements:

  • since Display is still infallible, we should add proptesting or fuzzing to the input to the signer to ensure that no matter what the input is, we can't cause a runtime panic
  • will you add a test (and probably make a small update to the error) to ensure that the error that we emit gives a clear message about why signing failed? I think the current error probably won't indicate anything about the problem being a header

This commit converts test_signing_utf8_headers to a proptest. The
original test only specified hardcoded non-UTF8 bytes in a request
header. To ensure that `crate::http_request::sign` does not panic no
matter what valid byte sequence for HeaderValue is, the proptest covers
a wider range of inputs.

For consistency, the test has been moved from `canonical_request.rs` to
`sign.rs`.
This commit introduces an error type InvalidHeaderError to indicate that
we ran into a problem in handling a HeaderValue within CanonicalRequest.

The error type contains a source error such as Utf8Error so a diagnostic
message can be printed if needed.
@ysaito1001
Copy link
Contributor Author

Thanks @rcoh ! I took a stab at incorporating the feedback.

we should add proptesting or fuzzing to the input to the signer to ensure that no matter what the input is, we can't cause a runtime panic

I converted test_signing_utf8_headers to a proptest (renamed it and moved it to sign.rs) to exercise a wide range of inputs for request header values. As far as I can see, besides the the passed-in Formatter returning an error, the Display trait implementation for CanonicalRequest can only fail because of std::str::from_utf8 for HeaderValue. So I specifically targeted the proptest at that use case. However, what you requested seems like exercising a higher entity than HeaderValue. To be on the same page, if the proptest is not aligned with what you thought, could you tell me the name of system under test, i.e. a type or a function to test, so I can test it further.

will you add a test (and probably make a small update to the error) to ensure that the error that we emit gives a clear message about why signing failed?

I have added an error type InvalidHeaderError in the canonical_request module (since I was not sure whether I put it in the right place, please let me know if you want me to move InvalidHeaderError to its own error module or somewhere else). I have added a test in sign.rs to verify that if signing failed we should expect an InvalidHeaderError. I have used downcast_ref to compare the actual error type against the expected one but did not attempt to print out a diagnostic message and use it for verification as comparing against a raw string for verification can make the test brittle (please let me know if you think otherwise, I am open to suggestions). FYI, if we were to print an error when signing failed, it would look something like:

invalid UTF-8 contained in header value: invalid utf-8 sequence of 1 bytes from index 0

Copy link
Collaborator

@jdisanti jdisanti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good! I think can merge it as is, although it would be nice to improve the overall error situation in this crate.

This commit effectively reverts 739b32c. Knowing that we will be cleaning
up error types, having InvalidHeaderError is too narrow a solution and
does not add value to the codebase.
@ysaito1001 ysaito1001 requested a review from a team as a code owner October 18, 2022 21:55
@ysaito1001 ysaito1001 enabled auto-merge (squash) October 19, 2022 01:52
@ysaito1001 ysaito1001 disabled auto-merge October 19, 2022 02:19
@ysaito1001 ysaito1001 merged commit 521fa8d into smithy-lang:main Oct 19, 2022
@ysaito1001 ysaito1001 deleted the fix-panic-in-formatting-canonical-request branch October 19, 2022 02:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants