Requests signed with "connection" header are rejected by AWS

[cfbao]

Describe the bug

Requests with "connection" header are rejected by API Gateway HTTP API with IAM auth.

To Reproduce

• Create an HTTP API in API Gateway (e.g. aws apigatewayv2 create-api)
• Add IAM auth to this API
• Send a request with "connection: keep-alive" header to this API, signed using this package with appropriate AWS creds
• Receive 403 {"message":"Forbidden"} response

Sample code

Environment.SetEnvironmentVariable("AWS_PROFILE", "<my-profile>");

var client = new HttpClient();

var request = new HttpRequestMessage(HttpMethod.Get, "<api-gateway-http-api-url>")
{
	Headers = {
		{ "Connection", ["keep-alive"] },
	},
};

var response = await client.SendAsync(
	request,
	"<region>",
	"execute-api",
	Amazon.Runtime.FallbackCredentialsFactory.GetCredentials()
);

Console.WriteLine(response.StatusCode);
Console.WriteLine(await response.Content.ReadAsStringAsync());

Expected behavior

The request is accepted.

Desktop (please complete the following information):

Windows 10 & Amazon Linux 2023

Additional context

Although not explicitly documented, it looks like AWS simply doesn't accept some headers in the signature (they accept them in the request, but not in the signature calculation).
See an explicit case here with the "connection" header: https://repost.aws/questions/QUWXtAMiggShedgHG3hLl3tg/ses-sigv4-usage-update-connection-header

Other libraries (including AWS SDKs) deal with this by hardcoding (& maintaining) a list of unsignable headers and/or allow users to supply a list of headers to sign/not sign.
e.g.

• Give choice in signed headers mhart/aws4#119
• https://github.com/aws/aws-sdk-java-v2/blob/7da518ab63807ac633951afc39fb833b361d7aeb/core/auth/src/main/java/software/amazon/awssdk/auth/signer/internal/AbstractAws4Signer.java#L70-L71
• https://github.com/aws/aws-sdk-net/blob/acae479fe47dc2583f625fe9f3d68413573ee15b/sdk/src/Core/Amazon.Runtime/Internal/Auth/AWS4Signer.cs#L61-L66
• https://github.com/aws/aws-sdk-js/blob/cc29728c1c4178969ebabe3bbe6b6f3159436394/lib/signers/v4.js#L190-L198

Supporting customization also makes it possible to use this package in environments where a proxy may alter request headers.

[github-actions]

Hi there and welcome to this repository!

A maintainer will be with you shortly, but first and foremost I would like to thank you for taking the time to report this issue. Quality is of the highest priority for us, and we would never release anything with known defects. We aim to do our best but unfortunately you are here because you encountered something we didn't expect. Lets see if we can figure out what went wrong and provide a remedy for it.

[FantasticFiasco]

Hi @cfbao.

Thanks for providing this information, in such detail. I'd propose to split this work into two separate sequential steps. The first one would implement a fix for the current problem, where defined headers are skipped. Optimally we would refer to the list of headers maintained by the C# SDK, but if that isn't possible we can probably implement some kind of test that is asserting that our list contains the same elements as the SDKs.

The second step would be to implement the feature of being able to configure a list of ignored headers, without breaking backward compatibility.

Does this sound like an acceptable plan, and would you like to contribute any of the steps?

[cfbao]

Yeah that sounds reasonable.
I'm happy to contribute.

From a cursory look, it seems the list of ignored headers are private members in AWS .NET SDK, they also contain fewer elements than some other languages' SDKs, and notably doesn't contain the "connection" header. It's possible that due to the .NET SDK signing code never being exposed for external use, it can get away with a more limited list, which however won't be sufficient for this library.

I can put up a PR first and we can discuss the detailed approach there.

[FantasticFiasco]

Sounds awesome!

[FantasticFiasco]

I'm currently migrating to .NET 8, and then I'll release a new version for you.

Thank you for being patient, this took longer than it ought to have.

[FantasticFiasco]

Your fix has been included in v5.0.0. Thanks for the contribution!

[cfbao]

Never apologize for OSS work!
Thanks for merging my PR. I also really appreciate the integration tests you added.