X-Amz-Content-SHA256 should be present when querying Amazon OpenSearch Serverless

[JCKortlang]

Describe the bug

Update: Issue described below with the VPCE is due to the VPCE mutating a signed header. X-Amz-Content-SHA256 header is still a relevant issue but not the one described below.

Only reproducible when querying an Amazon OpenSearchServerless collection with a private network policy (accessibly only via VPCE). I am unable to reproduce on a collection with a public network policy.

Method: GET, RequestUri: 'https://host-id.us-west-2.aoss.amazonaws.com/_cat/indices', Version: 1.1, Content: System.Net.Http.StringContent, Headers:

{

  Accept: */*
  accountid: 855676708012
  User-Agent: curl/8.4.0
  x-forwarded-for: 15.248.7.84
  x-forwarded-port: 443
  x-forwarded-proto: https
  X-Amz-Date: 20240422T182608Z
  x-amz-security-token:  <redacted>
  Host:host-id.us-west-2.aoss.amazonaws.com
  Authorization: AWS4-HMAC-SHA256 Credential=ASIA4OOSOMSWA3GBWWEE/20240422/us-west-2/aoss/aws4_request, SignedHeaders=accept;accountid;host;user-agent;x-amz-date;x-amz-security-token;x-forwarded-for;x-forwarded-port;x-forwarded-proto, Signature=ec37d19eaf6227efe60c26a8690242c2126ef5449ed29a3755105e939d93004b
  Content-Length: 0
  Content-Type: application/json; charset=utf-8

} with  made by arn:aws:sts::855676708012:assumed-role/InsightsStack-EtlStorageOpenSearchServerlessApiProx-UMqk35Yx0VO4/InsightsStack-EtlStorageOpenSearchServerlessApiPro-p6lr3QP2XtH5 failed with StatusCode: 403, ReasonPhrase: 'Forbidden', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:

{

  X-Request-ID: d8d04efc-0f9e-97d6-bf07-bc655fd42594
  Date: Mon, 22 Apr 2024 18:26:08 GMT
  x-aoss-response-hint: X01:gw-helper-deny
  Server: aoss-amazon
  Content-Type: application/json
  Content-Length: 121
} Forbidden is not successful with '{"status":403,"request-id":"d8d04efc-0f9e-97d6-bf07-bc655fd42594","error":{"reason":"403 Forbidden","type":"Forbidden"}}

--

Based on the documentation,

The following requirements apply when signing requests to OpenSearch Serverless collections when you construct HTTP requests with another clients.
You must specify the service name as aoss.
The x-amz-content-sha256 header is required for all AWS Signature Version 4 requests. It provides a hash of the request payload. If there's a request payload, set the value to its Secure Hash Algorithm (SHA) cryptographic hash (SHA256). If there's no request payload, set the value to e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, which is the hash of an empty string.

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-clients.html#serverless-signing

Expected Behavior

X-Amz-Content-SHA256 should be present when service identifier is 'aoss'

Current Behavior

X-Amz-Content-SHA256 is not present when querying Amazon OpenSearch Serverless

Reproduction Steps

Infra
Client -> APIG -> Lambda -> VPCE -> AOSS (private)

Via the APIGateway Service Console:

1. Create an API Gateway with a Lambda proxy integration.

Via the Lambda Service Console:

1. Create a Lambda function

Via the OpenSearch Service Console:

1. Create an Amazon OpenSearchServerless (AOSS) collection. When creating...
2. Configure the collection with a "private" network policy.
3. Create VPCE to access the private instance. Ensure the VPCE and Lambda are in the same subnet.
   1. Add an Inbound / Outbound rule for the Lambda SG
   2. Add an Inbound / Outbound rule for the AOSS SG
4. Configure the collection with a data access policy. Add the Lambda execution role to the policy.

---

Sample Lambda Code

using System.Net;
using System.Net.Security;
using System.Security.Authentication;
using System.Text;
using System.Text.Json;

using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.Core;
using Amazon.Lambda.Serialization.SystemTextJson;
using Amazon.Runtime;

using AWS.Lambda.Powertools.Logging;

using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;

using EnvironmentVariables = Insights.Shared.Variables.EnvironmentVariables;
using HttpMethod = System.Net.Http.HttpMethod;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(DefaultLambdaJsonSerializer))]

namespace Functions;

//You can simplify to just query aoss directly in the lambda. This function is a generic proxy.
public class Function
{
    private const string ServiceKey = "awsService";
    private const string ServiceUriKey = "awsUri";
    private static readonly string Region = "us-west-2";
    /// <summary>
    /// Filter out query parameters used to configure the proxy request as these may cause the receiving service to fail. e.g. OpenSearch returns BadRequest when there are excess parameters
    /// </summary>
    private static readonly IReadOnlySet<string> ReservedQueryParameterKeys = new HashSet<string>([ServiceKey, ServiceUriKey]);
    /// <summary>
    /// Headers which should not be forwarded.
    /// See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Connection
    /// See https://www.rfc-editor.org/rfc/rfc7230#section-6.1
    /// </summary>
    private static readonly IReadOnlySet<string> HopByHopHeaders = new HashSet<string>([
        //sigv4
        "authorization",
        //standard
        "host",
        "connection",
        "keep-alive",
        "proxy-authenticate",
        "proxy-authorization",
        "te",
        "trailer",
        "transfer-encoding",
        "upgrade"
    ]);

    private static readonly TimeSpan Timeout = TimeSpan.FromSeconds(30);
    private static HttpClient CreateHttpClient()
    {
        var handler = new SocketsHttpHandler
        {
            PooledConnectionLifetime = Timeout * 2,
            ConnectTimeout = Timeout,
            SslOptions = new SslClientAuthenticationOptions
            {
                EnabledSslProtocols = SslProtocols.None
            }
        };
        return new HttpClient(handler)
        {
            Timeout = Timeout
        };
    }

    public HttpClient HttpClient { get; init; } = CreateHttpClient();
    public ILogger Logger { get; init; } = AWS.Lambda.Powertools.Logging.Logger.Create<Function>();
    public Func<AWSCredentials> CredentialProvider { get; init; } = FallbackCredentialsFactory.GetCredentials;
    //Absurd levels of indirection to mock things out
    public Func<HttpClient, HttpRequestMessage, string, string, AWSCredentials, Task<HttpResponseMessage>> SignAndSendAsyncFunc { get; init; } = SignAndSendAsync;

    [Logging(LogEvent = true)]
    public async Task<APIGatewayHttpApiV2ProxyResponse> HandleRequestAsync(APIGatewayHttpApiV2ProxyRequest request, ILambdaContext context)
    {
        try
        {
            using HttpResponseMessage response = await this.SendRequestAsync(request);
            string content = await response.Content.ReadAsStringAsync();
            if (response.IsSuccessStatusCode == false)
            {
                string requestContent = await (response.RequestMessage?.Content?.ReadAsStringAsync() ?? Task.FromResult(string.Empty));
                this.Logger.LogError("'{Request}' with '{RequestContent}' failed with '{Response}' '{Content}'", response.RequestMessage, requestContent, response, content);
            }
            return new APIGatewayHttpApiV2ProxyResponse
            {
                StatusCode = (int) response.StatusCode,
                Headers = response.Headers.ToDictionary(kvp => kvp.Key, kvp => string.Join(",", kvp.Value)),
                Body = content,
                IsBase64Encoded = false
            };
        }
        catch (HttpRequestException e)
        {
            this.Logger.LogError(e, "Failed to proxy request to AWS service");
            return new APIGatewayHttpApiV2ProxyResponse
            {
                StatusCode = (int) (e.StatusCode ?? HttpStatusCode.InternalServerError),
                Headers = new Dictionary<string, string>
                {
                    { "Content-Type", "application/json; charset=utf-8" }
                },
                Body = JsonSerializer.Serialize(new ErrorResponse(e.StatusCode ?? HttpStatusCode.InternalServerError, e.Message, request.RequestContext.RequestId), new JsonSerializerOptions().ConfigureDefaults()),
                IsBase64Encoded = false
            };
        }
        catch (Exception e)
        {
            this.Logger.LogError(e, "Failed to proxy request to AWS service");
            return new APIGatewayHttpApiV2ProxyResponse
            {
                StatusCode = StatusCodes.Status400BadRequest,
                Headers = new Dictionary<string, string>
                {
                    { "Content-Type", "application/json; charset=utf-8" }
                },
                Body = JsonSerializer.Serialize(new ErrorResponse(HttpStatusCode.BadRequest, e.Message, request.RequestContext.RequestId), new JsonSerializerOptions().ConfigureDefaults()),
                IsBase64Encoded = false
            };
        }
    }

    private async Task<HttpResponseMessage> SendRequestAsync(APIGatewayHttpApiV2ProxyRequest apiRequest)
    {
        var headers = apiRequest.Headers
            .Select(kvp => new KeyValuePair<string, string>(kvp.Key.ToLowerInvariant(), kvp.Value))
            //Remove 'x-amz' and 'authorization' SigV4 headers as these will cause the signing to fail
            .Where(kvp => kvp.Key.StartsWith("x-amz") == false)
            .Where(kvp => HopByHopHeaders.Contains(kvp.Key) == false)
            .GroupBy(kvp => kvp.Key)
            .ToDictionary(group => group.Key, group => string.Join(',', group.Select(kvp => kvp.Value)));

        if (string.IsNullOrWhiteSpace(apiRequest.RequestContext.Http.Method))
        {
            throw new InvalidOperationException("APIGatewayHttpApiV2ProxyRequest.RequestContext.Http.Method is empty / not set");
        }
        var method = HttpMethod.Parse(apiRequest.RequestContext.Http.Method);

        bool isContentTypeRequired = HttpMethod.Put.Equals(method) || HttpMethod.Post.Equals(method) || HttpMethod.Patch.Equals(method);
        if (headers.TryGetValue("content-type", out string? contentType) == false || string.IsNullOrWhiteSpace(contentType))
        {
            if (isContentTypeRequired)
            {
                throw new InvalidOperationException($"APIGatewayProxyRequest.Header 'content-type' is empty / not set and required for method '{method}'");
            }
        }
        if (apiRequest.PathParameters.TryGetValue("proxy", out string? path) == false || string.IsNullOrWhiteSpace(path))
        {
            throw new InvalidOperationException("APIGatewayHttpApiV2ProxyRequest.PathParameters 'proxy' is empty / not set. Cannot forward request");
        }

        if (apiRequest.QueryStringParameters.TryGetValue(ServiceKey, out string? service) == false || string.IsNullOrWhiteSpace(service))
        {
            throw new InvalidOperationException(
                $"APIGatewayHttpApiV2ProxyRequest.QueryStringParameters '{ServiceKey}' is empty / not set. Cannot forward request. Expect service authorization token. " +
                $"e.g. 'aoss'. See https://docs.aws.amazon.com/service-authorization/latest/reference/reference_policies_actions-resources-contextkeys.html");
        }

        if (apiRequest.QueryStringParameters.TryGetValue(ServiceUriKey, out string? baseUri) == false || string.IsNullOrWhiteSpace(baseUri))
        {
            throw new InvalidOperationException(
                $"APIGatewayHttpApiV2ProxyRequest.QueryStringParameters '{ServiceUriKey}' is empty / not set. Cannot forward request. Expect absolute Uri. e.g. 'https://domain-id.us-west-2.aoss.amazonaws.com'");
        }

        var parameterList = apiRequest.QueryStringParameters
            .Where(kvp => ReservedQueryParameterKeys.Contains(kvp.Key) == false)
            .Select(kvp => $"{kvp.Key}={kvp.Value}")
            .ToList();
        string parameters = parameterList.Count == 0 ? string.Empty : "?" + string.Join("&", parameterList);
        //We need to add '/' because the proxy path parameter does not include it
        var requestUri = new Uri($"{baseUri}/{path}{parameters}", UriKind.Absolute);
        string body = (apiRequest.IsBase64Encoded ? Encoding.UTF8.GetString(Convert.FromBase64String(apiRequest.Body)) : apiRequest.Body) ?? string.Empty;
        var request = new HttpRequestMessage(method, requestUri)
        {
            Content = new StringContent(body)
        };

        foreach (KeyValuePair<string, string> header in headers)
        {
            //Content headers must go into the content headers else it throws
            if (header.Key.StartsWith("content"))
            {
                _ = request.Content.Headers.Remove(header.Key);
                _ = request.Content.Headers.TryAddWithoutValidation(header.Key, header.Value);
            }
            else
            {
                //Remove request defaults because .NET hates humanity
                _ = request.Headers.Remove(header.Key);
                _ = request.Headers.TryAddWithoutValidation(header.Key, header.Value);
            }
        }
        this.Logger.LogInformation("Sending {HttpRequestMessage} to {Service} at {Region}", request, service, Region);
        return await this.SignAndSendAsyncFunc.Invoke(this.HttpClient, request, Region, service, this.CredentialProvider.Invoke());
    }

    private static async Task<HttpResponseMessage> SignAndSendAsync(HttpClient client, HttpRequestMessage request, string region, string service, AWSCredentials credentials) =>
        await client.SendAsync(request, region, service, credentials);

    private sealed record ErrorResponse(HttpStatusCode StatusCode, string Message, string RequestId);
}

Possible Solution

Update https://github.com/FantasticFiasco/aws-signature-version-4/blob/master/src/Private/Signer.cs#L138

Additional Information/Context

Context

I am attempting to proxy requests through API Gateway to our private AOSS collection via VPCE.

Infra
Client -> APIG -> Lambda -> VPCE -> AOSS

Security

1. APIG
   1. IAM Authorizer
2. Lambda
   1. Execution Role
      1. Allows all aoss operations
   2. Verified it is in the same VPC / Subnets as VPCE
3. VPCE
   1. Security Group
      1. Allows Task Ingress / Egress
      2. Allows Lambda Ingress / Egress
      3. Allows AOSS Ingress / Egress
4. AOSS
   1. Network Policy - insightsstack-network-3cd87f61b1
      1. Allows VPCE
   2. DataAccess Policy - insightsstack-data-access-0c0bf1
      1. Allows ECS Task Role
      2. Allows Lambda Task Role

AWS .NET SDK and/or Package version used

3.7.303.38

Targeted .NET Platform

.NET 8

Operating System and version

MacOS 13.6.1 (22G313), Amazon Linux

[github-actions]

Hi there and welcome to this repository!

A maintainer will be with you shortly, but first and foremost I would like to thank you for taking the time to report this issue. Quality is of the highest priority for us, and we would never release anything with known defects. We aim to do our best but unfortunately you are here because you encountered something we didn't expect. Lets see if we can figure out what went wrong and provide a remedy for it.

[FantasticFiasco]

Hi @JCKortlang and thank you for the detailed issue! I'll try to build the environment and reproduce the error.

[JCKortlang]

@FantasticFiasco before you do. I'm engaging with the AOSS team and the issue appears to be caused by a signed 'x-forwarded-for' header and not the absence of the documented content header.

Unclear if the documented required header is actually required. May be worth adding if it doesn't break existing behavior.

[FantasticFiasco]

I'm happy that you're in contact with the responsible team! Let's give you the time to clarify the issue with the team, and once you're satisfied with their requirements, we'll implement them here.

[95horatio]

I love when you have a weird issue and you find a GitHub issue updated in the past day 😄

I had the same issue, I think AWS support might be steering you down the wrong path @JCKortlang because I'm also having this issue in C# with a Vector Search Collection in Amazon OpenSearch Serverless. Interestingly, I also get this issue using Amazon's own OpenSearch.Net/OpenSearch.Client, which made me jump over to AwsSignatureVersion4 in the first place.

Making the exact change recommended by @JCKortlang solved the problem for me. Specifically on line 138 of SIgner.cs:

request.AddHeaderIf(serviceName == ServiceName.S3, HeaderKeys.XAmzContentSha256Header, contentHash);

was changed to:

request.AddHeaderIf(serviceName == ServiceName.S3 || serviceName == ServiceName.OpenSearchServerless , HeaderKeys.XAmzContentSha256Header, contentHash);

And a corresponding entry was added to ServiceName:

internal const string OpenSearchServerless = "aoss";

And just like that, my searches started working :)

[JCKortlang]

For visibility, the issue with x-forwarded-for header is that it is:

1. The above proxy code is forwarding the header
2. The SigV4 signer from this repository is (correctly) using it to sign the request.
3. The header is then being mutated by the AWS system by adding an additional IP to the header (Wrong). Looks to be the VPCE
4. When the AOSS service tries to verify the signature, it includes the new value and causes the signature to differ. Thus the 403 response

Mitigation: Don't forward x-forwarded-* headers or exclude them from your request signing.

--

Based on the feedback from @95horatio looks like the adding of X-Amz-Content-SHA256 to this library's functionality is still relevant.

[FantasticFiasco]

v4.0.5 is now released on nuget.org. Thanks for reporting the issue!

[FantasticFiasco]

And I almost forgot, thanks for the excellent breakdown of the problem and the fix!