Access and modify req.body in request filters

[YOU54F]

What am I trying to do?

Trying to verify pacts against an AWS Lambda function, with an API gateway, that requires AWSv4 signed request headers

Problems I need to overcome

Signing Requests
To sign a request, you first calculate a hash (digest) of the request. Then you use the hash value, some other information from the request, and your secret access key to calculate another hash known as the signature. Then you add the signature to the request in one of the following ways:

Using the HTTP Authorization header.

Basically we need to pre-sign some headers based on information on our request

• For GET requests, Request query string is needed
• For POST requests, Request body is needed

We can acheive this with aws4 in a stateHandler for isAuthenticated, however I cannot get access to the pact under test's path or requestBody, which I need to pass to the pre-signed URL

What I've tried so far

Using the following Proposal: Allow customProviderHeaders to be dynamically added to different interactions , I have been able to get the verifier successfully working locally & in CI with a hardcoded request path, rather than the request path from the pact under test.

Examples in CI

CircleCI AWS Verification Step
AWS-Provider Pact
AWS-Provider Pact Verification Results

Steps taken in code

1. Generate temporary AWS credentials with a bash script and export to bash & run verify script

#!/bin/bash
set -o pipefail

  AWS_TEMP_CREDS=`aws sts assume-role --role-arn $ARN_ROLE --role-session-name api-gateway-access| jq -c '.Credentials'`
  export AWS_ACCESS_KEY_ID=`echo $AWS_TEMP_CREDS | jq -r '.AccessKeyId'`
  export AWS_SECRET_ACCESS_KEY=`echo $AWS_TEMP_CREDS | jq -r '.SecretAccessKey'`
  export AWS_SESSION_TOKEN=`echo $AWS_TEMP_CREDS | jq -r '.SessionToken'`

npx ts-node src/pact/verifier/verify.ts | grep -v Created

3. Pact is read, and state 'is authenticated' is met, passes over to the stateHandler.

• Request host, path and body need to be ascertained
• Request host comes from PROVIDER_BASE_URL which is set to https://3efkw1ju81.execute-api.us-east-2.amazonaws.com/default
• For GET requests, Request path needs to come from pact under test, currently hardcoded to default/helloworld
• For POST requests, Request body needs to come from pact under test

4. stateHandler for is Authenticated returns modified headers

let signedHost: string;
let signedXAmzSecurityToken: string;
let signedXAmzDate: string;
let signedAuthorization: string;
let authHeaders: any;

const opts: VerifierOptions = {
stateHandlers: {
  "Is authenticated": async () => {
    const requestUrl = process.env.PACT_PROVIDER_URL;
    const host = new url.URL(requestUrl).host;
    const apiroute = new url.URL(requestUrl).pathname;
    const pathname = `${apiroute}/helloworld`;
    const options = {
      host,
      path: pathname,
      headers: {}
    };
    await aws4.sign(options);
    aws4.sign(options, {
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
      sessionToken: process.env.AWS_SESSION_TOKEN
    });
    authHeaders = options.headers;
    signedHost = authHeaders.Host;
    signedXAmzSecurityToken = authHeaders["X-Amz-Security-Token"];
    signedXAmzDate = authHeaders["X-Amz-Date"];
    signedAuthorization = authHeaders.Authorization;
    return Promise.resolve(`AWS signed headers created`);
  },
  "Is not authenticated": async () => {
    signedHost = null;
    signedXAmzSecurityToken = null;
    signedXAmzDate = null;
    signedAuthorization = null;
    return Promise.resolve(`Blank aws headers created`);
  }
},

5. requestFilter will set amazon signed headers if they have been set

  requestFilter: (req, res, next) => {
    // over-riding request headers with AWS credentials
    if (signedHost != null) {
      req.headers.Host = signedHost;
    }
    if (signedXAmzSecurityToken != null) {
      req.headers["X-Amz-Security-Token"] = signedXAmzSecurityToken;
    }
    if (signedXAmzDate != null) {
      req.headers["X-Amz-Date"] = signedXAmzDate;
    }
    if (signedAuthorization != null) {
      req.headers.Authorization = signedAuthorization;
    }
    next();
  },

How can I get access to the path and body of the pact under test, in the stateHandler?

I logged out the req.path & req.body inside requestFilter

req.path /_pactSetup
req.body { consumer: 'consumer-service',
  state: 'Is authenticated',
  states: [ 'Is authenticated' ],
  params: {} }
creating AWS signed headers
created AWS signed headers
req.path /path/that/the/pact/test/is/calling
req.body undefined

It looks like

1. Pact setup is called req.path /_pactSetup with the body

{ consumer: 'consumer-service',
  state: 'Is authenticated',
  states: [ 'Is authenticated' ],
  params: {} }

2. The stateHandler is called

creating AWS signed headers
created AWS signed headers

3. The pact under tests, request path is called

req.path /helloworld
req.body undefined

4. For a post request, it might look like

req.path /helloworld
req.body {message:"hello world")

So my real question is, can the stateHandlers access the req object?

Cheers for any advice and help in advance!

[vgrigoruk]

I guess this will require changes in pact-ruby & pact-provider-verifier

[bernardobridge]

Actually, @YOU54F, a similar use case to yours was what originated me asking about being able to modify them dynamically (I was also working with Lambda behind an API Gateway with AWSv4 signing). I (wrongly) assumed that the state handlers they implemented had access to the request object, but it appears not. It means you can test the happy path with what they've implemented, but not the negative scenario!

I can share with you the workaround I had at the time, which would still work for you now! I wrote a blog post about it here: https://medium.com/dazn-tech/pact-contract-testing-dealing-with-authentication-on-the-provider-51fd46fdaa78 . Hopefully that helps!

But TLDR: instead of pulling the pacts directly from the broker, we point it to a local file instead. in a before hook to the pact verification, we pull the pact ourselves from the pact broker into a file, and then use a function to parse through it and add the request headers based on the request information. Since you'll have access to the provider states there too, you should be able to add custom logic based on that too (i.e. if "Is authenticated" -> add the header; otherwise, don't)

[YOU54F]

I've got this working nicely inside the requestFilter with access to the path for get requests, but I can't get access to the body of the pact under test for POST requests.

I'm not too fussed about the unhappy paths at the moment.

here is my req filter that works for GET requests

    requestFilter:  (req, res, next) => {
      const requestUrl =  PACT_PROVIDER_URL;
      const host =  new url.URL(requestUrl).host;
      const options = {
        host,
        path: '/Test' + req.path
        headers: {}
      };
       aws4.sign(options, {
        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
        accessKeyId: process.env.AWS_ACCESS_KEY_ID,
        sessionToken: process.env.AWS_SESSION_TOKEN
      });
      authHeaders =  options.headers;
      req.headers["Host"] =  authHeaders["Host"];
      req.headers["X-Amz-Security-Token"] =  authHeaders["X-Amz-Security-Token"];
      req.headers["X-Amz-Date"] =  authHeaders["X-Amz-Date"];
      req.headers["Authorization"] = authHeaders["Authorization"];
      next();
    },

For POST requests, I need to something like

    requestFilter:  (req, res, next) => {
      const requestUrl =  PACT_PROVIDER_URL;
      const host =  new url.URL(requestUrl).host;
      const options = {
        host,
        path: '/Test' + req.path,
        body: JSON.stringify(req.body),
        headers: {}
      };
       aws4.sign(options, {
        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
        accessKeyId: process.env.AWS_ACCESS_KEY_ID,
        sessionToken: process.env.AWS_SESSION_TOKEN
      });
      authHeaders =  options.headers;
      req.headers["Host"] =  authHeaders["Host"];
      req.headers["X-Amz-Security-Token"] =  authHeaders["X-Amz-Security-Token"];
      req.headers["X-Amz-Date"] =  authHeaders["X-Amz-Date"];
      req.headers["Authorization"] = authHeaders["Authorization"];
      next();
    },

but req.body is empty.

Reading up on Stack Overflow, as of Express v4.16, (https://stackoverflow.com/questions/11625519/how-to-access-the-request-body-when-posting-using-node-js-and-express#11631664)

we can use app.use(express.json())

which when placed above this line -

pact-js/src/dsl/verifier.ts

Line 148 in 9d118a8

app.use(this.config.requestFilter)

gives me access to the request body, in the requestFilter, with req.body, I can create the AWS header, with the body contents, but the verifier times out. This might be due to the changes, or due to my providers endpoint, which is painfully slow at times. (times out after 30 seconds, and tends to take a horrendous ~25 seconds to return)

Will do some more digging tomorrow

[YOU54F]

Well the data we have encoded in the pact is garbage (our term matchers generate the string "string" which throws a validation error if directly sent to the provider through postman, so I would expect the verifier to throw an error, and not time out)

[2019-06-12T22:33:11.879Z] DEBUG: pact@8.1.2/54019 on YOU54FMAC: Proxing /decision
{ Error: Timeout waiting for verification process to complete (PID: 54029)
    at Timeout._onTimeout (/Users/you54f/dev/compassdev/compass/compass-pact-provider/node_modules/q/q.js:1846:21)
    at listOnTimeout (internal/timers.js:535:17)
    at processTimers (internal/timers.js:479:7) code: 'ETIMEDOUT' }
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
➜  compass-pact-provider git:(awsVerify) ✗ 

which probably means there is something else amiss.

Time for a beer anyhoo =D

[combmag]

@YOU54F any updates please? since i'm still having the same issues for req.body as undefined

[mefellows]

Not all requests are JSON bodies (could be any type) so that parser hasn't been added. I'll double check, but I'm hopeful it looks at the media type and only attempts to parse a body into req.body if that's true, in which case I'll add that in as it's clearly going to be useful.

In the mean time, you can just read in the body yourself as you would any other node http request. Consult the node docs on how to do that.

[mefellows]

See https://stackoverflow.com/questions/62662605/how-to-modify-a-graphql-variable-in-a-contract-during-the-provider-verification for an example. Albeit it seems it won't let you modify the body anyway

[mefellows]

Summary of current issue.

req.body isn't automatically parsed. I think at the least, we should parse the body. Looking at the problem before, it may be an issue with the way the http proxy wires in to the process - right now, changing the body makes it hang. I suspect this is due to an event not being fired/closed on the event loop.

In any case, there are two issues:

1. req.body is not prepopulated for JSON bodies
2. Even if you parse the body (see https://stackoverflow.com/questions/62662605/how-to-modify-a-graphql-variable-in-a-contract-during-the-provider-verification) you can't modify it.

[combmag]

for now what i have done as a workaround is using an http proxy similar to https://github.com/http-party/node-http-proxy/blob/master/examples/middleware/bodyDecoder-middleware.js where i update the body based on some criteria, of course this is not the ideal solution but for now it's the best option i could think of

[zsotyooo]

Hi there,

You have an awesome product. I really enjoy working with PACT.

Do you have an ETA on this issue? The workaround from @combmag is something we can try as well, but we would rather avoid it because it adds a lot of complexity and hides the implementation from the tests.

[mefellows]

Hi Zsolt, thanks for the kind words!

Our v3 branch (the beta release you can find) has support for this. We hope to have it out of beta in the next couple of months, it is only lacking a few features (e.g. message support and Pact Web) and may have a few rough edges API wise.

[zsotyooo]

Hi @mefellows,

Thanks for the quick response. I'll give it a try.

Cheers

[m-radzikowski]

for now what i have done as a workaround is using an http proxy similar to https://github.com/http-party/node-http-proxy/blob/master/examples/middleware/bodyDecoder-middleware.js where i update the body based on some criteria, of course this is not the ideal solution but for now it's the best option i could think of

@combmag could you share a short snippet of how you set up the filter and the proxy? I'm having trouble getting it to work properly. I would appreciate it greatly.

[dineshk-qa]

Hi Zsolt, thanks for the kind words!

Our v3 branch (the beta release you can find) has support for this. We hope to have it out of beta in the next couple of months, it is only lacking a few features (e.g. message support and Pact Web) and may have a few rough edges API wise.

Hi @mefellows,
Great product you have here and it's the center point of many of our test suites here.
Eagerly waiting for this feature as a big chunk of work is pending due to this :)
Any updated timelines would be appreciated
I think, it would be better if we can access req.body in stateHandlers rather than requestFilter as this will give more control per transaction

[lonely-caat]

Summary of current issue.

req.body isn't automatically parsed. I think at the least, we should parse the body. Looking at the problem before, it may be an issue with the way the http proxy wires in to the process - right now, changing the body makes it hang. I suspect this is due to an event not being fired/closed on the event loop.

In any case, there are two issues:

1. `req.body` is not prepopulated for JSON bodies

2. Even if you parse the body (see https://stackoverflow.com/questions/62662605/how-to-modify-a-graphql-variable-in-a-contract-during-the-provider-verification) you can't modify it.

Hi @mefellows, really appreciate the work you've been doing this is an awesome tool!
that being said, any updates on the issue described here? we really need this functionality :(

[mefellows]

Thanks @dineshk-qa and @lonely-caat!

So we're currently focussed on our v3 upgrade (which has a new underlying Rust engine, replacing the current Ruby shared core) and several new features, whilst we look to extend Pact via plugins. You could give that a go, the headers and body can be mutated in this implementation: https://github.com/pact-foundation/pact-js#pact-js-v3.

If anyone was interested in revisiting the current code base, a PR would be welcome.

I think the main issue is in the actual http proxy we use currently, so that might need to be replaced (which could be a lot of work to prevent backwards incompatible behaviour).

[alexsotocx]

@mefellows Which version should i install in order to try that? I'm stuck because of the same reason of this issue. I'm at version "@pact-foundation/pact": "^9.15.3",

[alexsotocx]

I just tried the version v10.0.0-beta33 and still the same.

import * as path from 'path';
import { Verifier, VerifierOptions } from '@pact-foundation/pact';

const opts: VerifierOptions = {
  providerBaseUrl: process.env.PROVIDER_BASE_URL!,
  pactUrls: [path.resolve('packages/shared/pact-testing/contracts/loging-patient-rest-users.json')],
  logLevel: 'debug',
  requestFilter(req, _, next) {
    if (req.url === '/users/auth/login') {
      console.log('lol');
      req.body = {
        email: 'fakeuser@email.com',
        password: 'password1234',
      } as RestUsers.LoginReqDto;
    }

    next();
  }

};
const verifier = new Verifier(opts);
describe('Login provider verifier', () => {
  test('verifier', async () => {
    await verifier.verifyProvider();
  });
});

The body is not modified

[mefellows]

Apologies, I copied the wrong link @alexsotocx - it should be https://github.com/pact-foundation/pact-js#pact-js-v3. Please also note the new package import

[alexsotocx]

Apologies, I copied the wrong link @alexsotocx - it should be https://github.com/pact-foundation/pact-js#pact-js-v3. Please also note the new package import

Tested and working, thanks :)

[lonely-caat]

hey @mefellows, are there any news regarding the ability to modify the request body payload?

[mefellows]

Have you read the thread @lonely-caat ?

[lonely-caat]

@mefellows do you mean this message #304 (comment) from a month ago where you said that you have other priorities, and to check this out in alpha?

[mefellows]

Exactly, our focus is about the next major release. It's looking to be a big piece of work addressing this in the current lib, so it makes more sense on spending the effort on the next major version.

[lonely-caat]

that's cool, thank you. when is the next major release planned for?

[TimothyJones]

We’re hoping to get it out as soon as we can. It’s hard to put a time frame on it, because this is an open source project. Work largely happens in the spare time that maintainers and contributors have available. Personally, I’m hoping that initiatives like Pactflow (which I’m not involved with, but am pleased to see happen) and the occasional organisation dedicating (or contracting) someone to develop features they need mean that we’ll be able to knock a lot of these longer term goals off sooner than we have in the past. It’s an exciting time for the project- we’re currently seeing the most activity from contributors and community members since I joined. I can tell you that getting the next major release out is certainly the highest priority for pact-js at the moment.

…

Sent from my mobile

On 1 May 2021, at 5:17 pm, lonely-caat ***@***.***> wrote:  that's cool, thank you. when is the next major release planned for? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[mefellows]

FYI I have a spike branch that allows users to modify the request body in the request filters: https://github.com/pact-foundation/pact-js/tree/feat/request-filter-bodies.

I haven't touched it for a few weeks, but should be portable to both the current mainline and also the v3.x.x branch.

[YOU54F]

Nice one @mefellows I am going to finish the monster that I created with this issue xD have popped this on my list

[YOU54F]

We’re hoping to get it out as soon as we can. It’s hard to put a time frame on it, because this is an open source project. Work largely happens in the spare time that maintainers and contributors have available.

This! I balanced in between how long would it take me to make this tool do x, versus how long would it take me to roll something out myself.

If anyone reading this, who wants to see this and v3 and other features happen, get involved, we can't do this alone and it's obviously in all of our net interests to partake, so don't hesitate to reach out here or via slack.

[DaveClissold]

Have tested your nice little fix @mefellows in both v2 and v3, it all works as expected. @YOU54F not sure what I can do to help get this moving forward, does it just need a pr raising?

[mefellows]

Oh, if you can clean it up and raise a PR I would love that. It's a bit spikey, but if you're keen to tidy / clean it up and add some basic tests we'll get that in and shipped ASAP.

Also, thanks for confirming!

[YOU54F]

Hi @DaveClissold,

Amazing thanks for helping shift this along my good man!

You can reach out on https://slack.pact.io if you want to chat in more real-time. We could probably create a bit of a task list if that helps 👍🏾

We have a specific pact-js and pact-js-development channel with a fair few members.

[YOU54F]

Closing this issue down now as complete!,

Conclusion

• It is now possible to access the req.body in requestFilters and therefore sign AWS v4 requests for all method types
• Example shown here Aws gateway v4 testing YOU54F/template-jest-pact-typescript#428
• We will aim to increase the documentation coverage and provide some workable examples to support our existing recipes https://docs.pact.io/recipes/apigateway#use-case-3-signed-requests

If anyone wishes to discuss, feel free to reach out to us on here or via our Pact Slack

[YOU54F]

Just in case anyone lands on this issue

Created another e2e example here

https://github.com/you54f/aws-auth-pact

with the verifier code here

https://github.com/YOU54F/aws-auth-pact/blob/9e22d7f0b84fcf3d167adc104cbd1b122fd09e46/javascript/provider.test.ts#L44-L110

[mefellows]

Thanks saf! We should definitely add this to https://docs.pact.io/recipes. I'll take a look Monday :)

[orekav]

@YOU54F

I have looked at your example

        const authHeaders = options.headers;
        // console.log(authHeaders)
        req.headers['Host'] =
          authHeaders && authHeaders['Host']
            ? authHeaders['Host'].toString()
            : '';
        req.headers['X-Amz-Date'] =
          authHeaders && authHeaders['X-Amz-Date']
            ? authHeaders['X-Amz-Date'].toString()
            : '';
        req.headers['Authorization'] =
          authHeaders && authHeaders['Authorization']
            ? authHeaders['Authorization'].toString()
            : '';

        // The following is required if using AWS STS to assume a role
        req.headers['X-Amz-Security-Token'] =
          authHeaders && authHeaders['X-Amz-Security-Token']
            ? authHeaders['X-Amz-Security-Token'].toString()
            : '';

As it was it was not working for me, I had to add

        req.headers['Authorization'] =
          authHeaders && authHeaders['Authorization']
            ? authHeaders['Authorization'].toString()
            : '';

But then I have realised that it looks better if instead of setting the headers this way I do

   Object.assign(req.headers, signed.headers);

The entire piece looks like this

import * as express from "express";
import { Request, sign } from "aws4";
import { AssumeRoleCommand, STSClient } from "@aws-sdk/client-sts";
import { fromEnv } from "@aws-sdk/credential-provider-env";
import { AwsCredentialIdentity } from "@smithy/types";
import { URL } from "url";

const getRoleCredentials = async (
    roleArn: string
): Promise<AwsCredentialIdentity> => {
    const sts = new STSClient({});
    const { Credentials: credentials } = await sts.send(
        new AssumeRoleCommand({
            RoleArn: roleArn,
            RoleSessionName: "pact-verifier",
        })
    );

    if (
        !credentials ||
        !credentials.AccessKeyId ||
        !credentials.SecretAccessKey
    ) {
        throw new Error("Credentials are not defined");
    }

    return {
        accessKeyId: credentials.AccessKeyId,
        secretAccessKey: credentials.SecretAccessKey,
        sessionToken: credentials.SessionToken,
    };
};

const getCredentials = async (
    roleArn?: string
): Promise<AwsCredentialIdentity> =>
    roleArn ? getRoleCredentials(roleArn) : fromEnv()();

export const aws4SignAuth = async (
    apiEndpoint: string,
    requestOptions: Omit<Request, "host" | "path" | "headers">,
    roleArn?: string
): Promise<express.RequestHandler> => {
    const parsedUrl = new URL(apiEndpoint);
    const credentials = await getCredentials(roleArn);

    return (req, res, next) => {
        const options: Request = {
            host: parsedUrl.host,
            path: parsedUrl.pathname + req.path,
            headers: {},
            ...requestOptions,
            ...(req.method === "POST"
                ? {
                      body: String(req.body),
                      headers: { "Content-Type": "application/json" },
                  }
                : {}),
        };
        const signed = sign(options, credentials);
        Object.assign(req.headers, signed.headers);
        next();
    };
};

// usage
    const aws4SignMiddleware = await aws4SignAuth(apiUrl, {
        region: "eu-west-1",
        service: "appsync",
    });

About the usage, if like me you use Appsync Custom Domains, it does not get the service correctly (issue with the AWSv4 dependency --> returns appsync-api when using the default one) nor the region.

[orekav]

New update

export const aws4SignAuth = async (
    apiEndpoint: string,
    requestOptions: Omit<Request, "host" | "path" | "headers">,
    roleArn?: string
): Promise<express.RequestHandler> => {
    const parsedUrl = new URL(apiEndpoint);
    const credentials = await getCredentials(roleArn);

    return (req, res, next) => {
        const body =
            req.body instanceof Object
                ? JSON.stringify(req.body)
                : String(req.body);
        const options: Request = {
            host: parsedUrl.host,
            path: parsedUrl.pathname + req.path,
            headers: {},
            ...requestOptions,
            ...(req.method === "POST"
                ? {
                      body: Buffer.from(body),
                      headers: {
                          ...req.headers,
                          host: parsedUrl.host,
                      },
                  }
                : {}),
        };
        const signed = sign(options, credentials);
        Object.assign(req.headers, signed.headers);
        next();
    };
};