Add option to filter entire payload using whitelist approach

[fundead]

By request, a new config option filter_payload_with_whitelist (set to false by default) which reuses filter_parameters, but changes it to sanitize all values if they are not present in that array. This is applied recursively and for all nodes in the payload, e.g both branch and leaf nodes, barring the required occurredOn, details and client hashes.

Also includes a fix when running tests locally without a Rails gem installed (rails_env gets a nil tag appended).

[nikz]

@fundead why are you filtering the entire payload?

[fundead]

@nikz Business requirement from a customer - we'll be linking this to them to see if it suits their needs.

[nikz]

@fundead roger that, whitelist seems like a 👍 idea but the whole payload seems a bit extreme, are we sure we can't just do the params?

[tmorton]

(customer here)
👍 This looks great. If I can quibble with the details a little, I wouldn't overload the filter_parameters config to be both the blacklist of the params (original functionality), and a whitelist of the payload (new functionality). Maybe add a whitelist_payload config for the new whitelist?

[fundead]

Thanks for reviewing that @tmorton , and that does sounds like a good improvement. I'll update this to add a separate config value as opposed to the overloading approach.

[nikz]

@fundead how do you feel about a quick test that the querystring is NOT filtered if you haven't specified the whitelist option? If you were expecting that data in your error report and it got filtered out I can imagine that being disappointing.

[tmorton]

This change looks great, we're excited to use it. Any idea when will be merged?

[UberMouse]

Hey @tmorton, I'm a new dev at Raygun with Ruby experience so I've taken this over. Because this existing approach was proving a bit complicated and not as flexible as it could be I'm working on a new approach. Basically, instead of just being to provide a list of keys to be filtered out I'm making it so you can provide a payload "shape" with the specific keys that you want to keep in it.

For example,

 {
      machineName: true,
      version: true,
      client: true,
      error: {
        className: true,
        message: true,
        stackTrace: true
      },
      userCustomData: true,
      tags: true,
      request: {
        hostName: true,
        url: true,
        httpMethod: true,
        iPAddress: true,
        queryString: {
          param1: true,
          param2: true,
        },
        headers: {
          "Host" => true,
          "Connection" => true,
          "Upgrade-Insecure_requests" => true,
          "User-Agent" => false,
          "Accept" => true,
        },
        form: {
          controller: true,
          action: false
        },
        rawData: {
          controller: true,
          action: false
        }
      },
      user: false,
}

With this, you will be able to whitelist keys, but still if you want to be explicit blacklist them as well. This applies to the whole payload that the raygun4ruby client creates to send to the Raygun API.

Let me know if you have any feedback, but I'm making good progress so I hope to have it in your hands soon :)

[UberMouse]

Hey @tmorton,

I think I've finished up this functionality, here's some example configuration and output from my test Rails app.

https://gist.github.com/UberMouse/f04471e65ee3096c4321db99a7d7631a

If you're happy with this approach I can go ahead and merge it into master.

Taylor

[tmorton]

👍 This looks great.

[UberMouse]

Great @tmorton, I'll try and do a release today but I'm going to roll a few more issues into it so probably not until towards the end of my day (17:00 GMT+13)

[UberMouse]

Hey @tmorton, I've pushed a new release (1.2.0) with this change and a few others to Ruby Gems so you should be able to use it now.

[tmorton]

Thanks @UberMouse - we'll be testing it out in the next few days.