Add support for self-signed certificates

[sagar-khanna]

We have HAProxy setup between the datadog agents and datadoghq which is using a self-signed certificate.
At the moment there is no provision on this line given below to pass OpenSSL::SSL::VERIFY_NONE or something similar to handle self-signed certificates in case of HAProxy or similar:
https://github.com/DataDog/dogapi-rb/blob/9ccd737200d693a5a817f04b32b3c00aec6c8697/lib/dogapi/common.rb#L99

Please note: The root CA is already installed at the global os level.

Raising this issue so we can add support in this gem once the related issue (to add support) in dogapi is resolved:
DataDog/dogapi-rb#243

We can achieve this by either reading a chef attribute or using the datadog.yaml config (skip_ssl_validation: true) or something similar to make it work.

[albertvaka]

Ruby uses the system trust store, so it should accept your self-signed certificate if it's trusted by the system and it's valid (ie: it matches the domain name it's being used for, it's not expired, the chain of trust is correct...).

Note that you can set node['datadog']['url'] to point it to your HAProxy address, instead of trying to "intercept" traffic for datadoghq.com (since your self-signed certificate might not be valid for datadoghq.com, which might be the problem) and both the Agent and chef-handler-datadog will use that setting.

Adding an option to disable SSL verification to dogapi-rb is not out of the equation (given that such an option exists in the Agent), but note, however, that this disables most of the security you get by using TLS and it is strongly discouraged.

[sagar-khanna]

Hi @albertvaka ... thanks for responding.

The system trust store has the valid cert as I can run curl and wget without any issues.

We are already setting node['datadog']['url'] to HAProxy address that's why its picking up the HAProxy address and failing because of the self-signed certificate. I can confirm that it is point to the HAProxy address in the agent datadog.yaml file.

We are ok with either adding disable SSL verification or explicitly passing the trust store location which would mean that you may need to update the client initialisation with it if set.

[albertvaka]

The following code is what dogapi-rb does in ruby. Can you run this from irb and tell us if it fails with the same error?

uri = URI.parse('<yourInternalEndpoint>')
session = Net::HTTP.new(uri.host, uri.port)
session.use_ssl = (uri.scheme == 'https')
req = Net::HTTP::Get.new('/api/v1/metrics')
resp = session.request(req)

Sorry to insist on this 🙏 but I really want to understand what's going on, even if disabling SSL works for you. The code above should be really similar to doing a curl, so I'm even more confused if curl works but this doesn't.

Also, is there any reason you are using https and not http if this is an internal endpoint?

[sagar-khanna]

$ /opt/chef/embedded/bin/irb
irb(main):001:0> uri = URI.parse('https://<REDACTED>:<PORT>')
irb(main):002:0> session = Net::HTTP.new(uri.host, uri.port)
Traceback (most recent call last):
        4: from /opt/chef/embedded/bin/irb:23:in `<main>'
        3: from /opt/chef/embedded/bin/irb:23:in `load'
        2: from /opt/chef/embedded/lib/ruby/gems/2.7.0/gems/irb-1.2.3/exe/irb:11:in `<top (required)>'
        1: from (irb):2
NameError (uninitialized constant Net)
Did you mean?  Set
irb(main):003:0> require 'net/http'
=> true
irb(main):004:0> session = Net::HTTP.new(uri.host, uri.port)
irb(main):005:0> session.use_ssl = (uri.scheme == 'https')
irb(main):006:0> req = Net::HTTP::Get.new('/api/v1/metrics')
irb(main):007:0> resp = session.request(req)
Traceback (most recent call last):
       10: from /opt/chef/embedded/bin/irb:23:in `<main>'
        9: from /opt/chef/embedded/bin/irb:23:in `load'
        8: from /opt/chef/embedded/lib/ruby/gems/2.7.0/gems/irb-1.2.3/exe/irb:11:in `<top (required)>'
        7: from (irb):7
        6: from /opt/chef/embedded/lib/ruby/2.7.0/net/http.rb:1483:in `request'
        5: from /opt/chef/embedded/lib/ruby/2.7.0/net/http.rb:932:in `start'
        4: from /opt/chef/embedded/lib/ruby/2.7.0/net/http.rb:943:in `do_start'
        3: from /opt/chef/embedded/lib/ruby/2.7.0/net/http.rb:1009:in `connect'
        2: from /opt/chef/embedded/lib/ruby/2.7.0/net/protocol.rb:44:in `ssl_socket_connect'
        1: from /opt/chef/embedded/lib/ruby/2.7.0/net/protocol.rb:44:in `connect_nonblock'
OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate))
irb(main):008:0>
$ wget 'https://<REDACTED>:<PORT>/api/v1/metrics'
--2020-12-07 19:06:48--  https://<REDACTED>:<PORT>/api/v1/metrics
Resolving <REDACTED> (<REDACTED>)... <REDACTED>...
Connecting to <REDACTED> (<REDACTED>)... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-12-07 19:06:48 ERROR 404: Not Found.

$

[sagar-khanna]

@albertvaka - output above. I've also provided wget output that's working.

I agree that we could use http given its all suppose to be internal however there's a company-wide mandate to use https everywhere.

[albertvaka]

👍 Thanks a lot for your help, it really looks like Ruby and wget don't act the same way... 😕 I don't want to annoy you with more questions, so I will implement the fix now and add to my backlog to investigate this Ruby problem with a custom CA later. Thanks for your patience 🙇

[albertvaka]

Can you try the new version? If you are using chef to deploy the handler you should pass default['datadog']['handler_extra_config'] = { : skip_ssl_validation => true }.