[Agent] Allow CA cert pinning on the Elasticsearch output or any code that user tlscommon.TLSConfig builder.

[ph]

What does this PR do?

When multiples CA are presents on the system we cannot ensure that a
specific one was used to validate the chain exposed by the server.
This PRs adds a ca_sha256 option to the tlscommon.TLSConfig that is used by all
the code that has to create a TCP client with TLS support.

When the option is set, it will hook a new callback in the validation
chains that will inspect the verified and validated chains by Go to
ensure that a certificate in the chains matches the provided pin.

A usage example for the Elasticsearch output.

output.elasticsearch:
  hosts: [127.0.0.1:9200]
  ssl.ca_sha256: <base64_encoded_sha256>

You can generate the pin using the openssl binary with the
following command:

openssl x509 -in ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64

OpenSSL's documentation

You will need to start Elasticsearch with the following options

xpack.security.enabled: true
indices.id_field_data.enabled: true
xpack.license.self_generated.type: trial
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/pki/localhost/localhost.key"
xpack.security.http.ssl.certificate: /etc/pki/localhost/localhost.crt"
xpack.security.http.ssl.certificate_authorities: /etc/pki/ca/ca.crt"

This pull request also includes a new service in the docker-compose.yml
that will start a new Elasticsearch server with TLS and security
configured.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• Need to modify the configuration reference.
• Need to add the documentation.

How to test this PR locally

There are multiple ways to tests this, you can use the elasticsearch_ssl service defined in the docker-compose file that will start a configured Elasticsearch with TLS and the right CA/Certificate configured.

Or you can generate a CA and a server certificate using the elasticsearch-certutil, make sure to export them in the PEM format.

In both case you can use the following openssl command to generate the pin from the CA.

openssl x509 -in ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64

And use the following output configuration:

output.elasticsearch:
  hosts: [127.0.0.1:9200]
  username: <username>
  password: <password>
  ssl.ca_sha256: <base64_encoded_sha1>
  ssl.certificate_authorities: <path_to_ca>

Related issues

• Fixes: [Agent] Add CA certificate pinning to the elasticsearch output. #15717
• Relates: [Agent] The Checkin API need to verify the CA hash of the remote Kibana. #15718, [Agent] required changes for enroll #15716

[michalpristas]

Looks good

[elasticmachine]

Pinging @elastic/ingest (Project:fleet)

[ph]

@michalpristas I have addresses your concerns and made the PR reader for review.

[ph]

As discussed from last week @scunningham I ping you on the pull request.

[scunningham]

was expecting fingerprint as typically calculated in openssl, but the way you did it is smaller

openssl x509 -sha256 -fingerprint -in ca.crt -noout
SHA256 Fingerprint=4F:3C:B4:4E:BB:2B:3C:02:4F:DA:95:EA:81:19:3F:F5:76:75:D4:D6:C6:C1:A3:B9:46:57:24:AC:37:56:39:70

[ph]

I was using the hpkp strategy to make it smaller, I don't have a strong argument for going that route other than might be a bit better on the CLI. What do you think?

[scunningham]

Smaller is better; I like the way you did it..

[ph]

@urso When this goes it this will affect any client that uses tlscommon.TLSConfig I think its a good idea that you take a look.

[ph]

Jenkins test this please

[ph]

updated changelog, merging.