Cherry-pick #16019 to 7.x: [Agent] Allow CA cert pinning on the Elasticsearch output or any code that user tlscommon.TLSConfig builder.

[ph]

Cherry-pick of PR #16019 to 7.x branch. Original message:

What does this PR do?

When multiples CA are presents on the system we cannot ensure that a
specific one was used to validate the chain exposed by the server.
This PRs adds a ca_sha256 option to the tlscommon.TLSConfig that is used by all
the code that has to create a TCP client with TLS support.

When the option is set, it will hook a new callback in the validation
chains that will inspect the verified and validated chains by Go to
ensure that a certificate in the chains matches the provided pin.

A usage example for the Elasticsearch output.

output.elasticsearch:
  hosts: [127.0.0.1:9200]
  ssl.ca_sha256: <base64_encoded_sha256>

You can generate the pin using the openssl binary with the
following command:

openssl x509 -in ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64

OpenSSL's documentation

You will need to start Elasticsearch with the following options

xpack.security.enabled: true
indices.id_field_data.enabled: true
xpack.license.self_generated.type: trial
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/pki/localhost/localhost.key"
xpack.security.http.ssl.certificate: /etc/pki/localhost/localhost.crt"
xpack.security.http.ssl.certificate_authorities: /etc/pki/ca/ca.crt"

This pull request also includes a new service in the docker-compose.yml
that will start a new Elasticsearch server with TLS and security
configured.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• Need to modify the configuration reference.
• Need to add the documentation.

How to test this PR locally

There are multiple ways to tests this, you can use the elasticsearch_ssl service defined in the docker-compose file that will start a configured Elasticsearch with TLS and the right CA/Certificate configured.

Or you can generate a CA and a server certificate using the elasticsearch-certutil, make sure to export them in the PEM format.

In both case you can use the following openssl command to generate the pin from the CA.

openssl x509 -in ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64

And use the following output configuration:

output.elasticsearch:
  hosts: [127.0.0.1:9200]
  username: <username>
  password: <password>
  ssl.ca_sha256: <base64_encoded_sha1>
  ssl.certificate_authorities: <path_to_ca>

Related issues

• Fixes: [Agent] Add CA certificate pinning to the elasticsearch output. #15717
• Relates: [Agent] The Checkin API need to verify the CA hash of the remote Kibana. #15718, [Agent] required changes for enroll #15716