Add client-side TLS options support to Consul peer discovery

[michaelklishin]

This introduces TLS-related options support to Consul peer discovery, like 87bbf0c did for etc.

Closes #5116.

Kudos to @dcorbacho for helping me with Meck-based tests.

QA Environment

It took some effort to set up Consul with TLS for client connections, even though Consul docs are pretty great.

However, there are several caveats:

• There are configuration settings in the docs that Consul 1.12 reports as deprecated
• Certificate generation can be done using tls-gen but it might be easier to do it using Consul CLI
• HTTPS port must be enabled explicitly from the command line (in RabbitMQ it is done via the config file)
• TLS peer verification for HTTPS clients can be disabled separately from gRPC connections

Here is how I started Consul (a single server node with bootstrapping enabled):

# HTTPS port MUST be enabled explicitly even if config file has TLS settings
consul agent -config-file=./config.hcl -https-port 8501

Here is the rabbitmq.conf used:

cluster_formation.peer_discovery_backend = rabbitmq_peer_discovery_consul

# cluster_formation.consul.scheme = http
cluster_formation.consul.scheme = https
cluster_formation.consul.host = localhost
# uses HTTP
# cluster_formation.consul.port = 8500
# uses HTTPS
cluster_formation.consul.port = 8501
#
cluster_formation.consul.svc = rabbitmq

## do compute service address
# cluster_formation.consul.svc_addr_auto = true
## compute service address using node name
# cluster_formation.consul.svc_addr_use_nodename = true
## use long RabbitMQ node names?
# cluster_formation.consul.use_longname = true

cluster_formation.consul.svc_addr_auto = true
# cluster_formation.consul.svc_addr = warp10
cluster_formation.consul.use_longname = true

# cluster_formation.proxy.http_proxy = proxy.example.local
# cluster_formation.proxy.proxy_exclusions.1 = localhost
# cluster_formation.proxy.proxy_exclusions.2 = 127.0.0.1

cluster_formation.consul.svc_tags.1 = qa
cluster_formation.consul.svc_tags.2 = 3.8

cluster_formation.consul.svc_meta.owner = team-xyz
cluster_formation.consul.svc_meta.service = service-one
cluster_formation.consul.svc_meta.stats_url = https://service-one.eng.megacorp.local/stats/

# cluster_formation.consul.svc_ttl = 60

log.console.level = debug
log.file.level = debug

cluster_formation.consul.lock_prefix = rabbitmq-prefix

Here is the Consul config in HCL:

#
# General
#

server    = true
bootstrap = true

datacenter = "dc1"
data_dir = "./data"
log_level = "INFO"
node_name = "agent1"

server = true
ui_config {
  enabled = true
}

addresses {
  http = "0.0.0.0"
}

client_addr = "0.0.0.0",
bind_addr = "0.0.0.0",
advertise_addr = "127.0.0.1"

#
# TLS
#

# accept HTTPS connections without a certificate
verify_incoming = false
verify_incoming_rpc = true

verify_outgoing = true

# verify_server_hostname = true
ca_file = "consul-agent-ca.pem"
cert_file = "dc1-server-consul-0.pem"
key_file = "dc1-server-consul-0-key.pem"

[lukebakken]

Great!