Support for TLS client settings for clustering

[tiagorossig]

PR Description

Clustering in TLS environments was not possible before these changes.

Why is this change required

My organization runs alloy pods in environments that require all pod communication to be over HTTPS with TLS encryption. When enabling clustering, we noticed that communication between pods was blocked because it was going over HTTP.

Testing

Built alloy from this branch and used the enable-tls CLI flag with the appropriate certificate material to configure the TLS client. Made sure that pods can still communicate amongst themselves and constantly converge on the cluster's state.

Some logs that confirm pod communication still works:

{"ts":"2024-09-20T22:36:54.802881244Z","level":"debug","service":"cluster","component":"memberlist","ts":"2024/09/20 22:36:54","caller":"[DEBUG] memberlist: Initiating push/pull sync with: <ip>.ec2.internal <pod1-ip>"}
{"ts":"2024-09-20T22:36:54.804031852Z","level":"debug","msg":"merging remote state","service":"cluster","remote_time":34}
{"ts":"2024-09-20T22:37:02.656126162Z","level":"debug","msg":"received DNS query response","service":"cluster","addr":"<cluster-address>","record_type":"A/AAAA","records_count":3}
{"ts":"2024-09-20T22:37:02.656152001Z","level":"debug","msg":"discovered peers","service":"cluster","peers_count":3,"peers":"<pod0-ip>,<pod1-ip>,<pod2-ip>"}
{"ts":"2024-09-20T22:37:02.656158483Z","level":"info","msg":"rejoining peers","service":"cluster","peers_count":3,"peers":"<pod0-ip>,<pod1-ip>,<pod2-ip>"}
{"ts":"2024-09-20T22:37:02.657273362Z","level":"debug","service":"cluster","component":"memberlist","ts":"2024/09/20 22:37:02","caller":"[DEBUG] memberlist: Initiating push/pull sync with:  <pod0-ip>"}
{"ts":"2024-09-20T22:37:02.658438004Z","level":"debug","msg":"merging remote state","service":"cluster","remote_time":35}
{"ts":"2024-09-20T22:37:02.65934668Z","level":"debug","service":"cluster","component":"memberlist","ts":"2024/09/20 22:37:02","caller":"[DEBUG] memberlist: Initiating push/pull sync with:  <pod1-ip>"}
{"ts":"2024-09-20T22:37:02.660418062Z","level":"debug","msg":"merging remote state","service":"cluster","remote_time":34}
{"ts":"2024-09-20T22:37:02.660630938Z","level":"debug","service":"cluster","component":"memberlist","ts":"2024/09/20 22:37:02","caller":"[DEBUG] memberlist: Stream connection from=<pod2-ip>"}
{"ts":"2024-09-20T22:37:02.660723496Z","level":"debug","service":"cluster","component":"memberlist","ts":"2024/09/20 22:37:02","caller":"[DEBUG] memberlist: Initiating push/pull sync with:  <pod2-ip>"}
{"ts":"2024-09-20T22:37:02.661081355Z","level":"debug","msg":"merging remote state","service":"cluster","remote_time":36}

Grafana Alloy debugging UI also shows that clustered pods can still recognize each other:

Which issue(s) this PR fixes

Fixes #367

Notes to the Reviewer

This PR depends on grafana/ckit#63, which has not yet been merged.

PR Checklist

• CHANGELOG.md updated
• Documentation added
• Tests updated
• Config converters updated

[mattdurham]

LGTM

[mattdurham]

Looks good to me, if you can resolve the conflict automerge will merge it.

[tiagorossig]

@mattdurham Much appreciated! Resolved the CHANGELOG.md conflict

[tiagorossig]

Edited to pass CI https://github.com/grafana/alloy/actions/runs/11131417334/job/30934202706?pr=1724

[tiagorossig]

@mattdurham Could you give me a hand taking a look at the failing test? It doesn't seem related to any changes here, so I'm a bit puzzled.