Added PeerManagement.Type = 'etcd'

[Baliedge]

Which problem is this PR solving?

This PR offers an alternative to Redis for centralized peer discovery using an etcd service.

Short description of the changes

• This PR builds on the changes in Added PeerManagement.Timeout config option #491
• Added three new config options: xxx
• Added Close(ctx context.Context) error to the Peers interface to allow for clean shutdown and de-registration of an instance from the cluster.

Race condition in Peers interface design

Since peer.NewPeers() launches async go routines for both Redis and etcd, it is possible that a callback could occur before we call RegisterUpdatedPeersCallback() to register the call back, therefore missing the initial callback.

For example, in etcd_test.go we should be able to assert that p0 will receive 3 callbacks, one for itself, and then 2 for each additional member. However, because of the race condition, we might only get 2 callbacks.

We could change the Peers interface to instead register a callback during peer.NewPeers(conf, cb) call OR we could add a Start() method to the Peers interface, such that users would RegisterUpdatedPeersCallback() and then call Start() when ready to participate in receiving updates.

[MikeGoldsmith]

Thanks for opening @Baliedge. Let us know when you're ready for review.

I like the idea of expanding the peers interface to have a Start() func that would remove the race condition you've identified.

[github-actions]

Marking this PR as stale because it has been open 30 days with no activity. Please add a comment if this PR is still relevant; otherwise this PR will be automatically closed in 7 days.

[github-actions]

Closing this PR due to inactivity. Please see our Honeycomb OSS Lifecyle and Practices.