Support different service discovery systems

[yurishkuro]

Depends on go-kit push model for discovery: go-kit/kit#492

• upgrade to 0.5 version of go-kit (not released yet)
• decommission internal discovery API and replace it with go-kit
• add command line options for configuring different service discovery systems
• integration tests

[yurishkuro]

A few thoughts since this issue has been opened.

The direct integration with service discovery is quickly becoming an anti-pattern given the rise of service meshes and proxies that abstract away that infrastructural concern. Even at Uber internally our agents are communicating to collectors via a service mesh proxy

If direct integration is still desired for performance reasons, I think integration of numerous SD systems directly into the Jaeger binary is the wrong approach, as it bloats the binary size on one hand, and on the other it pushes us more into dependencies hell. I think the solution here is a plugins architecture (#422) that decouples the main binary from the dependencies of individual SD implementations. An interesting observation here is that unlike storage plugins, the SD plugins have very low QPS and therefore are much more suitable for the sidecar plugin model where the main binary communicates to the plugin over, say, gRPC streams. Of course, the sidecar plugin model is actually a special case of in-process plugin model, so we still need to start there.

[pavolloffay]

@yurishkuro do we want to still provide this functionality?

[yurishkuro]

I am not sure, there hasn't been much demand. My last comment was almost a year ago, since then I thought of yet another solution that requites no plugins. We can have an API endpoint where SD information can be pushed into by an external process, similar to data/control plane separation advocated by Envoy. This way the SD can be implemented as a sidecar to the Jaeger components.

But a harshicorp go-plugin would also work.

I don't see any of this work happening any time soon unless there are volunteers who actually need it in their environment.

[joshushua]

To be clear, no plug-in is available for Service Discovery? And no direct method of Service Discovery? So the recommended approach is to imitate Uber's use of a service mesh proxy?

[yurishkuro]

since 1.9 you can use gRPC to communicate between agent and collector, which is understood by many LB solutions.

[pavolloffay]

@yurishkuro can we close this with resolution "won't do"?

We also point to this issue from docs

[yurishkuro]

Closing per
#213 (comment)