Expose the tuning options via expvar

[jpkrohling]

Requirement - what kind of business use case are you trying to solve?

It's not possible to check what's the current value for different tuning options, such as:

--collector.num-workers
--collector.queue-size
--discovery.conn-check-timeout
--discovery.min-peers
--memory.max-traces
--processor.*.server-max-packet-size
--processor.*.server-queue-size
--processor.*.workers
--reporter.tchannel.discovery.conn-check-timeout
--reporter.tchannel.discovery.min-peers
--reporter.tchannel.report-timeout

Proposal - what do you suggest to solve the problem or improve the existing situation?

During the startup process, store values in expvar variables and make sure /debug/expvar is always mounted on the admin port (right now it is only mounted if --metrics-backend=expvar is specified).

[yurishkuro]

Good idea. the only issue I foresee is that some metrics systems (in particular, Uber's M3) flush all metrics periodically, which results in the single data point in the time series db, thus if you're looking at the last hour of metrics and that point was earlier, you won't get any data.

Perhaps instead of metrics (which are meant for time series, even if gauges), these values are better exposed via expvar.

[jpkrohling]

That would work too.

[annanay25]

@Abhi-khandelwal - would you like to give this a shot?

expvar is an interface which helps to instrument and expose metrics from a golang application, you can read more about it here - https://sysdig.com/blog/golang-expvar-custom-metrics/

[Abhi-khandelwal]

@annanay25, yeah sure.

[chandresh-pancholi]

I am picking it up.

[chandresh-pancholi]

@jpkrohling could you please give me a starting point to work on it?

[jpkrohling]

@chandresh-pancholi take a look at how the components are bootstrapped, like the collector. Look for the bootstrap of the Admin HTTP server, as that's where the metrics should be (including the expvars). As @yurishkuro mentioned, you probably want to use expvars directly, but I think we might have a few places where they are then exposed via /metrics as well.

Once you know where the code should live, you probably just need to get the relevant Options objects to the place that will expose them.

Let me know if you get stuck!

[dstdfx]

@chandresh-pancholi @yurishkuro is this one "in-progress" or I could pick it up?

[yurishkuro]

I don't think anyone's working on it, it's been idle since Feb

[dstdfx]

Alright, then I'll work on this

[dstdfx]

Looks like some of the options are deprecated or removed:

--discovery.conn-check-timeout
--discovery.min-peers

became deprecated in #1092

--reporter.tchannel.discovery.conn-check-timeout
--reporter.tchannel.discovery.min-peers
--reporter.tchannel.report-timeout

removed in #2115

[yurishkuro]

It is best not to rely on the exact list of parameters in this ticket, but look at the up-to-date list at https://www.jaegertracing.io/docs/1.19/cli/.

Also, it's ok not to cover every single parameter, as long as we establish a clean framework & nomenclature for adding them.