cmd/geth: dump config for metrics #22083

MariusVanDerWijden · 2020-12-28T12:56:43Z

closes #22007

./geth --metrics --metrics.influxdb  dumpconfig

MariusVanDerWijden · 2020-12-30T12:38:34Z

@lightclient @MysticRyuujin could you take a look if that works for you?

lightclient · 2020-12-31T16:32:10Z

Was able to test this with the metrics configuration I am using and it works great. Thanks @MariusVanDerWijden!

holiman · 2021-01-05T13:32:13Z

@MariusVanDerWijden is this still WIP, or is it ready for merge? If the latter, please change the title

holiman · 2021-01-05T13:42:12Z

I don't get it. This does dump out the metrics config to toml ... but it doesn't read the metrics config from the toml... ?

holiman · 2021-01-05T13:45:22Z

I would expect these two commands to give the exact same output:

[user@work go-ethereum]$ ./build/bin/geth --metrics --metrics.influxdb --metrics.port=7070  --metrics.expensive dumpconfig 2>/dev/null | grep Metrics -A10
[Metrics]
Enabled = true
EnabledExpensive = true
HTTP = "127.0.0.1"
Port = 7070
EnableInfluxDB = true
InfluxDBEndpoint = "http://localhost:8086"
InfluxDBDatabase = "geth"
InfluxDBUsername = "test"
InfluxDBPassword = "test"
InfluxDBTags = "host=localhost"
[user@work go-ethereum]$ ./build/bin/geth --metrics --metrics.influxdb --metrics.port=7070  --metrics.expensive dumpconfig  2>/dev/null  > test.conf && ./build/bin/geth --config test.conf dumpconfig 2>/dev/null | grep Metrics -A10

MariusVanDerWijden · 2021-01-05T15:46:55Z

@holiman It did read the configs from toml, but it did overwrite them every time^^ It works now

holiman · 2021-01-06T13:02:44Z

Almost there now, but CLI should take precedence, so this should spit out 7171, not 7070:

./build/bin/geth --metrics --metrics.port=7070 dumpconfig  2>/dev/null  > test.conf && ./build/bin/geth --metrics.port=7171 --config test.conf dumpconfig 2>/dev/null | grep Metrics -A10 | grep Port
Port = 7070

MariusVanDerWijden · 2021-01-06T14:56:16Z

Hmm it takes precedence if you enable --metrics explicitly

./build/bin/geth --metrics --metrics.port=7070 dumpconfig  2>/dev/null  > test.conf && ./build/bin/geth --metrics --metrics.port=7171 --config test.conf dumpconfig 2>/dev/null | grep Metrics -A10 | grep Port
Port = 7171

But I'll change it

MariusVanDerWijden · 2021-01-06T16:23:53Z

So the code is quite a bit worse now but I think I need it to convey the whole logic.

If --metrics are enabled it should read the default values
If the metrics are read from file, they should be overwritten by the flags

holiman · 2021-01-07T10:31:36Z

I don't quite understand why you give special treatment to the enabled flags. If I set geth --x.foo.bar=baz --x.foo.enabled=true dumpconfig > conf, and later do geth --config conf --x.foo.enabled=false, IMO the config that is produced should have the feature disabled but the foo.bar field set to baz.

So geth --config conf.toml --metrics=false dumpconfig > conf.toml should only ever flip a boolean in the conf.toml, not erase the entire section, which I think the current implementation does... ?

Am I missing something?

MariusVanDerWijden · 2021-01-07T12:39:45Z

No you're right, I thought that we wouldn't want things like influxdbpassword written in the config if it was disabled, but you're right that it should be written out anyway. changed now.
Now the metric section is always dumped even if --metrics or --influxdb are disabled.

holiman

One minor comment, but generally looks ok. Will test it a bit

holiman · 2021-01-07T12:50:35Z

cmd/geth/config.go

@@ -115,12 +129,28 @@ func defaultNodeConfig() node.Config {
 	return cfg
 }

+func defaultMetricConfig(ctx *cli.Context) metricConfig {


Perhaps this should be metricsFromCliArgs.. When I saw defaultMetricConfig being used before, I intuitively though that it just initialized some defaults, and didn't grokk that it actually read the cli arguments

Yeah, that's the thing with this strange logic, we need to read the metrics twice.
First we need to read the "default" metrics and apply them s.th. if we don't override them in the load config, they are set to the default values.
Then, after reading the config, we need to read the metrics again (this time only the specially set metrics) to overwrite the config file configs with the configs from cli.

So basically we do:

read config from cli regardless of whether they are set (reads defaults)

read config from file (overwrites the default configs if applicable)

read only the set configs from cli (overwrites the file configs)

I'll change the naming!

holiman

Works for me!

fjl · 2021-01-13T10:35:12Z

cmd/geth/config.go

-		Node: defaultNodeConfig(),
+		Eth:     eth.DefaultConfig,
+		Node:    defaultNodeConfig(),
+		Metrics: metricsFromCliArgs(ctx),


I don't understand why you need this function (metricsFromCliArgs) if you also have applyMetricConfig. The idea behind the config file loading is:

The config is always created using the Go code defaults.
Then, the TOML file is loaded to override some things.
Finally, the flags are applied to override more things.

In this PR, for metrics, it takes the defaults from flags, then overrides with config file, then applies the flags again. I think we should remove metricsFromCliArgs and just define the defaults for metrics directly in the code.

Yeah, the problem with this approach is that we have now two different places where the default value of the flags is defined.
Once here and once in the cli.Flags.

The way we usually solve this is by taking the flag default from the Go struct default.

https://github.com/ethereum/go-ethereum/blob/master/cmd/utils/flags.go#L136

I created a new default var in the metrics package

* cmd/geth: dump config * cmd/geth: dump config * cmd/geth: properly read config again * cmd/geth: override metrics if flags are set * cmd/geth: write metrics regardless if enabled * cmd/geth: renamed to metricsfromcliargs * metrics: add default configuration

MariusVanDerWijden added 2 commits December 28, 2020 13:54

cmd/geth: dump config

c81a1ad

cmd/geth: dump config

43d5396

MariusVanDerWijden marked this pull request as ready for review December 31, 2020 22:32

cmd/geth: properly read config again

e2bc64c

MariusVanDerWijden changed the title ~~WIP cmd/geth: dump config~~ cmd/geth: dump config for metrics Jan 5, 2021

cmd/geth: override metrics if flags are set

c86fc9f

cmd/geth: write metrics regardless if enabled

e57fb55

holiman reviewed Jan 7, 2021

View reviewed changes

holiman approved these changes Jan 7, 2021

View reviewed changes

cmd/geth: renamed to metricsfromcliargs

8a68c74

DGKSK8LIFE approved these changes Jan 7, 2021

View reviewed changes

holiman added the status:triage label Jan 12, 2021

fjl reviewed Jan 13, 2021

View reviewed changes

metrics: add default configuration

c9abb41

holiman merged commit 10555d4 into ethereum:master Jan 18, 2021

holiman added this to the 1.10.0 milestone Jan 18, 2021

quorumbot mentioned this pull request Sep 3, 2021

[Upgrade] Go-Ethereum release v1.10.0 Consensys/quorum#1249

Merged

9 tasks

baptiste-b-pegasys mentioned this pull request Sep 3, 2021

[Upgrade] Go-Ethereum release v1.10.0 baptiste-b-pegasys/quorum#16

Closed

9 tasks

MariusVanDerWijden deleted the metrics_dump branch November 30, 2021 15:32

jsvisa mentioned this pull request Sep 13, 2023

cmd/geth: read metric from config file #28101

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/geth: dump config for metrics #22083

cmd/geth: dump config for metrics #22083

MariusVanDerWijden commented Dec 28, 2020 •

edited

Loading

MariusVanDerWijden commented Dec 30, 2020

lightclient commented Dec 31, 2020

holiman commented Jan 5, 2021

holiman commented Jan 5, 2021

holiman commented Jan 5, 2021

MariusVanDerWijden commented Jan 5, 2021

holiman commented Jan 6, 2021

MariusVanDerWijden commented Jan 6, 2021 •

edited

Loading

MariusVanDerWijden commented Jan 6, 2021

holiman commented Jan 7, 2021 •

edited

Loading

MariusVanDerWijden commented Jan 7, 2021

holiman left a comment

holiman Jan 7, 2021

MariusVanDerWijden Jan 7, 2021

holiman left a comment

fjl Jan 13, 2021 •

edited

Loading

MariusVanDerWijden Jan 13, 2021

fjl Jan 13, 2021

MariusVanDerWijden Jan 18, 2021

cmd/geth: dump config for metrics #22083

cmd/geth: dump config for metrics #22083

Conversation

MariusVanDerWijden commented Dec 28, 2020 • edited Loading

MariusVanDerWijden commented Dec 30, 2020

lightclient commented Dec 31, 2020

holiman commented Jan 5, 2021

holiman commented Jan 5, 2021

holiman commented Jan 5, 2021

MariusVanDerWijden commented Jan 5, 2021

holiman commented Jan 6, 2021

MariusVanDerWijden commented Jan 6, 2021 • edited Loading

MariusVanDerWijden commented Jan 6, 2021

holiman commented Jan 7, 2021 • edited Loading

MariusVanDerWijden commented Jan 7, 2021

holiman left a comment

Choose a reason for hiding this comment

holiman Jan 7, 2021

Choose a reason for hiding this comment

MariusVanDerWijden Jan 7, 2021

Choose a reason for hiding this comment

holiman left a comment

Choose a reason for hiding this comment

fjl Jan 13, 2021 • edited Loading

Choose a reason for hiding this comment

MariusVanDerWijden Jan 13, 2021

Choose a reason for hiding this comment

fjl Jan 13, 2021

Choose a reason for hiding this comment

MariusVanDerWijden Jan 18, 2021

Choose a reason for hiding this comment

MariusVanDerWijden commented Dec 28, 2020 •

edited

Loading

MariusVanDerWijden commented Jan 6, 2021 •

edited

Loading

holiman commented Jan 7, 2021 •

edited

Loading

fjl Jan 13, 2021 •

edited

Loading