Kibana APM configuration should read from config/kibana.dev.yml

[trentm]

Kibana version: git master, commit 4f3d72b

Elasticsearch version: 8.0.0 via ./scripts/compose.py start master --no-kibana in apm-integration-test.git

Server OS version: macos 10.15.7

Browser version: Firefox 83.0

Browser OS version: macos 10.15.7

Original install method (e.g. download page, yum, from source, etc.): from git

Describe the bug:

It appears from my brief playing with @TinaHeiligers that a recent kibana.git commit to the master branch broke reading if "config/kibana.dev.yml". We were playing with turning on APM in a dev build of Kibana, and after updating today to the latest master changes we observed:

% cat config/kibana.dev.yml
elastic.apm.active: true
elastic.apm.serverUrl: 'http://127.0.0.1:8200'
elastic.apm.centralConfig: false
elastic.apm.breakdownMetrics: false
elastic.apm.transactionSampleRate: 1
elastic.apm.metricsInterval: '120s'

elasticsearch.username: "admin"
elasticsearch.password: "changeme"

% ELASTIC_APM_LOG_LEVEL=trace yarn start
yarn run v1.22.10
$ node scripts/kibana --dev
Elastic APM agent disabled (`active` is false)
...

I can get APM active via its environment variable configuration:

% ELASTIC_APM_ACTIVE=true ELASTIC_APM_LOG_LEVEL=trace yarn start
yarn run v1.22.10
$ node scripts/kibana --dev
agent configured correctly {
...

and via "config/kibana.yml":

% git diff
diff --git a/config/kibana.yml b/config/kibana.yml
index ce9fe28dae9..5d827704baf 100644
--- a/config/kibana.yml
+++ b/config/kibana.yml
@@ -1,3 +1,5 @@
+elastic.apm.active: true
+
 # Kibana is served by a back end server. This setting specifies the port to use.
 #server.port: 5601

% ELASTIC_APM_LOG_LEVEL=trace yarn start
yarn run v1.22.10
$ node scripts/kibana --dev
agent configured correctly {
...

My guess is this is due to commit b593781 (Jest multi-project configuration) from #77894 because that is the only new commit that has modified "packages/kbn-apm-config-loader/..."

[trentm]

/cc @tylersmalley in case I'm right about #77894 causing the breakage.

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[TinaHeiligers]

cc @elastic/kibana-core

[tylersmalley]

@trentm, I don't believe the APM configs have ever been picked up in the kibana.dev.yml. Is it possible you were placing it in the kibana.yml before?

I was recently discussing this with @watson, and we agree that it should be updated to also read from the kibana.dev.yml to avoid confusion. It currently supports providing a config/apm.dev.js which we plan on keeping for now as it allows for additional control.

[tylersmalley]

For reference, here is where the @kbn/apm-config-loader is reading the dev config: https://github.com/elastic/kibana/blob/master/packages/kbn-apm-config-loader/src/config.ts#L122-L129

[trentm]

@tylersmalley You are right. Sorry about that! I only learned about "kibana.dev.yml" yesterday when speaking with @watson -- along with learning about "config/apm.dev.js" at the same time. Before that I had been using a local diff to "config/kibana.yml". I must have misinterpreted Thomas and assumed I could use "kibana.dev.yml" for "elastic.apm.*" config vars.

I hadn't realized that "kibana.dev.yml" is only picked up by some code paths.

[tylersmalley]

No worries, I have updated this issue to reflect that we should support the elastic.apm settings in the kibana.dev.yml as well.

[pgayvallet]

It appears from my brief playing with @TinaHeiligers that a recent kibana.git commit to the master branch broke reading if "config/kibana.dev.yml"

1. Is this file still a thing? See [APM Agent] Replace support for apm.dev.js with support for kibana.dev.yml in apm-config-loader #79490 (comment)

2. Are you sure this has been working ever? The apm-config-loader never explicitly looked at that file. It's supposed to use the same configuration files that the server has been asked to use.

kibana/packages/kbn-apm-config-loader/src/utils/get_config_file_paths.ts

Lines 31 to 37 in aa9774f

	export const getConfigurationFilePaths = (argv: string[]): string[] => {
	const rawPaths = getArgValues(argv, ['-c', '--config']);
	if (rawPaths.length) {
	return rawPaths.map((path) => resolve(process.cwd(), path));
	}
	return [getConfigPath()];
	};

and getConfigPath only returns the kibana.yml file. Or is the CLI starting the kibana process with --config kibana.yml --config kibana.dev.yml (@tylersmalley)?

[trentm]

Are you sure this has been working ever?

No, I was wrong when I filed this ticket. "kibana.dev.yml" allows one to specify some config vars for parts of Kibana I believe (at least there are doc references to using it), but it does not (and did not) result in the kbm-apm-config-loader picking up elasitc.apm.* vars.

[tylersmalley]

Realized there was already an issue open for this: #79490

Closing in favor of that.