High cpu usage when checking series cardinality

[dazoot]

We have been using influx for over a year. The number of series we have in DB is about 110 mil. We have one database: graphite, with a 1 year retention policy. The number of queries is not that high. We see consistent CPU usage even if there is no read query being ran on the server. We have upgraded to latest influxdb version and TSI indexes.

Server is quite big:
Dual Hexa Core
256 GB of Ram.

[jwilder]

Can you run the following when cpu usage is high and attach the output to the issue?

curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=true"

curl -o vars.txt "http://localhost:8086/debug/vars"
iostat -xd 1 30 > iostat.txt

[jwilder]

@dazoot It looks like the CPU usage is due to some the series cardinality monitoring checks that are expensive in your env. Your shards appear to be inmem and the monitoring calls are spending a lot of time sorting measurement and tag keys as well as allocating quite a bit of garbage. I have a potential fix that should reduce CPU usage for you a bit in #8702. If you are able to test that out and see if it helps, that would be great.

This is an issue specifically with inmem shards and won't occur with tsi shards.

[dazoot]

Is there a way to migrate inmem to tsi ?

[jwilder]

Switching the config setting will cause new shards to be created as tsi shards.

There is also a PR to add some offline tooling to convert existing shards to tsi: #8669.