Add internal stat to record number of dropped series/measurements

[gunnaraasen]

Feature Request

Proposal: Add internal stats to track the number of series and measurements dropped.

Current behavior: Series and measurements are dropped silently when shard are dropped or DROP statements are executed.

Desired behavior: It would be nice to have num_series_dropped and num_measurements_dropped fields in the database measurement or whichever internal measurement makes the most sense.

Use case: There is currently no easy way to track the number of dropped series and measurements. Knowing the number of dropped series/measurements is useful for determining the rate series are being removed by a retention policy capacity planning. It is also hard to determine the number of series measurements removed when DROP statements are executed.

[e-dard]

Related to: #7691.

[jwilder]

The number of series dropped can be seen using a difference query over the current counters. If it's positive, it's the number series added. If it's negative, it's the number of series deleted.

[gunnaraasen]

Using the difference function works in some cases, but doesn't allow aggregation over a time period that is longer than the internal stats collection interval. For example, running

SELECT difference(max("numSeries")) FROM "influxdb_database" WHERE "host" = 'monitor' AND time > now() - 14d GROUP BY time(1d)

in a busy system with >1M series, the number of net dropped series becomes harder to discern. Sub-queries and the integral function would make this possible, but it feels like an explicit stat is easier to reason about. Tracking (and alerting on) series created/dropped is an integral part of debugging InfluxDB performance issues.

I was going to open some other PRs to add a numSeriesCreated and maybe even shard-level dropped stats, e.g. shard..seriesDropped (mirroring shard..seriesCreate). If this isn't the way to go, then we really need to finish up the _internal stats documentation and add examples.

Also, I'm not sure if these stats will be effectively moot when the TSI is merged.