MySQL Metrics Collision

[Esity]

When using one of the most recent versions of Telegraf(1.12.4) and using the MySQL input plugin against multiple MySQL based systems there is some sort of collision going on to cause write drops.

We purged all MySQL measurements in InfluxDB, waited for compactions to finish. We then re enabled the mysql input plugin and immediatly started seeing drops in InfluxDB. Here is the config we used

[[inputs.mysql]]
  servers = ["root:password@tcp(127.0.0.1:3306)/"]
  metric_version = 2
  perf_events_statements_digest_text_limit  = 120
  perf_events_statements_limit              = 250
  perf_events_statements_time_limit         = 86400
  table_schema_databases                    = []
  gather_table_schema                       = false
  gather_process_list                       = false
  gather_user_statistics                    = false
  gather_info_schema_auto_inc               = false
  gather_innodb_metrics                     = false
  gather_slave_status                       = false
  gather_binary_logs                        = false
  gather_table_io_waits                     = false
  gather_table_lock_waits                   = false
  gather_index_io_waits                     = false
  gather_event_waits                        = false
  gather_file_events_stats                  = false
  gather_perf_events_statements             = false
  interval_slow                             = "30m"

We run Percona, MySQL and MariaDB across many servers. What we noticed is that something is causing the collision and the dropped writes but I can't seem to figure out the exact metric.

All configs on all MySQL based servers are identical except the username/password

[danielnelson]

Could you add the log message during write?

[Esity]

We are currently sending between 20k and 25k PPS to each InfluxDB env. We have debug enabled for our writers to see what is going on so I will have to do some digging

[Esity]

@danielnelson got it

2019-11-06T17:09:43Z E! [outputs.influxdb]: when writing to [http://hostname]: received error partial write: field type conflict: input field "ssl_ctx_verify_depth" on measurement "mysql" is type float, already exists as type integer dropped=1; discarding points

[danielnelson]

This is #5529, I'll bump up the priority. Workaround is fielddrop:

[[inputs.mysql]]
  fielddrop = ["ssl_ctx_verify_depth"] # Workaround #5529

Closing this one and we can continue on the other issue.