fix: Prevent concurrent map writes to c.UnusedFields

[sspaink]

resolve: #11308

Telegraf is panicking reporting the error "fatal error: concurrent map writes" because maps in Go are not safe for concurrent use. The method missingTomlField was writing to the map c.UnusedFields concurrently so this pull request wraps the write with a mutex to allow the map to be updated safely. This panic is easily reproducible with a config defining multiple similar inputs that have the data_format set as "csv". I think this bug was a side effect from removing csv from the switch case to migrate it to the new parser. The function missingTomlField gets called "when the decoder encounters a key for which no matching struct field exists" , so I don't think this has anything to do with the parser restructure itself but just helped identify this issue because this could happen by just providing an invalid config.

I tested this by using the following configuration (Click to expand!)

[[inputs.file]]
  files =[ "test.csv" ]

  data_format = "csv"
  csv_header_row_count = 1
  csv_column_names = [ "usage_user_env","usage_user_server","usage_user_start_dt","usage_user_count" ]
  csv_tag_columns = [ "usage_user_env" ]
  csv_measurement_column = "usage_user_count"
  csv_timestamp_column = "usage_user_start_dt"
  csv_timestamp_ormat = "Monday, 02-01-06 15:04:05 MST"

[[inputs.file]]
    files =[ "test.csv" ]
    data_format = "csv"
    csv_header_row_count = 0
    csv_column_names = ["Backup_Type","Backup_Slug","Backup_State","Backup_Owner","Backup_Failed","Backup_Size","Backup_Duration","Backup_TimeStamp"]
    csv_tag_columns = [ "Backup_Type","Backup_Slug","Backup_State","Backup_Owner" ]
    csv_measurement_column = "Backup_Failed,Backup_Size,Backup_Duration"
     csv_timestamp_column = "Backup_TimeStamp"
    csv_timestamp_ormat = "Monday, 02-01-06 15:04:05 MST"

[[inputs.file]]
    files =[ "test.csv" ]
    data_format = "csv"
    csv_header_row_count = 1
    csv_column_names = [ "usage_user_env","usage_user_server","usage_user_start_dt","usage_user_count" ]
    csv_tag_columns = [ "usage_user_env" ]
    csv_measurement_column = "usage_user_count"
    csv_timestamp_column = "usage_user_start_dt"
    csv_timestamp_format = "Monday, 02-01-06 15:04:05 MST"

[[inputs.file]]
    files =[ "test.csv" ]
    data_format = "csv"
    csv_header_row_count = 1
    csv_column_names = [ "usage_process_env","usage_process_server","usage_process_start_dt","usage_process_count" ]
    csv_tag_columns = [ "usage_process_env" ]
    csv_measurement_column = "usage_process_count"
    csv_timestamp_column = "usage_process_start_dt"
    csv_timestamp_format = "Monday, 02-01-06 15:04:05 MST"

[[outputs.file]]
    files = ["stdout"]

[srebhan]

@sspaink while the fix looks good, I wonder how this is called concurrently at all. Can you outline the call-sequence? Furthermore, in the reported bug, the error happens after the config phase, at a point where addInput (the only user of missingTomlFields should not be active anymore...

[sspaink]

I was also confused why it was happening after the config phase, looking at it closer, the file input plugin (http and tail plugins seem to do it as well) both end up calling the method missingTomlField during their Gather methods and these Gather methods are run in goroutines.

This method is called because the file input plugin calls parser, err := f.parserFunc() during it's Gather loop https://github.com/influxdata/telegraf/blob/master/plugins/inputs/file/file.go#L98. And the parserFunc() calls c.addParser(name, table) when a parser is set (relevant code: https://github.com/influxdata/telegraf/blob/master/config/config.go#L1314-L1322). This call to addParser will lead to the calling of missingTomlField because that function is assigned to the Configs MissingField function here: https://github.com/influxdata/telegraf/blob/master/config/config.go#L1866-L1868. And this MissingField method of Config is called by the TOML package when decoding the config.

Snippet from stack-trace that shows example of this behavior:

github.com/influxdata/telegraf/config.(*Config).addParser(0xc000455320, {0xc0005f6370, 0x4}, 0xc0005c6d70)
	/go/src/github.com/influxdata/telegraf/config/config.go:1127 +0x216 fp=0xc0010f5d98 sp=0xc0010f5cf8 pc=0xae62b6
github.com/influxdata/telegraf/config.(*Config).addInput.func1()
	/go/src/github.com/influxdata/telegraf/config/config.go:1325 +0x29 fp=0xc0010f5dd0 sp=0xc0010f5d98 pc=0xae81e9
github.com/influxdata/telegraf/plugins/inputs/file.(*File).readMetric(0xc0000711a0, {0xc001374a40, 0x40})
	/go/src/github.com/influxdata/telegraf/plugins/inputs/file/file.go:125 +0x32f fp=0xc0010f5ec8 sp=0xc0010f5dd0 pc=0x16792ef
	github.com/influxdata/telegraf/plugins/inputs/file.(*File).Gather(0xc0000711a0, {0x5dbdc10, 0xc0004c8640})
/go/src/github.com/influxdata/telegraf/plugins/inputs/file/file.go:76 +0xaa fp=0xc0010f5f50 sp=0xc0010f5ec8 pc=0x1678b2a

To summarize, when the Gather method for File gets called, it ends up calling addParser so that the plugin can get the parser. Now that I see what's happening, I wonder if we should instead update this function https://github.com/influxdata/telegraf/blob/master/config/config.go#L1314-L1322 to instead of calling addParser inside SetParserFunc, it should call it before and then SetParserFunc will just return the *models.RunningParser. I guess then we can get rid of this mutex. @srebhan thoughts?

[sspaink]

@powersj and @srebhan as my comment before, I went ahead and posted the change I was suggesting instead of using a mutex because it seems to work locally. Now the config unmarhsalling logic for parsers isn't propagated to the plugins. Let me know what you think.

[srebhan]

No this is not the right fix. We need a copy of the parser as otherwise we run into trouble with stateful parsers. So keeping the original fix was better as we now get side-effects between the files all using the CSV parser. One of the side effects is that all column names get mixed for example, have fun with SkipRows, HeaderRowCount and friends...

[sspaink]

Argh darn stateful parsers getting me again, okay good catch I'll revert the fix back to using the mutex's then.

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz