Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Outputs.Stackdriver Start and End Timestamp Appear to be the Same #13725

Closed
crflanigan opened this issue Aug 4, 2023 · 13 comments · Fixed by #13745
Closed

Outputs.Stackdriver Start and End Timestamp Appear to be the Same #13725

crflanigan opened this issue Aug 4, 2023 · 13 comments · Fixed by #13745
Labels
bug unexpected problem or unintended behavior

Comments

@crflanigan
Copy link
Contributor

crflanigan commented Aug 4, 2023 •
edited
Loading

Relevant telegraf.conf

# Telegraf Configuration
#
# Telegraf is entirely plugin driven. All metrics are gathered from the
# declared inputs, and sent to the declared outputs.
#
# Plugins must be declared in here to be active.
# To deactivate a plugin, comment out the name and any variables.
#
# Use 'telegraf -config telegraf.conf -test' to see what metrics a config
# file would generate.
#
# Environment variables can be used anywhere in this config file, simply prepend
# them with $. For strings the variable must be within quotes (ie, "$STR_VAR"),
# for numbers and booleans they should be plain (ie, $INT_VAR, $BOOL_VAR)
# Global tags can be specified here in key="value" format.
[global_tags]
env = "Production_MacOS"
instance = 'computername'
# rack = "1a"
# Environment variables can be used as tags, and throughout the config file
# user = "$USER"
# Configuration for telegraf agent
[agent]
  ## Default data collection interval for all inputs
  interval = "1m"
  ## Rounds collection interval to 'interval'
  ## ie, if interval="10s" then always collect on :00, :10, :20, etc.
  round_interval = true
  ## Telegraf will send metrics to outputs in batches of at most
  ## metric_batch_size metrics.
  ## This controls the size of writes that Telegraf sends to output plugins.
  metric_batch_size = 199
  ## For failed writes, telegraf will cache metric_buffer_limit metrics for each
  ## output, and will flush this buffer on a successful write. Oldest metrics
  ## are dropped first when this buffer fills.
  ## This buffer only fills when writes fail to output plugin(s).
  metric_buffer_limit = 10000
  ## Collection jitter is used to jitter the collection by a random amount.
  ## Each plugin will sleep for a random time within jitter before collecting.
  ## This can be used to avoid many plugins querying things like sysfs at the
  ## same time, which can have a measurable effect on the system.
  collection_jitter = "0s"
  ## Default flushing interval for all outputs. Maximum flush_interval will be
  ## flush_interval + flush_jitter
  flush_interval = "1m"
  ## Jitter the flush interval by a random amount. This is primarily to avoid
  ## large write spikes for users running a large number of telegraf instances.
  ## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
  flush_jitter = "0s"
  ## By default or when set to "0s", precision will be set to the same
  ## timestamp order as the collection interval, with the maximum being 1s.
  ##   ie, when interval = "10s", precision will be "1s"
  ##       when interval = "250ms", precision will be "1ms"
  ## Precision will NOT be used for service inputs. It is up to each individual
  ## service input to set the timestamp at the appropriate precision.
  ## Valid time units are "ns", "us" (or "µs"), "ms", "s".
  precision = ""
  ## Logging configuration:
  ## Run telegraf with debug log messages.
  debug = true
  ## Run telegraf in quiet mode (error log messages only).
  quiet = false
  ## Specify the log file name. The empty string means to log to stderr.
  logfile = ""
  ## The logfile will be rotated after the time interval specified.  When set
  ## to 0 no time based rotation is performed.
  logfile_rotation_interval = "24h"
  ## The logfile will be rotated when it becomes larger than the specified
  ## size.  When set to 0 no size based rotation is performed.
  logfile_rotation_max_size = "50MB"
  ## Maximum number of rotated archives to keep, any older logs are deleted.
  ## If set to -1, no archives are removed.
  logfile_rotation_max_archives = 1
  ## Override default hostname, if empty use os.Hostname()
  hostname = ""
  ## If set to true, do no set the "host" tag in the telegraf agent.
  omit_hostname = false


[[inputs.internal]]

[[outputs.stackdriver]]
  project = "GOOGLE PROJECT"
  resource_type = "prometheus_target"
  #namespace = "casey" 
  metric_name_format = "official"
  metric_data_type = "double"
  metric_type_prefix  = "prometheus.googleapis.com" 
  tags_as_resource_label = ["instance"]
  [outputs.stackdriver.resource_labels]
    cluster = "onprem"
    job = "CaseyTest"
    location = "us-east1"
    namespace = "metrics"

Logs from Telegraf

2023-08-04T20:30:07Z E! [outputs.stackdriver] Unable to write to Stackdriver: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[106].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_agent_metrics_dropped/unknown:counter'.; Field timeSeries[39].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_gather_errors/unknown:counter'.; Field timeSeries[74].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_memstats_total_alloc_bytes/unknown:counter'.; Field timeSeries[119].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_memstats_heap_sys_bytes/unknown:counter'.; Field timeSeries[47].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_metrics_filtered/unknown:counter'.; Field timeSeries[91].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_metrics_written/unknown:counter'.; Field timeSeries[142].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_serializer_bytes_serialized/unknown:counter'.; Field timeSeries[144].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_metrics_dropped/unknown:counter'.; Field timeSeries[50].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_gather_metrics_gathered/unknown:counter'.; Field timeSeries[5].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_memstats_sys_bytes/unknown:counter'.; Field timeSeries[129].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_gather_gather_time_ns/unknown:counter'.; Field timeSeries[40].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_write_time_ns/unknown:counter'.; Field timeSeries[120].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_metrics_dropped/unknown:counter'.; Field timeSeries[4].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_memstats_alloc_bytes/unknown:counter'.; Field timeSeries[97].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_memstats_heap_alloc_bytes/unknown:counter'.; Field timeSeries[70].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_memstats_heap_in_use_bytes/unknown:counter'.; Field timeSeries[114].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_gather_errors/unknown:counter'.; Field timeSeries[138].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_metrics_added/unknown:counter'.; Field timeSeries[51].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_errors/unknown:counter'.; Field timeSeries[42].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_agent_gather_errors/unknown:counter'.; Field timeSeries[15].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_metrics_written/unknown:counter'.; Field timeSeries[6].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_errors/unknown:counter'.; Field timeSeries[9].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_buffer_limit/unknown:counter'.; Field timeSeries[62].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_gather_errors/unknown:counter'.
error details: name = Unknown  desc = total_point_count:149  success_point_count:125  errors:{status:{code:3}  point_count:24}
2023-08-04T20:30:07Z D! [outputs.stackdriver] Buffer fullness: 38 / 10000 metrics
2023-08-04T20:30:07Z E! [agent] Error writing to outputs.stackdriver: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[106].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_agent_metrics_dropped/unknown:counter'.; Field timeSeries[39].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_gather_errors/unknown:counter'.; Field timeSeries[74].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_memstats_total_alloc_bytes/unknown:counter'.; Field timeSeries[119].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_memstats_heap_sys_bytes/unknown:counter'.; Field timeSeries[47].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_metrics_filtered/unknown:counter'.; Field timeSeries[91].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_metrics_written/unknown:counter'.; Field timeSeries[142].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_serializer_bytes_serialized/unknown:counter'.; Field timeSeries[144].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_metrics_dropped/unknown:counter'.; Field timeSeries[50].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_gather_metrics_gathered/unknown:counter'.; Field timeSeries[5].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_memstats_sys_bytes/unknown:counter'.; Field timeSeries[129].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_gather_gather_time_ns/unknown:counter'.; Field timeSeries[40].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_write_time_ns/unknown:counter'.; Field timeSeries[120].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_metrics_dropped/unknown:counter'.; Field timeSeries[4].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_memstats_alloc_bytes/unknown:counter'.; Field timeSeries[97].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_memstats_heap_alloc_bytes/unknown:counter'.; Field timeSeries[70].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_memstats_heap_in_use_bytes/unknown:counter'.; Field timeSeries[114].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_gather_errors/unknown:counter'.; Field timeSeries[138].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_metrics_added/unknown:counter'.; Field timeSeries[51].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_errors/unknown:counter'.; Field timeSeries[42].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_agent_gather_errors/unknown:counter'.; Field timeSeries[15].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_metrics_written/unknown:counter'.; Field timeSeries[6].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_errors/unknown:counter'.; Field timeSeries[9].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_buffer_limit/unknown:counter'.; Field timeSeries[62].points[0].interval.start_time had an invalid value of "2023-08-04T13:29:00-07:00": The start time must be before the end time (2023-08-04T13:29:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_gather_errors/unknown:counter'.



### System info

1.27.3

### Docker

_No response_

### Steps to reproduce

1. Use inputs.internal 
2. Start Telegraf
3. You either get errors from stackdriver if you configure using `[[inputs.internal]]`.
...


### Expected behavior

Inputs.internal works like expected. 

### Actual behavior

You get many errors from the Google/Stackdriver API with metrics written to file. 

### Additional info

It looks like this is a metric of type counter which has a start and end time of the same timestamp which Google is rejecting when in basic mode.
@crflanigan crflanigan added the bug unexpected problem or unintended behavior label Aug 4, 2023
@crflanigan
Copy link
Contributor Author

@powersj I think the problem I was having with fieldpass was a config problem on my part so I have updated the issue to remove it. Hopefully I didn't muddy the waters, however the inputs.internal counter issue is still an issue.

Have a wonderful weekend!

@crflanigan crflanigan changed the title Inputs.Internal Odd Behavior Outputs.Stackdriver Start and End Timestamp Appear to be the Same Aug 9, 2023
@crflanigan
Copy link
Contributor Author

@powersj
I'm thinking this problem isn't tied to just the inputs.internal plugin, but to any counter plugin.

I see this bit of code here:

func NewCounterCacheEntry(value *monpb.TypedValue, ts *tspb.Timestamp) *counterCacheEntry {
	// Start times must be _before_ the end time, so backdate our original start time
	// to 1ms before the observed time.
	backDatedStart := ts.AsTime().Add(time.Millisecond * -1)
	return &counterCacheEntry{LastValue: value, StartTime: tspb.New(backDatedStart)}

This is a great idea, but I'm wondering if the timestamp generated maintains that timestamp precision as when I look at the returned error from the GCM API I see this:

Field timeSeries[40].points[0].interval.start_time had an invalid value of "2023-08-07T11:12:00-07:00": The start time must be before the end time (2023-08-07T11:12:00-07:00) for the non-gauge metric 'prometheus.googleapis.com/internal_write_buffer_size/unknown:counter'.

I wonder if we increased the timestamp precision to millisecond if that would help?
Or perhaps decrementing the start time to be a higher value, like 1000 ms?

Thoughts?

powersj added a commit to powersj/telegraf that referenced this issue Aug 9, 2023
@powersj
fix(outputs.stackdriver): regenerate time interval for unkown metrics
75d9c44 
fixes: influxdata#13725
@powersj powersj mentioned this issue Aug 9, 2023
Merged
@powersj
Copy link
Contributor

powersj commented Aug 9, 2023

@crflanigan,

Looking at the error messages, all the errors are with the metric unknown:counter. If you recall this was the second metric send that we make for untyped metrics that Google had us add as a part of #13454. I wondered why we had not seen this before and it makes sense why it is only in the new code.

The time handling for counters is different for gauges and this is a different metric type, so our counter cache needs to store it a second time. Rather than taking the existing data point, we need to recalculate the time interval.

I have put up #13745 for you to try out!

Thanks!

@powersj powersj added the waiting for response waiting for response from contributor label Aug 9, 2023
@crflanigan
Copy link
Contributor Author

Awesome, will do!

Thanks for the help @powersj !!

@telegraf-tiger telegraf-tiger bot removed the waiting for response waiting for response from contributor label Aug 10, 2023
@crflanigan
Copy link
Contributor Author

@powersj

Dang buddy, you're a wizard!

That worked like a charm :)

@crflanigan
Copy link
Contributor Author

crflanigan commented Aug 10, 2023 •
edited
Loading

One thing though, the metrics have a type of unknown, should they be typed?:

image image

Is that expected for inputs.internal metrics?

@crflanigan
Copy link
Contributor Author

Also,
Seeing a lot of these log messages:

023-08-10T14:22:34Z D! [outputs.stackdriver] received 6 metrics
2023-08-10T14:22:34Z D! [outputs.stackdriver] split into 1 groups by timestamp
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:22:34Z D! [outputs.stackdriver] Wrote batch of 6 metrics in 285.349584ms
2023-08-10T14:22:34Z D! [outputs.stackdriver] Buffer fullness: 0 / 10000 metrics
2023-08-10T14:23:34Z D! [outputs.stackdriver] received 6 metrics
2023-08-10T14:23:34Z D! [outputs.stackdriver] split into 1 groups by timestamp
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.file] Wrote batch of 6 metrics in 1.22525ms
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 61
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] age: 121
2023-08-10T14:23:34Z D! [outputs.stackdriver] Wrote batch of 6 metrics in 280.804291ms
2023-08-10T14:23:34Z D! [outputs.stackdriver] Buffer fullness: 0 / 10000 metrics
2023-08-10T14:24:34Z D! [outputs.stackdriver] received 6 metrics
2023-08-10T14:24:34Z D! [outputs.stackdriver] split into 1 groups by timestamp
2023-08-10T14:24:34Z D! [outputs.stackdriver] age: 181
2023-08-10T14:24:34Z D! [outputs.stackdriver] age: 181
2023-08-10T14:24:34Z D! [outputs.stackdriver] age: 181
2023-08-10T14:24:34Z D! [outputs.stackdriver] age: 181

@powersj
Copy link
Contributor

powersj commented Aug 10, 2023

Dang buddy, you're a wizard!

:) Thanks for verifying

One thing though, the metrics have a type of unknown, should they be typed?:

Yes, for metrics that are "untyped" they will are sent twice, once with "unknown" and another with "unknown:counter".

Seeing a lot of these log messages:

The counter cache has a debug statement for age, I don't think this is helpful and will remove it.

@crflanigan
Copy link
Contributor Author

:) Thanks for verifying

You bet, thanks for the help!

Yes, for metrics that are "untyped" they will are sent twice, once with "unknown" and another with "unknown:counter".

The internal metrics are sent twice, once for unknown and once unknown:counter? Is this just the nature of the beast, or can it be just counter sent once?

The counter cache has a debug statement for age, I don't think this is helpful and will remove it.

Got it.

@powersj
Copy link
Contributor

powersj commented Aug 10, 2023

The internal metrics are sent twice, once for unknown and once unknown:counter? Is this just the nature of the beast, or can it be just counter sent once?

It is not just internal metrics, but untyped metrics. That was at the request of Google review here: #13454 (comment)

@crflanigan
Copy link
Contributor Author

It is not just internal metrics, but untyped metrics. That was at the request of Google review here: #13454 (comment)

Got it, the metrics look fine outside of Google. Just curious, is there anyway for us to know if a metric is untyped by looking at the input plugins documentation? Or is it basically we would just find out when we send them?

@powersj
Copy link
Contributor

powersj commented Aug 10, 2023

There is not a clear way as far as I can tell without looking at the code. What I generally say is that basically every metric produced by telegraf will be untyped, except metrics that are from prometheus, where the types are set. I'm sure there are exceptions to that large generalization.

@crflanigan
Copy link
Contributor Author

Ok, that's cool.
Just tested out the change you made to the fix and it looks good to me.

Thanks again for all the help buddy!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug unexpected problem or unintended behavior
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(outputs.stackdriver): regenerate time interval for unkown metrics powersj/telegraf
2 participants
@powersj @crflanigan