Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(js/plugins/google-cloud): Make metric export time intervals mutually exclusive #1869

Merged
merged 3 commits into from
Feb 11, 2025

Conversation

schnecle
Copy link
Contributor

@schnecle schnecle commented Feb 5, 2025

The short of it:

Modify the startTime of the datapoints so that we ensure there are no overlapping export intervals.

Add a shutdown hook to our metrics wrapper ensures we are awaiting the callback function from the wrapped cloud exporter which delegates to an async function with a callback.

The long of it:

We were seeing occasions where exported metrics are being overwritten. This is because Cloud does not currently support the delta metric kind for custom metrics. Instead, it will convert delta metrics into "pseudo delta" metrics by calling them cumulative for the time interval provided. Where we run into trouble is that the PeriodicMetricExporterReader creates a time interval based on the last export, setting the start time of the subsequent export to the end time of the previous export. The overlap in time period means that the last in wins and is counted as the cumulative amount, resulting in what appear to be dropped metrics or sometimes, even more confusingly, metrics that increase one minute and decrease the next.

Manually tested locally using forceDevExport = true to make sure metrics are still being exported.

Checklist (if applicable):

@schnecle schnecle force-pushed the schnecle/metrics-shutdown-hooks branch from 801608d to b1ae857 Compare February 10, 2025 21:29
@schnecle schnecle changed the title fix(js/plugins/google-cloud): Implement forceFlush and shutdown fix(js/plugins/google-cloud): Make metric export time intervals mutually exclusive Feb 11, 2025
@schnecle schnecle merged commit 198e91e into main Feb 11, 2025
7 checks passed
@schnecle schnecle deleted the schnecle/metrics-shutdown-hooks branch February 11, 2025 17:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants