Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sanitize NAN values from timeseries data #353

Merged
merged 9 commits into from
May 30, 2024

Conversation

aestoltm
Copy link
Contributor

@aestoltm aestoltm commented Sep 25, 2023

Description

Add check to sanitize job timeseries data of NAN values due to Symfony throwing double NAN error when json encoding. Added helper function which replaces any NAN values in the series data with null.

Motivation and Context

Charts with NAN data would return a null record.

Tests performed

Observed changes on xdmod-dev. Job that has a NaN value(s) can be seen here .

After checking mongodb-access, it was confirmed the value is NaN inside Mongodb for the job linked above. There was an error for the cpuuser value that was NaN stating that it couldn't collect enough data for the specified time period.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project as found in the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

Copy link
Contributor

@aaronweeden aaronweeden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please explain an example of how to reproduce the problem in the "Motivation and Context" section and explain in more detail how you tested it on xdmod-dev in the "Tests performed" section?

Without being sure how to reproduce the error, I'm not sure yet if changing NaN to zero is the right solution.

classes/DataWarehouse/Query/SUPREMM/JobMetadata.php Outdated Show resolved Hide resolved
classes/DataWarehouse/Query/SUPREMM/JobMetadata.php Outdated Show resolved Hide resolved
classes/DataWarehouse/Query/SUPREMM/JobMetadata.php Outdated Show resolved Hide resolved
@jpwhite4
Copy link
Member

jpwhite4 commented Jan 3, 2024

Having thought about this - I think that the NaN values should be removed from the data series rather than set to 0 - since 0 could be a valid value from a valid measurement and NaN typically indicates that there was not a valid measurement.

@aestoltm aestoltm added this to the 11.0.0 milestone May 2, 2024
@aaronweeden
Copy link
Contributor

Looks like the PR description should be updated to say it replaces them with null instead of 0.

@aestoltm
Copy link
Contributor Author

aestoltm commented May 29, 2024

Looks like the PR description should be updated to say it replaces them with null instead of 0.

Updated the description that the value is replaced with null

@aestoltm aestoltm merged commit 587a9a5 into ubccr:xdmod11.0 May 30, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants