Reduce and breakout MR processing #16
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Update, turns out this fixes an issue I saw with missing data before. About every 55 minutes, APM would drop data for a single minute for each server. It only showed up on the aggregations that used the 1 minute time interval. I didn't really mind the dropped data since it wasn't significant, but the changes I made to reduce the CPU usage also happened to fix whatever was causing that issue.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The amount of processing needed for the mongo MR operations is currently way out of hand. I'm running a smallish app with two servers and something under 200 DAL right now, and the MR operations are chewing through ~$50/month worth of CPU processing on AWS after only a few days of stored APM data.
This drops the processing requirement significantly, from some early data down to 20% of what it was before. And I'm guessing this will not significantly affect the real-time data usage, after all how many times do you look at a 3 hour aggregation to understand what happened in the last 3 hours?