Improve Performance for monthly active users API endpoint #234
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
johnbaldwin
changed the title
Codecov Report
@@ Coverage Diff @@
## master #234 +/- ##
==========================================
+ Coverage 91.30% 91.36% +0.05%
==========================================
Files 39 40 +1
Lines 1956 1968 +12
==========================================
+ Hits 1786 1798 +12
Misses 170 170
Continue to review full report at Codecov.
|
johnbaldwin
force-pushed
the
john/update-sdm-active-users
branch
from
4eee541 to
fdb04b4
Compare
johnbaldwin
commented
Jul 12, 2020
figures/metrics.py
Outdated
| """ | ||
| month_for = datetime.datetime.utcnow() | ||
| site_sm = figures.sites.get_student_modules_for_site(site) | ||
| curr_sm = site_sm.filter(modified__year=month_for.year, | ||
| modified__month=month_for.month) | ||
| return curr_sm.values('student__id').distinct().count() | ||
| return curr_sm.values('student__id').distinct() |
There was a problem hiding this comment.
We do this so we're able to use this function to retrieve the distinct ids, then we can get the count from the querset for when we need the count
johnbaldwin
force-pushed
the
john/update-sdm-active-users
branch
from
fdb04b4 to
21d88df
Compare
This comment has been minimized.
This comment has been minimized.
|
@OmarIthawi I didn't need to do the 3rd commit, which was going to update the viewset. Switching logic at I'll revise the git commit message |
OmarIthawi
approved these changes
Jul 14, 2020
|
Thanks Omar! |
The purpose of this is to improve API performance. Figures currently performs a live query on the `edx-platform` `courseware.models.StudentModule` model, but the query is too slow when there are many `StudentModule` records. The rough order of magnitude (ROM) for "many" appears to be about 10^6. The slow query causes an API timeout, degrading performance for Figures UI. This commit provides the storage for this metric so the following commit can update the pipeline code and metrics code
The purpose of this is to improve API performance. See the previous commit with SHA 92ea8c4 for an explanation. This commit adds the pipeline code to populate 'SiteDailyMetrics.mau' with the latest numbers for the month as of yesterday We also clean up the old live MAU collection from StudentModule, removing dead coe and the tests that call the dead code
johnbaldwin
force-pushed
the
john/update-sdm-active-users
branch
from
21d88df to
887f28e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is to improve the performance of Figures API to improve Figures UI page loading and prevent timeouts on retrieving monthly active users, which gets slow as we're currently pulling live data for the current month
Figures currently performs a live query on the
edx-platformcourseware.models.StudentModulemodel, but the query is too slow when there are manyStudentModulerecords. The rough order of magnitude (ROM) for "many" appears to be about 10^6.There are two commits.
The data model update adds 'mau' field to 'SiteDailyMetrics' model
The pipeline code now populates 'SiteDailyMetrics.mau' field with the latest numbers for the month as of when the pipeline is run.
The slow query causes an API timeout, degrading performance for Figures UI.
This commit provides the storage for this metric so the following commit can update the pipeline code and the 3rd commit to update the API code
Removed the old MAU calculation code which is now dead code and removed the tests