Filter plugin installs data

[klai95]

For the /activity/<plugin> endpoint, we should filter installs data to the past n months, where n can be specified by the frontend, but otherwise defaults to 12. Additionally, the current month's data is not returned, so the data would always end at the previous month (e.g. in October 2022, we would return available data for October 2021 - September 2022).

Additional notes

Currently on staging, the data is bunched together as the frontend is not set up to handle more than 12 data points:

[codemonkey800]

specifically, it should be the last 12 months. if the plugin is less than 12 months old, it should still show all 12 data points but with a null y value for the months where it didn't exist. for example:

napari-hub/frontend/src/fixtures/activity.json

Lines 3 to 52 in ac88e26

	"points": [
	{
	"x": 1627801200000,
	"y": null
	},
	{
	"x": 1630479600000,
	"y": null
	},
	{
	"x": 1633071600000,
	"y": null
	},
	{
	"x": 1635750000000,
	"y": null
	},
	{
	"x": 1638345600000,
	"y": null
	},
	{
	"x": 1641024000000,
	"y": null
	},
	{
	"x": 1643702400000,
	"y": 6
	},
	{
	"x": 1646121600000,
	"y": 2
	},
	{
	"x": 1648796400000,
	"y": 1
	},
	{
	"x": 1651388400000,
	"y": 3
	},
	{
	"x": 1654066800000,
	"y": 2
	},
	{
	"x": 1656658800000,
	"y": 1
	}
	],

[klai95]

@manasaV3 @codemonkey800 and me have just discussed the following points and we hope to get feedback from @richaagarwal about them:

Problem: Currently, the activity dashboard implementation plan for displaying plugin installs chart is to include the data points from the past 12 months. However, the issue with this approach is that it is not flexible. For example, a user may only want to look at all data points in 2020, but based on the current method the user would not be able to do so.

Solution: To solve this problem, we can build APIs that are more extensible and flexible by introducing additional parameters in the URL, namely, the start date and end date.

Note: If there is a need to avoid malicious attacks, we cab set a limit as a param to avoid users from hitting our endpoints too many times, or even just limiting the time period between start and end months to no more than 12 months. I can see how this work could help drive frontend-to-backend shared data schema efforts as well.

[richaagarwal]

@klai95 Thanks for noting this, Kevin! I've updated the issue's description above. Let me know if that captures this. In the future, we can certainly look into accommodating specific start and end dates as well.

I also removed the bug label as this work is not yet released to production :)

[klai95]

@klai95 Thanks for noting this, Kevin! I've updated the issue's description above. Let me know if that captures this. In the future, we can certainly look into accommodating specific start and end dates as well.

I also removed the bug label as this work is not yet released to production :)

Just two clarifying questions:

(1) Does this mean we are set on not displaying the current month's data? I know during the meeting some folks were talking about prediction models, so just want to get some clarity!

(2) When you say we should filter installs data to the past n months, is there an upper bound for what n could be? For example, if someone requests for a value of 100 for n, do we have a set limit (say arbitrarily 36 just for example) that cannot be exceeded to limit the amount of data that needs to be returned?

@richaagarwal

[richaagarwal]

@klai95 For our initial launch (what we've been referring to internally as P0 work), we won't be displaying the current month's data. In follow-up work, there will likely be changes to what & how we display the data.

We haven't yet discussed this particular ticket in a team meeting so it's possible there will be some more changes - nothing is set in stone at the moment, but we'll finalize the criteria for this ticket once we bring it into a sprint. At this time, I don't see too much value in adding an upper limit, but we can discuss more in refinement.

[richaagarwal]

Just a heads up that we may need to clarify whether to distinguish plugins that were released in the current month (in which case we'd check the plugin's release date and return data accordingly) vs. older plugins that happen to have zero installs - depending on the outcome of that, it may impact the work in this ticket (we could open up a separate issue, but just wanted to flag here). Let's clarify in sprint planning today. cc @klai95 @neuromusic @codemonkey800 @liaprins-czi

[richaagarwal]

@klai95 @codemonkey800 To summarize what was discussed in sprint planning today - we will move forward with what's captured in Lia's designs and show a special message for when a plugin is new (e.g. released in the current month), and otherwise show data as defined in this ticket. So I will let you two work out what the back-end should return so that the front-end knows it is a new plugin and to display the "There is not yet enough data..." message per the design.

[klai95]

@richaagarwal On the frontend @codemonkey800 uses null to show gaps, and so on the backend the months that have no installs will take the values of None. It is also worth noting that, by doing so, we can differentiate months that actually have no installs v.s. months that have gaps.

[richaagarwal]

@klai95 @codemonkey800 Quick clarification - we actually never see zeros/no install data in Ben's table, because there's only a row for an item if it was an install. So unfortunately we don't have any way to meaningfully distinguish between "gaps" and zero installs. Based on discussions we've had so far, it seems we are comfortable inferring these as zero installs, so we should probably show this as zeroes in the UI as well.

[klai95]

@klai95 @codemonkey800 Quick clarification - we actually never see zeros/no install data in Ben's table, because there's only a row for an item if it was an install. So unfortunately we don't have any way to meaningfully distinguish between "gaps" and zero installs. Based on discussions we've had so far, it seems we are comfortable inferring these as zero installs, so we should probably show this as zeroes in the UI as well.

@codemonkey800 Just some clarification, we would still want to return None instead of 0 in this case right?

[codemonkey800]

yeah it would be ideal to return None if there is a gap. I think the main use case is for when a plugin is less than a year old:

in this case, there would be points for October - January, but the y values would be None.

another option we could do is have the backend only return the amount of points it has. so if there's only 6 data points, and we request 12, then just send 6. on the frontend we can fill in the remaining data points for the rest of the months and use y: null for the gaps

[klai95]

yeah it would be ideal to return None if there is a gap. I think the main use case is for when a plugin is less than a year old:

in this case, there would be points for October - January, but the y values would be None.

another option we could do is have the backend only return the amount of points it has. so if there's only 6 data points, and we request 12, then just send 6. on the frontend we can fill in the remaining data points for the rest of the months and use y: null for the gaps

The second option is what we had before, so I think at this point perhaps we can just go with option 1?