Move count from GetImages payload to GetStats

[nathanielrindlaub]

Problem

We've discovered that counting all images that match a given aggregation pipeline is incredibly expensive and is now causing timeouts on GetImages queries. In our testing, getting the count took ~1000x longer than finding the paginated images themselves:

  images: async (_, { input }, context) => {
    const count = await context.models.Image.countImages(input, context); // this often takes >10 seconds when querying projects with lots of images (>250k).
    const response = await context.models.Image.queryByFilter(input, context); // this typically takes less than 15ms
    const { previous, hasPrevious, next, hasNext, results } = response;
    return {
      pageInfo: {
        previous,
        hasPrevious,
        next,
        hasNext,
        count
      },
      images: results
    };
  },

Apparently this is a known inefficiency with MongoDB (see this and this for more info).

Solution

Providing an accurate count of images that match a given set of filters is definitely a requirement, but as a compromise, we've decided to move the image count out of GetImages queries and into GetStats. The downside for the users is that they don't always get to see the matching image count displayed every time they change the filters, but they would still be able to access that information if they wanted to dig into the stats for their current selection.

Tasks

• Remove count from GetImages payload
• Query image count separately from image records, indicated getImagesCount loading state in "matching images" display on frontend
• Build out better async task management (TaskMan #159)
• Move GetStats function over to its own lambda, create a queue for it, and convert the GetStats graphql-api resolver to send an SQS message requesting GetStats which can be picked up and handled by the new Lambda
• worth streaming the records into the Lambda instead of pulling all of them into memory and iterating over them (Stream images from DB into getStats reducer #67)
• Wire that up to the frontend and implement polling.
• Style the stats modal (Style stats modal animl-frontend#91)

[nathanielrindlaub]

FYI @ingalls and @jue-henry, I implemented what I hoped might be a temporary fix (https://github.com/tnc-ca-geo/animl-api/tree/image-count-temp-fix) in which we split out the query for GetImagesCount into it's own resolver. The idea was if the frontend can request the count independently of requesting the images, even if the count times out we can display that on the front end, but users can still view and interact with their filtered images (corresponding frontend branch is here: https://github.com/tnc-ca-geo/animl-frontend/tree/images-count-temp-fix).

This seems to work ok and as expected for most queries I've tried in prod thus far, except for trying to filter out (unselect) 'reviewed images' on the sci_biosecurity project. For that one, it's the GetImages query that is taking longer and timing out(!), not the GetImagesCount. Super odd...

@jue-henry that would be a good one to drill into in your indexing exploration.

[nathanielrindlaub]

Still have to style the stats dashboard, but I'm going to consider this done.