Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issue on count with Geo constraints and mongo (issue #5285) #5286

Merged
merged 4 commits into from
Apr 25, 2019

Conversation

jlnquere
Copy link
Contributor

@jlnquere jlnquere commented Jan 9, 2019

Here is a unit test that demonstrates count issues with Geo constraints ( cf #5285 ).

This test actually fails.

@flovilmart
Copy link
Contributor

Thanks for reporting the issue this way! This is very helpful. I believe this is related to the way mongodb’s driver behaves since the 3.0 update. I recall it changed the default count behaviors. Are you willing to have a look for the fix? Most likely, this is located in the storage adapter implementation.

@dplewis
Copy link
Member

dplewis commented Jan 9, 2019

Caused by #5025

There is a guide in the following link: Migrating count to countDocuments

@jlnquere
Copy link
Contributor Author

I dug into the code, and the differences between 3.0 (the last working version) and 3.1. I wasn't able to spot the precise cause of the problem (disclaimer: I’m really new to backend / JS / Mongo dev :/ ).

I saw the Mongo Adapter refactor. It might be related to this, but I'm unable to say how.

@dplewis
Copy link
Member

dplewis commented Jan 10, 2019

@jlnquere In my previous comment I posted the cause and a potential solution.

https://github.com/parse-community/parse-server/blob/master/src/Adapters/Storage/Mongo/MongoCollection.js#L93

The query must be transformed before passing it into countDocument

@jlnquere
Copy link
Contributor Author

@dplewis Thanks for the pointer. I read again your post and dug into the code. I don't see how I can replace $nearSphere by $geoWithin with $centerSphere into this method.

Is it something that should be done in count ? Or is it something that should be done when the query is built ? If yes, how ?

@codecov
Copy link

codecov bot commented Jan 16, 2019

Codecov Report

Merging #5286 into master will increase coverage by 0.52%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #5286      +/-   ##
==========================================
+ Coverage   93.95%   94.48%   +0.52%     
==========================================
  Files         123      123              
  Lines        8970    10485    +1515     
==========================================
+ Hits         8428     9907    +1479     
- Misses        542      578      +36
Impacted Files Coverage Δ
src/Adapters/Storage/Mongo/MongoCollection.js 97.67% <100%> (+0.23%) ⬆️
...rc/Adapters/Storage/Mongo/MongoSchemaCollection.js 93.96% <0%> (-1.33%) ⬇️
src/ParseServer.js 95.68% <0%> (-1.07%) ⬇️
src/Adapters/Storage/Mongo/MongoStorageAdapter.js 91.48% <0%> (-0.73%) ⬇️
src/RestWrite.js 93.2% <0%> (-0.41%) ⬇️
src/Options/Definitions.js 100% <0%> (ø) ⬆️
src/Controllers/SchemaController.js 96.93% <0%> (+0.61%) ⬆️
src/RestQuery.js 96.82% <0%> (+0.75%) ⬆️
...dapters/Storage/Postgres/PostgresStorageAdapter.js 98.01% <0%> (+0.86%) ⬆️
src/Controllers/DatabaseController.js 96.22% <0%> (+1.29%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 46ac7e7...0fe7cc9. Read the comment docs.

@jlnquere
Copy link
Contributor Author

I just pushed a fix based on a @dplewis idea.

Simply rewriting the query to replace $nearSphere in query by $geoWithin with $centerSphere.

    for (const key in query) {
      if (query[key].$nearSphere) {
        const geoQuery = {
          $geoWithin: {
            $centerSphere: [query[key].$nearSphere, query[key].$maxDistance],
          },
        };
        query[key] = geoQuery;
      }
    }

All credit goes to @dplewis. Thanks for the idea 👍

@dplewis
Copy link
Member

dplewis commented Jan 16, 2019

@jlnquere Actually I think its best to keep the transforms inside of mongoTransform.js, just check if you are doing a count operation. The fix I provided doesn't take into account complex queries such as $or

Edit: You also have to add transform for where and near

@jlnquere jlnquere changed the title Add a tests that fails due to issue #5285 Fix issue on count with Geo constraints and mongo (issue #5285) Jan 16, 2019
@jlnquere
Copy link
Contributor Author

@dplewis I don't see how we can do that in mongoTransform.js. At this point, we don't have any information of the context of the query. We don't know if we are transforming for a count or not.

@dplewis
Copy link
Member

dplewis commented Jan 16, 2019

I don't see how we can do that in mongoTransform.js. At this point, we don't have any information of the context of the query.

A little debugging here will answer all your questions.

https://github.com/parse-community/parse-server/blob/master/src/Adapters/Storage/Mongo/MongoStorageAdapter.js#L696

We don't know if we are transforming for a count or not.

Add an extra parameter to transformWhere like isCount.

@stale
Copy link

stale bot commented Mar 3, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Mar 3, 2019
@stale stale bot closed this Mar 10, 2019
@dplewis dplewis removed the wontfix label Mar 12, 2019
@dplewis dplewis reopened this Mar 12, 2019
@dplewis
Copy link
Member

dplewis commented Apr 25, 2019

@acinader Mongo driver update that forces us to use the non-sort geo query when we use count. Since we aren't sorting the query this results in faster requests. Moving the logic to the MongoTransform allows to count with complex queries.

@dplewis dplewis requested a review from acinader April 25, 2019 00:59
Copy link
Contributor

@acinader acinader left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me.

collections.map(
collection => (fast ? collection.deleteMany({}) : collection.drop())
collections.map(collection =>
fast ? collection.deleteMany({}) : collection.drop()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this change covered in a test?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prettier making the code look good

@dplewis dplewis merged commit 7122ca0 into parse-community:master Apr 25, 2019
UnderratedDev pushed a commit to UnderratedDev/parse-server that referenced this pull request Mar 21, 2020
…ity#5285) (parse-community#5286)

* Add a tests that fails due to issue parse-community#5285

* Make test code much simpler

* Fix parse-community#5285 by rewriting query (replacing $nearSphere by $geoWithin)

All credit goes to @dplewis !

* move logic to transform
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants