PUT request causes server to stop responding and memory use increases when user is a part of many Roles

[spenceps]

• You've met the prerequisites.
• You're running the latest version of Parse Server.
• You've searched through existing issues. Chances are that your issue has been reported or resolved before.

Environment Setup

• running parse-sever v2.4.5 on Heroku and locally, this has been happening since at least version 2.2.0
• Using Parse iOS SDK version 1.13.0 installed by Cocoa Pods

Steps to reproduce

A user needs to be logged in and be a part of many Roles. The situation I have is that the user is a part of 14k roles (Mainly through roles relation).

Update an object:

PFUser* user = [PFUser currentUser];
user[@"notUsed"] = @(random());

[user saveInBackgroundWithBlock:^(BOOL succeeded, NSError * _Nullable error) {
    NSLog(@"saved user %d",succeeded);
}];

Server never responds running locally or in Heroku's case a 30 second timeout is hit.

Memory usage keeps climbing.

Logs/Trace

Parse-server logs with VERBOSE

verbose: PUT /parse/classes/_User/CkSazcRv3Q { host: 'subdomain.herokuapp.com',
  'x-parse-client-version': 'i1.13.0',
  accept: '*/*',
  'x-parse-session-token': 'r:390c7f9(rest of session token)',
  'x-parse-application-id': 'zIIH9cru(rest of app id)',
  'x-parse-client-key': 'VBt6qm7Zq(rest of client key)',
  'x-parse-installation-id': 'cc25c6c4-eda1-45e5-a744-ffb843b64d22',
  'x-parse-os-version': '9.3 (15E65)',
  'accept-language': 'en-us',
  'accept-encoding': 'gzip, deflate',
  'content-type': 'application/json; charset=utf-8',
  'content-length': '22',
  'user-agent': 'Brandr/65 CFNetwork/758.3.15 Darwin/15.4.0',
  connection: 'keep-alive',
  'x-parse-app-build-version': '65',
  'x-parse-app-display-version': '3.4' } {
  "notUsed": 1804289383
}

Server Errors (Heroku)

Apr 05 09:07:29 brandr-staging heroku/router:  at=error code=H12 desc="Request timeout" method=PUT path="/parse/classes/_User/CkSazcRv3Q" host=subdomain.herokuapp.com request_id=2394afd7-4abd-4fcd-b6b4-853a7211566e fwd="76.8.212.59" dyno=web.1 connect=0ms service=30010ms status=503 bytes=0 

What I've figured out so far

I tried stepping through the save process with a local running parse-server and a remote database (Object Rocket). I found that in the RestQuery.js execute function, _this.runFind() gets called alot but _this.runCount() not so much. So I added the print statement shown below.

    RestQuery.prototype.execute = function () {
        var _this = this;

      return Promise.resolve().then(function () {
        return _this.buildRestWhere();
      }).then(function () {
        console.log("will run find " + _this.className + " id:" + JSON.stringify(_this.restWhere));
        return _this.runFind();
      }).then(function () {
        return _this.runCount();
      }).then(function () {
        return _this.handleInclude();
      }).then(function () {
        return _this.response;
      });
    };

I found that this was being executed on every Role that the user was a part of via the roles relation. There are around 14K roles in this DB with each role having a roles relation to an Admin role that the user I logged in was a part of. An exerpt from the console showed the following (somewhere around 14k times):

will run find _Role id:{"roles":{"__type":"Pointer","className":"_Role","objectId":"ywpqrQfmZ8"}}
will run find _Role id:{"roles":{"__type":"Pointer","className":"_Role","objectId":"yxTfXBCKWO"}}
will run find _Role id:{"roles":{"__type":"Pointer","className":"_Role","objectId":"yxvmYSDRCF"}}
will run find _Role id:{"roles":{"__type":"Pointer","className":"_Role","objectId":"yy45AJQjEN"}}
will run find _Role id:{"roles":{"__type":"Pointer","className":"_Role","objectId":"yyFqpgF634"}}
will run find _Role id:{"roles":{"__type":"Pointer","className":"_Role","objectId":"yyMvs0jzar"}}
will run find _Role id:{"roles":{"__type":"Pointer","className":"_Role","objectId":"yyRL1PccbJ"}}
will run find _Role id:{"roles":{"__type":"Pointer","className":"_Role","objectId":"yycMseo1zA"}}
will run find _Role id:{"roles":{"__type":"Pointer","className":"_Role","objectId":"yysqrApSC7"}}
will run find _Role id:{"roles":{"__type":"Pointer","className":"_Role","objectId":"yzFbtjaklN"}}
will run find _Role id:{"roles":{"__type":"Pointer","className":"_Role","objectId":"yzLihKLOb4"}}

It appears that in order to make an authorized update request parse-server gets all of the roles that the authorized user is a part of. It would make more sense to see what roles would be required by the object that is being updated and get those if needed. It is possible in my situation to change the ACL on a couple classes so that no user has more than a few Roles that they are a part of. However, for compatiblity from the Parse hosted server to self-hosted, open source parse-server it would make more sense to change the way auth handles Roles for update requests.

[flovilmart]

@spenceps we've just released 2.2.6 that has an optimization for role gathering #1366. We could also optimize the role gathering process into 1 query per level.

How big is you role hierarchy in terms of relation levels?

[gfosco]

edit: after some discussion offline, it sounds like we could improve performance here by changing how roles are queried and cached.

[spenceps]

The hierarchy is just one level deep. I have one Admin role that is the only role relation on a TeamAdmin role that is made for every Team object that is created.

[flovilmart]

@spenceps I've refactored the code in the PR it should run in O(n) time. As you only have 1 level of hierarchy that should be constant time. For the memory usage, can you let us know how it's behaving with the fix?

You can run it from the development package through babel node

[spenceps]

@flovilmart It is working a lot better now. The request is taking 10-15 seconds now as opposed to never returning. Memory looks a lot better too. I tried to get the changes pushed up to Heroku to make it easier to quantify the memory improvements but for some reason Heroku wasn't recognizing the submodule. When this change gets released I'll look at the memory closer. Thanks.

[flovilmart]

That's still a pretty long time. We'll need to provide some caching opportunity so you can handle that caching yourself. and prevent fetching all those roles. What would be interesting is to have a trace of the requests, so we can understand where we spend the time.

[spenceps]

It would be cool to be able to cache to Redis. I forgot to mention the first time that only the first request was slow, subsequent requests were very fast, but that probably wouldn't be the case all the time if I was in production mode and had more than one server.

I'm thinking that the best thing for me to do at this point is to change the way I do security on a couple classes so that I don't have a few users that are a part of 14k roles.

[flovilmart]

I'm thinking that the best thing for me to do at this point is to change the way I do security on a couple classes so that I don't have a few users that are a part of 14k roles.

That would be bring a serious improvement. What I suggest is that you create a SuperAdmin role and attach that role to all the objects that would require it. Then add your users to SuperAdmin. That would work, but also increase the burden to flag all the correct objects.

It would be cool to be able to cache to Redis.

We could expose a role Cache somewhere, with RolesCache.get(roleName), RolesCache.update(roleName) then it would be your responsibility to invalidate, and managed that cache.
But I don't see it as a high priority for now.

[spenceps]

@flovilmart Here is the difference that this change made. Before v35 is without this change and after is the same thing done with the change. It comes close to using 1 GB before and 150 MB after.

This is on a Heroku dyno with 512 MB RAM so Heroku might have done something because it was over the limit.

[flovilmart]

that's a dramatic improvement!
We could definitely investigate the caching through in memory or redis. I you have time to implement, I'd welcome the PR!