Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rollback back-end *node* engine support to LTS #1628

Merged
merged 1 commit into from
Jun 28, 2019

Conversation

Martii
Copy link
Member

@Martii Martii commented Jun 28, 2019

  • Keeping applicable TLS restricted for eventual 12.x+ compatibility
  • Search query (modelQuery) on, at the very least, about field causes maximum CPU usage in node@12.0.0 through node@12.5.0 but not v11.x or v10.x. This causes severe lag for everyone up to observed sustained 20+ seconds on queries.
  • Restarting and exposing with transforming to secondary indexing for Script model as per all mongoose/MongoDB documentation. Use syncIndexes to maintain DB indexes.
  • Squash new deprecation warning for MongoDB/*mongoose* deprecations #1516

NOTES:

  • This is going to be a major issue when v10.x is EOL'd if node and related dep issues (mongoose, mongodb) aren't resolved
  • For some reason mongoose is emitting the index event twice when used globally... unknown reason atm. Picking local disable in favor.

Applies to #1548 and post #1603 with observed issue since then.

Refs:

Dev startup:

$ node app.js
Starting application...
Disabling GitHub `hooks` in unsecure mode
S3rver initialized
error: Error creating bucket. Bucket "openuserjs.org" already exists
info: PUT /openuserjs.org 409 11ms -
Default dev S3 bucket already exists
MongoDB connection is opened
Connected to MongoDB v3.6.12
GitHub client authenticated
Index event triggered/trapped for Script model
Script indexes:
 [ { v: 2,
    key: { _id: 1 },
    name: '_id_',
    ns: 'openuserjs_devel.scripts' },
  { v: 2,
    key: { _authorId: 1, flagged: 1, isLib: 1 },
    name: '_authorId_1_flagged_1_isLib_1',
    ns: 'openuserjs_devel.scripts',
    background: true },
  { v: 2,
    key: { installName: 1 },
    name: 'installName_1',
    ns: 'openuserjs_devel.scripts',
    background: true },
  { v: 2,
    key: { isLib: 1, author: 1, name: 1 },
    name: 'isLib_1_author_1_name_1',
    ns: 'openuserjs_devel.scripts',
    background: true } ]

Recent db stats:

> db.scripts.aggregate( [ { $indexStats: { } } ] );
{ "name" : "_id_", "key" : { "_id" : 1 }, "host" : "<db>:<port>", "accesses" : { "ops" : NumberLong(12842), "since" : ISODate("2019-06-27T04:10:25.453Z") } }
{ "name" : "isLib_1_author_1_name_1", "key" : { "isLib" : 1, "author" : 1, "name" : 1 }, "host" : "<db>:<port>", "accesses" : { "ops" : NumberLong(7348), "since" : ISODate("2019-06-27T12:26:13.574Z") } }
{ "name" : "installName_1", "key" : { "installName" : 1 }, "host" : "<db>:<port>", "accesses" : { "ops" : NumberLong(144052), "since" : ISODate("2019-06-27T04:10:25.453Z") } }
{ "name" : "_authorId_1_flagged_1_isLib_1", "key" : { "_authorId" : 1, "flagged" : 1, "isLib" : 1 }, "host" : "<db>:<port>", "accesses" : { "ops" : NumberLong(16691), "since" : ISODate("2019-06-27T04:10:25.453Z") } }

* Keeping applicable TLS restricted for eventual 12.x+ compatibility
* Search query *(modelQuery)* on, at the very least, `about` field causes maximum CPU usage in *node*@12.0.0 through *node*@12.5.0 but not v11.x or v10.x. This causes severe lag for everyone up to observed sustained 20+ seconds on queries.
* Restarting and exposing with transforming to secondary indexing for Script model as per all [*mongoose*/MongoDB documentation](https://mongoosejs.com/docs/guide.html#indexes). Use `syncIndexes` to maintain DB indexes.
* Squash new deprecation warning for OpenUserJS#1516

NOTES:
* This is going to be a major issue when v10.x is EOL'd if *node* and related dep issues *(mongoose, mongodb)* aren't resolved
* For some reason *mongoose* is emitting the `index` event twice when used globally... unknown reason atm. Picking local disable in favor.

Applies to OpenUserJS#1548 and post OpenUserJS#1603 with observed issue since then.

Refs:
* https://mongoosejs.com/docs/guide.html#indexes

Dev startup:

``` console
$ node app.js
Starting application...
Disabling GitHub `hooks` in unsecure mode
S3rver initialized
error: Error creating bucket. Bucket "openuserjs.org" already exists
info: PUT /openuserjs.org 409 11ms -
Default dev S3 bucket already exists
MongoDB connection is opened
Connected to MongoDB v3.6.12
GitHub client authenticated
Index event triggered/trapped for Script model
Script indexes:
 [ { v: 2,
    key: { _id: 1 },
    name: '_id_',
    ns: 'openuserjs_devel.scripts' },
  { v: 2,
    key: { _authorId: 1, flagged: 1, isLib: 1 },
    name: '_authorId_1_flagged_1_isLib_1',
    ns: 'openuserjs_devel.scripts',
    background: true },
  { v: 2,
    key: { installName: 1 },
    name: 'installName_1',
    ns: 'openuserjs_devel.scripts',
    background: true },
  { v: 2,
    key: { isLib: 1, author: 1, name: 1 },
    name: 'isLib_1_author_1_name_1',
    ns: 'openuserjs_devel.scripts',
    background: true } ]
```

Recent db stats:

``` console
> db.scripts.aggregate( [ { $indexStats: { } } ] );
{ "name" : "_id_", "key" : { "_id" : 1 }, "host" : "<db>:<port>", "accesses" : { "ops" : NumberLong(12842), "since" : ISODate("2019-06-27T04:10:25.453Z") } }
{ "name" : "isLib_1_author_1_name_1", "key" : { "isLib" : 1, "author" : 1, "name" : 1 }, "host" : "<db>:<port>", "accesses" : { "ops" : NumberLong(7348), "since" : ISODate("2019-06-27T12:26:13.574Z") } }
{ "name" : "installName_1", "key" : { "installName" : 1 }, "host" : "<db>:<port>", "accesses" : { "ops" : NumberLong(144052), "since" : ISODate("2019-06-27T04:10:25.453Z") } }
{ "name" : "_authorId_1_flagged_1_isLib_1", "key" : { "_authorId" : 1, "flagged" : 1, "isLib" : 1 }, "host" : "<db>:<port>", "accesses" : { "ops" : NumberLong(16691), "since" : ISODate("2019-06-27T04:10:25.453Z") } }
```
@Martii Martii added tracking upstream Waiting, watching, wanting. migration Use this to indicate that it may apply to an existing or announced migration. CODE Some other Code related issue and it should clearly describe what it is affecting in a comment. needs mitigation Needs additional followup. labels Jun 28, 2019
@Martii Martii merged commit 2df9157 into OpenUserJS:master Jun 28, 2019
@Martii Martii deleted the rollbackNode branch June 28, 2019 01:28
@Martii Martii added needs testing Anyone can add this but it is primarily there for the Assignee indicating that Testers are wanted. security Usually relates to something critical. labels Jun 28, 2019
@Martii
Copy link
Member Author

Martii commented Jun 28, 2019

Guhhh still present with multiple queries outside of myself... ughh... plan B

Martii added a commit to Martii/OpenUserJS.org that referenced this pull request Jun 28, 2019
* Limit a few more things to possibly curtail some 431's
* Ease up on MongoDB having issues with script querying and CPU load. Limit to first copied block of characters in `about`. This amount may get smaller and is currently in flux.
* Create new clipped searchable fields. I spent a lot of time to see if MongoDB/*mongoose* had a limiter on their indexer and couldn't find one so we get to do it manually.

NOTE:
* This will temporarily remove everyone's Summary but it will be manually migrated in the DB in a while. If you are impatient... edit your script *(no changes necessary)* and resave it... that will do the migration also.

Post OpenUserJS#1628 and applies to OpenUserJS#1548
@Martii Martii mentioned this pull request Jun 28, 2019
Martii added a commit that referenced this pull request Jun 28, 2019
* Limit a few more things to possibly curtail some 431's
* Ease up on MongoDB having issues with script querying and CPU load. Limit to first copied block of characters in `about`. This amount may get smaller and is currently in flux.
* Create new clipped searchable fields. I spent a lot of time to see if MongoDB/*mongoose* had a limiter on their indexer and couldn't find one so we get to do it manually.

NOTE:
* This will temporarily remove everyone's Summary but it will be manually migrated in the DB in a while. If you are impatient... edit your script *(no changes necessary)* and resave it... that will do the migration also.

Post #1628 and applies to #1548

Auto-merge
@Martii Martii added the DB Pertains inclusively to the Database operations. label Jun 28, 2019
Martii added a commit to Martii/OpenUserJS.org that referenced this pull request Dec 4, 2019
* Please read their CHANGELOGs
* Delete op retested
* Post OpenUserJS#1628 partial rollback now that v12.x is LTS and no longer the issue in OpenUserJS#1548
@Martii Martii mentioned this pull request Dec 4, 2019
Martii added a commit that referenced this pull request Dec 4, 2019
* Please read their CHANGELOGs
* Delete op retested
* Post #1628 partial rollback now that v12.x is LTS and no longer the issue in #1548

Auto-merge
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 1, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
CODE Some other Code related issue and it should clearly describe what it is affecting in a comment. DB Pertains inclusively to the Database operations. migration Use this to indicate that it may apply to an existing or announced migration. needs mitigation Needs additional followup. needs testing Anyone can add this but it is primarily there for the Assignee indicating that Testers are wanted. security Usually relates to something critical. tracking upstream Waiting, watching, wanting.
Development

Successfully merging this pull request may close these issues.

1 participant