Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some dep exchanges #1559

Merged
merged 1 commit into from
Dec 20, 2018
Merged

Some dep exchanges #1559

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 12 additions & 13 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,8 @@ Repository | Reference | Recent Version
[connect-mongo][connect-mongoGHUrl] | [Documentation][connect-mongoDOCUrl] | [![NPM version][connect-mongoNPMVersionImage]][connect-mongoNPMUrl]
[diff][diffGHUrl] | [Documentation][diffDOCUrl] | [![NPM version][diffNPMVersionImage]][diffNPMUrl]
[express][expressGHUrl] | [Documentation][expressDOCUrl] | [![NPM version][expressNPMVersionImage]][expressNPMUrl]
[express-brute][express-bruteGHUrl] | [Documentation][express-bruteDOCUrl] | [![NPM version][express-bruteNPMVersionImage]][express-bruteNPMUrl]
[express-brute-mongo][express-brute-mongoGHUrl] <br />&#x22D4; [`MongoDBv3.x`][express-brute-mongoGHMongoDBv3.xUrl] | [Documentation][express-brute-mongoDOCUrl] | [![NPM version][express-brute-mongoNPMVersionImage]][express-brute-mongoNPMUrl]
[express-minify][express-minifyGHUrl] | [Documentation][express-minifyDOCUrl] | [![NPM version][express-minifyNPMVersionImage]][express-minifyNPMUrl]
[express-rate-limit][express-rate-limitGHUrl] | [Documentation][express-rate-limitDOCUrl] | [![NPM version][express-rate-limitNPMVersionImage]][express-rate-limitNPMUrl]
[express-session][express-sessionGHUrl] | [Documentation][express-sessionDOCUrl] | [![NPM version][express-sessionNPMVersionImage]][express-sessionNPMUrl]
[font-awesome][font-awesomeGHUrl] | [Documentation][font-awesomeDOCUrl] | [![NPM version][font-awesomeNPMVersionImage]][font-awesomeNPMUrl]
[formidable][formidableGHUrl] | [Documentation][formidableDOCUrl] | [![NPM version][formidableNPMVersionImage]][formidableNPMUrl]
Expand Down Expand Up @@ -71,6 +70,7 @@ Repository | Reference | Recent Version
[passport-twitter][passport-twitterGHUrl] | [Documentation][passport-twitterDOCUrl] | [![NPM version][passport-twitterNPMVersionImage]][passport-twitterNPMUrl] ![oauth1][oauth1Logo]
[passport-yahoo][passport-yahooGHUrl] <br />&#x22D4; [`OpenID2`][passport-yahooGHOpenIDUrl] | [Documentation][passport-yahooDOCUrl] | [![NPM version][passport-yahooNPMVersionImage]][passport-yahooNPMUrl] ![OpenID][openidLogo] [&#x22D4;][passport-openid]
[pegjs][pegjsGHUrl] | [Documentation][pegjsDOCUrl] | [![NPM version][pegjsNPMVersionImage]][pegjsNPMUrl]
[rate-limit-mongo][rate-limit-mongoGHUrl] | [Documentation][rate-limit-mongoDOCUrl] | [![NPM version][rate-limit-mongoNPMVersionImage]][rate-limit-mongoNPMUrl]
[request][requestGHUrl] | [Documentation][requestDOCUrl] | [![NPM version][requestNPMVersionImage]][requestNPMUrl]
[rfc2047][rfc2047GHUrl] | [Documentation][rfc2047DOCUrl] | [![NPM version][rfc2047NPMVersionImage]][rfc2047NPMUrl]
[S3rver][s3rverGHUrl] | [Documentation][s3rverDOCUrl] | [![NPM version][s3rverNPMVersionImage]][s3rverNPMUrl]
Expand Down Expand Up @@ -190,22 +190,16 @@ Outdated dependencies list can also be achieved with `$ npm --depth 0 outdated`
[expressNPMUrl]: https://www.npmjs.com/package/express
[expressNPMVersionImage]: https://img.shields.io/npm/v/express.svg?style=flat

[express-bruteGHUrl]: https://github.com/AdamPflug/express-brute
[express-bruteDOCUrl]: https://github.com/AdamPflug/express-brute/blob/master/README.md
[express-bruteNPMUrl]: https://www.npmjs.com/package/express-brute
[express-bruteNPMVersionImage]: https://img.shields.io/npm/v/express-brute.svg?style=flat

[express-brute-mongoGHUrl]: https://github.com/auth0/express-brute-mongo
[express-brute-mongoGHMongoDBv3.xUrl]: https://github.com/OpenUserJs/express-brute-mongo/tree/MongoDBv3.x
[express-brute-mongoDOCUrl]: https://github.com/auth0/express-brute-mongo/blob/master/README.md
[express-brute-mongoNPMUrl]: https://www.npmjs.com/package/express-brute-mongo
[express-brute-mongoNPMVersionImage]: https://img.shields.io/npm/v/express-brute-mongo.svg?style=flat

[express-minifyGHUrl]: https://github.com/breeswish/express-minify
[express-minifyDOCUrl]: https://github.com/breeswish/express-minify/blob/master/README.md
[express-minifyNPMUrl]: https://www.npmjs.com/package/express-minify
[express-minifyNPMVersionImage]: https://img.shields.io/npm/v/express-minify.svg?style=flat

[express-rate-limitGHUrl]: https://github.com/nfriedly/express-rate-limit
[express-rate-limitDOCUrl]: https://github.com/nfriedly/express-rate-limit/blob/master/README.md
[express-rate-limitNPMUrl]: https://www.npmjs.com/package/express-rate-limit
[express-rate-limitNPMVersionImage]: https://img.shields.io/npm/v/express-rate-limit.svg?style=flat

[express-sessionGHUrl]: https://github.com/expressjs/session
[express-sessionDOCUrl]: https://github.com/expressjs/session/blob/master/README.md
[express-sessionNPMUrl]: https://www.npmjs.com/package/express-session
Expand Down Expand Up @@ -395,6 +389,11 @@ Outdated dependencies list can also be achieved with `$ npm --depth 0 outdated`
[pegjsNPMUrl]: https://www.npmjs.com/package/pegjs
[pegjsNPMVersionImage]: https://img.shields.io/npm/v/pegjs.svg?style=flat

[rate-limit-mongoGHUrl]: https://github.com/2do2go/rate-limit-mongo
[rate-limit-mongoDOCUrl]: https://github.com/2do2go/rate-limit-mongo/blob/master/README.md
[rate-limit-mongoNPMUrl]: https://www.npmjs.com/package/rate-limit-mongo
[rate-limit-mongoNPMVersionImage]: https://img.shields.io/npm/v/rate-limit-mongo.svg?style=flat

[requestGHUrl]: https://github.com/request/request
[requestDOCUrl]: https://github.com/request/request/blob/master/README.md
[requestNPMUrl]: https://www.npmjs.com/package/request
Expand Down
104 changes: 1 addition & 103 deletions controllers/scriptStorage.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,6 @@ var sizeOf = require('image-size');
var ipRangeCheck = require("ip-range-check");
var colors = require('ansi-colors');

var MongoClient = require('mongodb').MongoClient;
var ExpressBrute = require('express-brute');
var MongoStore = require('express-brute-mongo');

//--- Model inclusions
var Script = require('../models/script').Script;
var User = require('../models/user').User;
Expand Down Expand Up @@ -182,73 +178,6 @@ if (isPro) {
var stats = fs.statSync('./node_modules/terser/package.json');
var mtimeTerser = new Date(util.inspect(stats.mtime));

// Brute initialization
var store = null;
if (isPro) {
store = new MongoStore(function (ready) {
MongoClient.connect('mongodb://127.0.0.1:27017/test', {
useNewUrlParser: true
}, function(aErr, aClient) {
if (aErr) {
throw aErr;
}

ready(aClient.db().collection('bruteforce-store'));
});
});
} else {
store = new ExpressBrute.MemoryStore(); // stores state locally, don't use this in production
}

var tooManyRequests = function (aReq, aRes, aNext, aNextValidRequestDate) {
var secondUntilNextRequest = null;

if (isDev) {
secondUntilNextRequest = Math.ceil((aNextValidRequestDate.getTime() - Date.now())/1000);
aRes.header('Retry-After', secondUntilNextRequest);
}
aRes.status(429).send(); // Too Many Requests
}

var sweetFactor = ensureIntegerOrNull(process.env.BRUTE_SWEETFACTOR) || (2);

var installMaxBruteforce = new ExpressBrute(store, {
freeRetries: ensureIntegerOrNull(process.env.BRUTE_FREERETRIES) || (0),
minWait: ensureIntegerOrNull(process.env.BRUTE_MINWAIT) || (1000 * 60), // sec
maxWait: ensureIntegerOrNull(process.env.BRUTE_MAXWAIT) || (1000 * 60 * 15), // min
lifetime: ensureIntegerOrNull(process.env.BRUTE_LIFETIME) || undefined, //
failCallback: tooManyRequests
});

var sourceMaxBruteforce = new ExpressBrute(store, {
freeRetries: ensureIntegerOrNull(process.env.BRUTE_FREERETRIES) || (0),
minWait: ensureIntegerOrNull(process.env.BRUTE_MINWAIT / sweetFactor) ||
ensureIntegerOrNull((1000 * 60) / sweetFactor), // sec
maxWait: ensureIntegerOrNull(process.env.BRUTE_MAXWAIT / sweetFactor) ||
ensureIntegerOrNull((1000 * 60 * 15) / sweetFactor), // min
lifetime: ensureIntegerOrNull(process.env.BRUTE_LIFETIME) || undefined, //
failCallback: tooManyRequests
});

// Enabled with meta requests
var installMinBruteforce = new ExpressBrute(store, {
freeRetries: ensureIntegerOrNull(process.env.BRUTE_FREERETRIES) || (0),
minWait: ensureIntegerOrNull(process.env.BRUTE_MINWAIT) || ensureIntegerOrNull(1000 * (60 / 4)), // sec
maxWait: ensureIntegerOrNull(process.env.BRUTE_MAXWAIT) || ensureIntegerOrNull(1000 * (60 / 4)), // min
lifetime: ensureIntegerOrNull(process.env.BRUTE_LIFETIME) || undefined, //
failCallback: tooManyRequests
});

var sourceMinBruteforce = new ExpressBrute(store, {
freeRetries: ensureIntegerOrNull(process.env.BRUTE_FREERETRIES) || (0),
minWait: ensureIntegerOrNull(process.env.BRUTE_MINWAIT / sweetFactor) ||
ensureIntegerOrNull((1000 * (60 / 4)) / sweetFactor), // sec
maxWait: ensureIntegerOrNull(process.env.BRUTE_MAXWAIT / sweetFactor) ||
ensureIntegerOrNull((1000 * (60 / 4) * 15) / sweetFactor), // min
lifetime: ensureIntegerOrNull(process.env.BRUTE_LIFETIME) || undefined, //
failCallback: tooManyRequests
});

var githubHookAddresses = [];

if (isSecured) {
Expand Down Expand Up @@ -525,11 +454,6 @@ exports.unlockScript = function (aReq, aRes, aNext) {
let hasUnacceptable = false;
let hasAcceptable = false;

let rMetaJS = /\.meta\.js$/;
let wantsUserScriptMeta = null;

let isSource = /^\/src\//.test(pathname);

// Test known extensions
if (!rMetaMinUserLibJS.test(pathname)) {
aRes.status(400).send(); // Bad request
Expand Down Expand Up @@ -597,33 +521,7 @@ exports.unlockScript = function (aReq, aRes, aNext) {
return;
}

// Determine if .meta.js is wanted
wantsUserScriptMeta =
(aReq.headers.accept || '*/*').split(',').indexOf('text/x-userscript-meta') > -1 ||
rMetaJS.test(pathname);

// Test cacheable
if (isSource) {
if (cacheableScript(aReq) && process.env.FORCE_SCRIPT_NOCACHE !== 'true') {
aNext();
} else {
if (wantsUserScriptMeta) {
sourceMinBruteforce.getMiddleware({key : keyScript})(aReq, aRes, aNext);
} else {
sourceMaxBruteforce.getMiddleware({key : keyScript})(aReq, aRes, aNext);
}
}
} else {
if (cacheableScript(aReq) && process.env.FORCE_SCRIPT_NOCACHE !== 'true') {
aNext();
} else {
if (wantsUserScriptMeta) {
installMinBruteforce.getMiddleware({key : keyScript})(aReq, aRes, aNext);
} else {
installMaxBruteforce.getMiddleware({key : keyScript})(aReq, aRes, aNext);
}
}
}
aNext();
}

exports.sendScript = function (aReq, aRes, aNext) {
Expand Down
4 changes: 2 additions & 2 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,8 @@
"connect-mongo": "2.0.3",
"diff": "3.5.0",
"express": "4.16.4",
"express-brute": "1.0.1",
"express-brute-mongo": "git://github.com/OpenUserJs/express-brute-mongo#MongoDBv3.x",
"express-minify": "1.0.0",
"express-rate-limit": "3.3.2",
"express-session": "1.15.6",
"font-awesome": "4.7.0",
"formidable": "1.2.1",
Expand Down Expand Up @@ -55,6 +54,7 @@
"passport-twitter": "1.0.4",
"passport-yahoo": "git://github.com/OpenUserJs/passport-yahoo#OpenID2",
"pegjs": "0.10.0",
"rate-limit-mongo": "1.0.3",
"request": "2.88.0",
"rfc2047": "2.0.1",
"s3rver": "2.2.7",
Expand Down
23 changes: 21 additions & 2 deletions routes.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ var isPro = require('./libs/debug').isPro;
var isDev = require('./libs/debug').isDev;
var isDbg = require('./libs/debug').isDbg;

var rateLimit = require('express-rate-limit');
var MongoStore = require('rate-limit-mongo');

//
var main = require('./controllers/index');
var authentication = require('./controllers/auth');
Expand All @@ -22,6 +25,22 @@ var document = require('./controllers/document');

var statusCodePage = require('./libs/templateHelpers').statusCodePage;

var waitMin = isDev ? 1 : 10;
var installLimiter = rateLimit({
store: (isDev ? undefined : new MongoStore({
uri: 'mongodb://127.0.0.1:27017/test_db',
expireTimeMs: waitMin * 60 * 1000 // n minutes for mongo store
})),
windowMs: (isDev ? waitMin * 60 * 1000 : undefined), // n minutes for memory store
max: 100, // limit each IP to n requests per windowMs for memory store or expireTimeMs for mongo store
handler: function (aReq, aRes, aNext) {
// if (isDev) {
aRes.header('Retry-After', waitMin * 60);
// }
aRes.status(429).send();
}
});

module.exports = function (aApp) {
//--- Middleware

Expand Down Expand Up @@ -74,7 +93,7 @@ module.exports = function (aApp) {
aRes.redirect(301, '/users/' + aReq.params.username + '/scripts'); // NOTE: Watchpoint
});

aApp.route('/install/:username/:scriptname').get(scriptStorage.unlockScript, scriptStorage.sendScript);
aApp.route('/install/:username/:scriptname').get(installLimiter, scriptStorage.unlockScript, scriptStorage.sendScript);

aApp.route('/meta/:username/:scriptname').get(scriptStorage.sendMeta);

Expand All @@ -88,7 +107,7 @@ module.exports = function (aApp) {
aApp.route('/libs/:username/:scriptname/source').get(script.lib(user.editScript));

// Raw source
aApp.route('/src/:type(scripts|libs)/:username/:scriptname').get(scriptStorage.unlockScript, scriptStorage.sendScript);
aApp.route('/src/:type(scripts|libs)/:username/:scriptname').get(installLimiter, scriptStorage.unlockScript, scriptStorage.sendScript);

// Issues routes
aApp.route('/:type(scripts|libs)/:username/:scriptname/issues/:open(open|closed|all)?').get(issue.list);
Expand Down
2 changes: 1 addition & 1 deletion views/includes/documents/Frequently-Asked-Questions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ A: Yes, use the raw source route like this in the UserScript metadata block:

... notice the **src**/**scripts** p.a.t.h/t.o instead of **install**.

As an added advantage and incentive to utilizing this source route *(URL path)* with `@downloadURL` the "Too Many Requests" period is divided by a sweet factor. In other words this route is currently "less" managed than the install route. This UserScript metadata block key is not currently required but highly encouraged especially due to faulty .user.js engine updaters.
The `@downloadURL` UserScript metadata block key is not currently required but highly encouraged especially due to potential faulty .user.js engine updaters.

[greasemonkeyForFirefox]: Greasemonkey-for-Firefox
[metaJSExample]: https://openuserjs.org/meta/Marti/oujs_-_Meta_View.meta.js
Expand Down