Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test, benchmark: create shared runBenchmark func #15004

Closed
wants to merge 6 commits into from
Closed

test, benchmark: create shared runBenchmark func #15004

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7ab1619
test, benchmark: create shared runBenchmark func
maclover7 Aug 24, 2017
a6395a9
[squash] require common + convert forgotten arrays benchmarks
maclover7 Aug 24, 2017
ae90f1f
[squash] fix typo
maclover7 Aug 24, 2017
4388c22
[squash] rename benchmarks to benchmark
maclover7 Aug 25, 2017
3e5e2e2
[squash] add initial documentation
maclover7 Sep 1, 2017
b3f4c41
[squash] convert test-benchmark-dns
maclover7 Sep 2, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions test/common/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,21 @@ This directory contains modules used to test the Node.js implementation.

## Table of Contents

* [Benchmark module](#benchmark-module)
* [Common module API](#common-module-api)
* [WPT module](#wpt-module)

## Benchmark Module

The `benchmark` module is used by tests to run benchmarks.

### runBenchmark(name, args, env)

* `name` [<String>] Name of benchmark suite to be run.
* `args` [<Array>] Array of environment variable key/value pairs (ex:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I think a better name for the parameter may be "settings"? Also the environment variable seems to be a misnomer, as does the "key/value pair" verbiage which sounds like an array (though the example definitely helps for that case). The "Array" type annotation could be changed to string[] to better represent the member types.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I was thinking "args" because this just does --set for whatever is passed in (args as in command line args). This argument is semi confusing, but I am planning on submitting a PR after this is merged to convert this argument to be an Object so that way it's easier to understand -- using the array for right now is just to avoid a giant diff :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW it's called configs in createBenchmark, though in a context like this it should be config.

  • configs {Object} The benchmark parameters. createBenchmark will run all
    possible combinations of these parameters, unless specified otherwise.
    Each configuration is a property with an array of possible values.
    Note that the configuration values can only be strings or numbers.

`n=1`) to be applied via `--set`.
* `env` [<Object>] Environment variables to be applied during the run.

## Common Module API

The `common` module is used by tests for consistency across repeated
Expand Down
30 changes: 30 additions & 0 deletions test/common/benchmark.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
/* eslint-disable required-modules */

'use strict';

const assert = require('assert');
const fork = require('child_process').fork;
const path = require('path');

const runjs = path.join(__dirname, '..', '..', 'benchmark', 'run.js');

function runBenchmark(name, args, env) {
const argv = [];

for (let i = 0; i < args.length; i++) {
argv.push('--set');
argv.push(args[i]);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: a for-of loop could be used here, and the argv.push lines can be coalesced.

}

argv.push(name);

const mergedEnv = Object.assign({}, process.env, env);

const child = fork(runjs, argv, { env: mergedEnv });
child.on('exit', (code, signal) => {
assert.strictEqual(code, 0);
assert.strictEqual(signal, null);
});
}

module.exports = runBenchmark;
18 changes: 2 additions & 16 deletions test/parallel/test-benchmark-arrays.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,6 @@

require('../common');

// Minimal test for arrays benchmarks. This makes sure the benchmarks aren't
// horribly broken but nothing more than that.
const runBenchmark = require('../common/benchmark');

const assert = require('assert');
const fork = require('child_process').fork;
const path = require('path');

const runjs = path.join(__dirname, '..', '..', 'benchmark', 'run.js');
const argv = ['--set', 'n=1',
'--set', 'type=Array',
'arrays'];

const child = fork(runjs, argv);
child.on('exit', (code, signal) => {
assert.strictEqual(code, 0);
assert.strictEqual(signal, null);
});
runBenchmark('arrays', ['n=1', 'type=Array']);
19 changes: 2 additions & 17 deletions test/parallel/test-benchmark-cluster.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,6 @@

require('../common');

// Minimal test for cluster benchmarks. This makes sure the benchmarks aren't
// horribly broken but nothing more than that.
const runBenchmark = require('../common/benchmark');

const assert = require('assert');
const fork = require('child_process').fork;
const path = require('path');

const runjs = path.join(__dirname, '..', '..', 'benchmark', 'run.js');
const argv = ['--set', 'n=1',
'--set', 'payload=string',
'--set', 'sendsPerBroadcast=1',
'cluster'];

const child = fork(runjs, argv);
child.on('exit', (code, signal) => {
assert.strictEqual(code, 0);
assert.strictEqual(signal, null);
});
runBenchmark('cluster', ['n=1', 'payload=string', 'sendsPerBroadcast=1']);
41 changes: 15 additions & 26 deletions test/parallel/test-benchmark-crypto.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,29 +8,18 @@ if (!common.hasCrypto)
if (common.hasFipsCrypto)
common.skip('some benchmarks are FIPS-incompatible');

// Minimal test for crypto benchmarks. This makes sure the benchmarks aren't
// horribly broken but nothing more than that.

const assert = require('assert');
const fork = require('child_process').fork;
const path = require('path');

const runjs = path.join(__dirname, '..', '..', 'benchmark', 'run.js');
const argv = ['--set', 'algo=sha256',
'--set', 'api=stream',
'--set', 'keylen=1024',
'--set', 'len=1',
'--set', 'n=1',
'--set', 'out=buffer',
'--set', 'type=buf',
'--set', 'v=crypto',
'--set', 'writes=1',
'crypto'];

const child = fork(runjs, argv, { env: Object.assign({}, process.env, {
NODEJS_BENCHMARK_ZERO_ALLOWED: 1 }) });

child.on('exit', (code, signal) => {
assert.strictEqual(code, 0);
assert.strictEqual(signal, null);
});
const runBenchmark = require('../common/benchmark');

runBenchmark('crypto',
[
'n=1',
'algo=sha256',
'api=stream',
'keylen=1024',
'len=1',
'out=buffer',
'type=buf',
'v=crypto',
'writes=1'
],
{ NODEJS_BENCHMARK_ZERO_ALLOWED: 1 });
21 changes: 2 additions & 19 deletions test/parallel/test-benchmark-dns.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,9 @@

require('../common');

// Minimal test for dns benchmarks. This makes sure the benchmarks aren't
// horribly broken but nothing more than that.

const assert = require('assert');
const fork = require('child_process').fork;
const path = require('path');

const runjs = path.join(__dirname, '..', '..', 'benchmark', 'run.js');
const runBenchmark = require('../common/benchmark');

const env = Object.assign({}, process.env,
{ NODEJS_BENCHMARK_ZERO_ALLOWED: 1 });

const child = fork(runjs,
['--set', 'n=1',
'--set', 'all=false',
'--set', 'name=127.0.0.1',
'dns'],
{ env });

child.on('exit', (code, signal) => {
assert.strictEqual(code, 0);
assert.strictEqual(signal, null);
});
runBenchmark('dns', ['n=1', 'all=false', 'name=127.0.0.1'], env);
18 changes: 2 additions & 16 deletions test/parallel/test-benchmark-domain.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,6 @@

require('../common');

// Minimal test for domain benchmarks. This makes sure the benchmarks aren't
// horribly broken but nothing more than that.
const runBenchmark = require('../common/benchmark');

const assert = require('assert');
const fork = require('child_process').fork;
const path = require('path');

const runjs = path.join(__dirname, '..', '..', 'benchmark', 'run.js');
const argv = ['--set', 'arguments=0',
'--set', 'n=1',
'domain'];

const child = fork(runjs, argv);
child.on('exit', (code, signal) => {
assert.strictEqual(code, 0);
assert.strictEqual(signal, null);
});
runBenchmark('domain', ['n=1', 'arguments=0']);
17 changes: 2 additions & 15 deletions test/parallel/test-benchmark-events.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,6 @@

require('../common');

// Minimal test for events benchmarks. This makes sure the benchmarks aren't
// horribly broken but nothing more than that.
const runBenchmark = require('../common/benchmark');

const assert = require('assert');
const fork = require('child_process').fork;
const path = require('path');

const runjs = path.join(__dirname, '..', '..', 'benchmark', 'run.js');
const argv = ['--set', 'n=1',
'events'];

const child = fork(runjs, argv);
child.on('exit', (code, signal) => {
assert.strictEqual(code, 0);
assert.strictEqual(signal, null);
});
runBenchmark('events', ['n=1']);
17 changes: 2 additions & 15 deletions test/parallel/test-benchmark-os.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,6 @@

require('../common');

// Minimal test for os benchmarks. This makes sure the benchmarks aren't
// horribly broken but nothing more than that.
const runBenchmark = require('../common/benchmark');

const assert = require('assert');
const fork = require('child_process').fork;
const path = require('path');

const runjs = path.join(__dirname, '..', '..', 'benchmark', 'run.js');
const argv = ['--set', 'n=1',
'os'];

const child = fork(runjs, argv);
child.on('exit', (code, signal) => {
assert.strictEqual(code, 0);
assert.strictEqual(signal, null);
});
runBenchmark('os', ['n=1']);
30 changes: 10 additions & 20 deletions test/parallel/test-benchmark-path.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,13 @@

require('../common');

// Minimal test for path benchmarks. This makes sure the benchmarks aren't
// horribly broken but nothing more than that.

const assert = require('assert');
const fork = require('child_process').fork;
const path = require('path');

const runjs = path.join(__dirname, '..', '..', 'benchmark', 'run.js');
const argv = ['--set', 'n=1',
'--set', 'path=',
'--set', 'pathext=',
'--set', 'paths=',
'--set', 'props=',
'path'];

const child = fork(runjs, argv);
child.on('exit', (code, signal) => {
assert.strictEqual(code, 0);
assert.strictEqual(signal, null);
});
const runBenchmark = require('../common/benchmark');

runBenchmark('path',
[
'n=1',
'path=',
'pathext=',
'paths=',
'props='
]);
26 changes: 8 additions & 18 deletions test/parallel/test-benchmark-process.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,11 @@

require('../common');

// Minimal test for process benchmarks. This makes sure the benchmarks aren't
// horribly broken but nothing more than that.

const assert = require('assert');
const fork = require('child_process').fork;
const path = require('path');

const runjs = path.join(__dirname, '..', '..', 'benchmark', 'run.js');
const argv = ['--set', 'millions=0.000001',
'--set', 'n=1',
'--set', 'type=raw',
'process'];

const child = fork(runjs, argv);
child.on('exit', (code, signal) => {
assert.strictEqual(code, 0);
assert.strictEqual(signal, null);
});
const runBenchmark = require('../common/benchmark');

runBenchmark('process',
[
'millions=0.000001',
'n=1',
'type=raw'
]);
30 changes: 9 additions & 21 deletions test/parallel/test-benchmark-timers.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,12 @@

require('../common');

// Minimal test for timers benchmarks. This makes sure the benchmarks aren't
// horribly broken but nothing more than that.

const assert = require('assert');
const fork = require('child_process').fork;
const path = require('path');

const runjs = path.join(__dirname, '..', '..', 'benchmark', 'run.js');
const argv = ['--set', 'type=depth',
'--set', 'millions=0.000001',
'--set', 'thousands=0.001',
'timers'];

const env = Object.assign({}, process.env,
{ NODEJS_BENCHMARK_ZERO_ALLOWED: 1 });

const child = fork(runjs, argv, { env });
child.on('exit', (code, signal) => {
assert.strictEqual(code, 0);
assert.strictEqual(signal, null);
});
const runBenchmark = require('../common/benchmark');

runBenchmark('timers',
[
'type=depth',
'millions=0.000001',
'thousands=0.001'
],
{ NODEJS_BENCHMARK_ZERO_ALLOWED: 1 });
28 changes: 9 additions & 19 deletions test/parallel/test-benchmark-zlib.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,12 @@

require('../common');

// Minimal test for zlib benchmarks. This makes sure the benchmarks aren't
// horribly broken but nothing more than that.

const assert = require('assert');
const fork = require('child_process').fork;
const path = require('path');

const runjs = path.join(__dirname, '..', '..', 'benchmark', 'run.js');
const argv = ['--set', 'method=deflate',
'--set', 'n=1',
'--set', 'options=true',
'--set', 'type=Deflate',
'zlib'];

const child = fork(runjs, argv);
child.on('exit', (code, signal) => {
assert.strictEqual(code, 0);
assert.strictEqual(signal, null);
});
const runBenchmark = require('../common/benchmark');

runBenchmark('zlib',
[
'method=deflate',
'n=1',
'options=true',
'type=Deflate'
]);
Loading