Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jest-circus runs children in shuffled order #12922

Merged
merged 92 commits into from
Feb 23, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
e43ca99
jest-circus runs children in shuffled order
jhwang98 Jun 7, 2022
bbb1cf3
added shuffle and pseudorandom number generator
jhwang98 Jun 8, 2022
4c1d07c
merged master
Joshua-Hwang Aug 25, 2022
99ac4cd
added cli argv
Joshua-Hwang Sep 3, 2022
882f920
removed unnecessary dependencies
Joshua-Hwang Sep 3, 2022
4df23a9
add console logs about the seed
Joshua-Hwang Sep 3, 2022
dc6e64f
More readable Math.pow()
Joshua-Hwang Sep 3, 2022
3d09cfb
Used recommended logging approach
Joshua-Hwang Sep 3, 2022
f8860e2
Used recommended logging approach
Joshua-Hwang Sep 3, 2022
0bec19f
added prando rng
Joshua-Hwang Sep 3, 2022
3a1d712
bug where rng was recreated every run
Joshua-Hwang Sep 3, 2022
95590dc
swapped order of logs
Joshua-Hwang Sep 3, 2022
14349bd
added typing for the rng
Joshua-Hwang Sep 5, 2022
549988d
Merge branch 'main' of github.com:facebook/jest into feat-randomise-t…
Joshua-Hwang Sep 5, 2022
4ae8ff2
linting issues fixed
jhwang98 Sep 29, 2022
967b32e
unit tests are green
jhwang98 Sep 29, 2022
561146c
used snapshots instead
jhwang98 Sep 30, 2022
2f95b65
added e2e tests
jhwang98 Sep 30, 2022
50bada0
changed valid seeds
jhwang98 Sep 30, 2022
b78fdc2
added changelog message
jhwang98 Sep 30, 2022
3f33508
improved cli help and added documentation
jhwang98 Sep 30, 2022
d84caa1
Merge branch 'main' into feat-randomise-tests
jhwang98 Sep 30, 2022
b5c34fe
Update docs/CLI.md
jhwang98 Sep 30, 2022
b9c5139
Update docs/CLI.md
jhwang98 Sep 30, 2022
1b412b4
skipped jasmine
jhwang98 Sep 30, 2022
c45244c
updated cli help and docs for randomize
jhwang98 Sep 30, 2022
e1e73a0
merged
jhwang98 Sep 30, 2022
2152031
updated documentation order
jhwang98 Sep 30, 2022
6bb9449
Update docs/CLI.md
jhwang98 Sep 30, 2022
164d6fa
Update docs/CLI.md
jhwang98 Sep 30, 2022
79401d1
removed whitespace
jhwang98 Sep 30, 2022
b6c4da4
added copyright headers
jhwang98 Oct 4, 2022
19cf2b5
modifed snapshots to not rely on the actual lines
jhwang98 Oct 4, 2022
ca3a582
Merge branch 'main' into feat-randomise-tests
SimenB Oct 4, 2022
2007d47
move changelog entry
SimenB Oct 4, 2022
9361b80
Merge branch 'facebook:main' into feat-randomise-tests
Joshua-Hwang Oct 6, 2022
4d61ba2
merged master
Joshua-Hwang Oct 12, 2022
a087ce6
added documentation for configuration
Joshua-Hwang Oct 14, 2022
7b4bd2b
merged main
Joshua-Hwang Oct 14, 2022
b175652
using pure-rand
Joshua-Hwang Oct 14, 2022
54cc38c
updated cli message
Joshua-Hwang Oct 14, 2022
974bc5b
added randomize to config
Joshua-Hwang Oct 14, 2022
b4e448f
randomize gets passed to config objects
Joshua-Hwang Oct 14, 2022
816b8e6
lint fix
Joshua-Hwang Oct 14, 2022
4ba33d4
group options returns randomize
Joshua-Hwang Oct 14, 2022
7b40ca5
removed unnecessary messages
Joshua-Hwang Oct 14, 2022
ebf01d5
improved docs
Joshua-Hwang Oct 14, 2022
9172451
updated types
Joshua-Hwang Oct 14, 2022
0797346
changed the type of the rng
Joshua-Hwang Oct 14, 2022
9c0ff4e
unit tests green
Joshua-Hwang Oct 14, 2022
604499d
added unit tests for the shuffling of tests
Joshua-Hwang Oct 14, 2022
001d150
e2e tests green
Joshua-Hwang Oct 14, 2022
30b3452
small change to docs
Joshua-Hwang Oct 14, 2022
8661cad
e2e test for the config
Joshua-Hwang Oct 14, 2022
81d526b
tweaked changelog
Joshua-Hwang Oct 14, 2022
825fdd9
linting fix
Joshua-Hwang Oct 14, 2022
7512738
factorised test
Joshua-Hwang Oct 14, 2022
888f565
Merge branch 'main' into feat-randomise-tests
Joshua-Hwang Oct 17, 2022
a6f6982
Merge branch 'main' into feat-randomise-tests
Joshua-Hwang Oct 20, 2022
57e9836
linting fix
Joshua-Hwang Oct 21, 2022
b844e1b
yarn dedupe
Joshua-Hwang Oct 21, 2022
bbc70a1
show seed true for randomize
Joshua-Hwang Oct 21, 2022
d4509cb
updated docs about randomize
Joshua-Hwang Oct 21, 2022
6ce659d
e2e tests also test randomize
Joshua-Hwang Oct 21, 2022
7bdad9f
updated error message
Joshua-Hwang Oct 21, 2022
08ed42c
Merge branch 'main' of github.com:facebook/jest into feat-randomise-t…
Joshua-Hwang Oct 22, 2022
5fd2e2a
move shuffleArray to jest-circus
Joshua-Hwang Oct 22, 2022
2bffe52
prettier
SimenB Oct 22, 2022
f5b919c
improved randomize docs
Joshua-Hwang Oct 22, 2022
2688290
merged master
Joshua-Hwang Oct 28, 2022
ec85fb8
Merge branch 'main' of github.com:facebook/jest into feat-randomise-t…
Joshua-Hwang Oct 31, 2022
94c0740
Merge branch 'main' of github.com:facebook/jest into feat-randomise-t…
Joshua-Hwang Nov 3, 2022
e212ed9
Merge branch 'main' into feat-randomise-tests
SimenB Nov 7, 2022
5946e84
lint fix
jhwang98 Nov 8, 2022
ebe2485
merged main
jhwang98 Nov 8, 2022
c5145a6
Merge branch 'main' of https://github.com/facebook/jest into feat-ran…
jhwang98 Nov 14, 2022
4705449
merged main
jhwang98 Jan 12, 2023
af913e2
bumped pure-rand package
jhwang98 Jan 13, 2023
a6b4d06
Merge branch 'main' into feat-randomise-tests
jhwang98 Jan 25, 2023
e4f0b24
Merge branch 'main' into feat-randomise-tests
jhwang98 Feb 13, 2023
c4baaaa
yarn.lock updated
jhwang98 Feb 14, 2023
87547f3
fixed copyright headers
jhwang98 Feb 14, 2023
8bfa4c8
Merge branch 'main' into feat-randomise-tests
Joshua-Hwang Feb 16, 2023
0202230
Merge branch 'main' into feat-randomise-tests
SimenB Feb 23, 2023
3a0a4f1
Update packages/jest-config/src/normalize.ts
jhwang98 Feb 23, 2023
43d6f8b
Update packages/jest-core/src/cli/index.ts
jhwang98 Feb 23, 2023
7bd3fe1
Update packages/jest-circus/src/shuffleArray.ts
jhwang98 Feb 23, 2023
cff3995
remove bounds check as it's being done at the normalize level
jhwang98 Feb 23, 2023
7c8bfc1
changed error message slightly
jhwang98 Feb 23, 2023
c3c6516
swapped order for generating rng
jhwang98 Feb 23, 2023
d57bf1c
Update packages/jest-circus/src/run.ts
SimenB Feb 23, 2023
1137ce1
Update packages/jest-circus/src/run.ts
SimenB Feb 23, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
### Features

- `[jest-changed-files]` Support Sapling ([#13941](https://github.com/facebook/jest/pull/13941))
- `[jest-circus, @jest/cli, jest-config]` Add feature to randomize order of tests via CLI flag or through the config file([#12922](https://github.com/facebook/jest/pull/12922))
- `[jest-cli, jest-config, @jest/core, jest-haste-map, @jest/reporters, jest-runner, jest-runtime, @jest/types]` Add `workerThreads` configuration option to allow using [worker threads](https://nodejs.org/dist/latest/docs/api/worker_threads.html) for parallelization ([#13939](https://github.com/facebook/jest/pull/13939))
- `[jest-worker]` Add `start` method to worker farms ([#13937](https://github.com/facebook/jest/pull/13937))

Expand Down
16 changes: 16 additions & 0 deletions docs/CLI.md
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,22 @@ If configuration files are found in the specified paths, _all_ projects specifie

:::

### `--randomize`

Shuffle the order of the tests within a file. The shuffling is based on the seed. See [`--seed=<num>`](#--seednum) for more info.

Seed value is displayed when this option is set. Equivalent to setting the CLI option [`--showSeed`](#--showseed).

```bash
jest --randomize --seed 1234
```

:::note

This option is only supported using the default `jest-circus` test runner.

:::

### `--reporters`

Run tests with specified reporters. [Reporter options](configuration#reporters-arraymodulename--modulename-options) are not available via CLI. Example with multiple reporters:
Expand Down
6 changes: 6 additions & 0 deletions docs/Configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -1216,6 +1216,12 @@ With the `projects` option enabled, Jest will copy the root-level configuration

:::

### `randomize` \[boolean]

Default: `false`

The equivalent of the [`--randomize`](CLI.md#--randomize) flag to randomize the order of the tests in a file.

### `reporters` \[array&lt;moduleName | \[moduleName, options]&gt;]

Default: `undefined`
Expand Down
89 changes: 89 additions & 0 deletions e2e/__tests__/__snapshots__/randomize.test.ts.snap
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
// Jest Snapshot v1, https://goo.gl/fbAQLP

exports[`works with each 1`] = `
" ✓ test1
✓ test2
✓ test3
describe2
✓ test4
✓ test6
✓ test5
describe1
✓ test4
✓ test6
✓ test5
describe3
✓ test11
✓ test12
✓ test10
describe4
✓ test14
✓ test15
✓ test13"
`;

exports[`works with hooks 1`] = `
" ✓ test1
✓ test2
✓ test3
describe2
✓ test7
✓ test9
✓ test8
describe1
✓ test4
✓ test6
✓ test5
describe3
✓ test11
✓ test12
✓ test10
describe4
✓ test14
✓ test15
✓ test13"
`;

exports[`works with passing tests 1`] = `
" ✓ test1
✓ test2
✓ test3
describe2
✓ test7
✓ test9
✓ test8
describe1
✓ test4
✓ test6
✓ test5
describe3
✓ test11
✓ test12
✓ test10
describe4
✓ test14
✓ test15
✓ test13"
`;

exports[`works with snapshots 1`] = `
" ✓ test1
✓ test2
✓ test3
describe2
✓ test4
✓ test6
✓ test5
describe1
✓ test4
✓ test6
✓ test5
describe3
✓ test11
✓ test12
✓ test10
describe4
✓ test14
✓ test15
✓ test13"
`;
85 changes: 85 additions & 0 deletions e2e/__tests__/randomize.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
/**
* Copyright (c) Meta Platforms, Inc. and affiliates.
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree.
*/

import * as path from 'path';
SimenB marked this conversation as resolved.
Show resolved Hide resolved
import {skipSuiteOnJasmine} from '@jest/test-utils';
import {extractSummary} from '../Utils';
import runJest, {RunJestResult} from '../runJest';

skipSuiteOnJasmine();

const dir = path.resolve(__dirname, '../randomize');

const trimFirstLine = (str: string): string =>
str.split('\n').slice(1).join('\n');

function runJestTwice(
dir: string,
args: Array<string>,
): [RunJestResult, RunJestResult] {
return [
runJest(dir, [...args, '--randomize']),
runJest(dir, [...args, '--config', 'different-config.json']),
];
}

test('works with passing tests', () => {
const [result1, result2] = runJestTwice(dir, [
'success.test.js',
'--seed',
'123',
]);

const rest1 = trimFirstLine(extractSummary(result1.stderr).rest);
const rest2 = trimFirstLine(extractSummary(result2.stderr).rest);

expect(rest1).toEqual(rest2);
expect(rest1).toMatchSnapshot();
});

test('works with each', () => {
const [result1, result2] = runJestTwice(dir, [
'each.test.js',
'--seed',
'123',
]);

const rest1 = trimFirstLine(extractSummary(result1.stderr).rest);
const rest2 = trimFirstLine(extractSummary(result2.stderr).rest);

expect(rest1).toEqual(rest2);
expect(rest1).toMatchSnapshot();
});

test('works with hooks', () => {
const [result1, result2] = runJestTwice(dir, [
'hooks.test.js',
'--seed',
'123',
]);

// Change in formatting could change this one
const rest1 = trimFirstLine(extractSummary(result1.stderr).rest);
const rest2 = trimFirstLine(extractSummary(result2.stderr).rest);

expect(rest1).toEqual(rest2);
expect(rest1).toMatchSnapshot();
});

test('works with snapshots', () => {
const [result1, result2] = runJestTwice(dir, [
'snapshots.test.js',
'--seed',
'123',
]);

const rest1 = trimFirstLine(extractSummary(result1.stderr).rest);
const rest2 = trimFirstLine(extractSummary(result2.stderr).rest);

expect(rest1).toEqual(rest2);
expect(rest1).toMatchSnapshot();
});
50 changes: 26 additions & 24 deletions e2e/__tests__/showSeed.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,39 +14,41 @@ const dir = path.resolve(__dirname, '../jest-object');
const randomSeedValueRegExp = /Seed:\s+<<REPLACED>>/;
const seedValueRegExp = /Seed:\s+1234/;

test('--showSeed changes report to output seed', () => {
const {stderr} = runJest(dir, ['--showSeed', '--no-cache']);
describe.each(['showSeed', 'randomize'])('Option %s', option => {
test(`--${option} changes report to output seed`, () => {
const {stderr} = runJest(dir, [`--${option}`, '--no-cache']);

const {summary} = extractSummary(stderr);
const {summary} = extractSummary(stderr);

expect(replaceSeed(summary)).toMatch(randomSeedValueRegExp);
});
expect(replaceSeed(summary)).toMatch(randomSeedValueRegExp);
});

test('if --showSeed is not present the report will not show the seed', () => {
const {stderr} = runJest(dir, ['--seed', '1234']);
test(`if --${option} is not present the report will not show the seed`, () => {
const {stderr} = runJest(dir, ['--seed', '1234']);

const {summary} = extractSummary(stderr);
const {summary} = extractSummary(stderr);

expect(replaceSeed(summary)).not.toMatch(randomSeedValueRegExp);
});
expect(replaceSeed(summary)).not.toMatch(randomSeedValueRegExp);
});

test('if showSeed is present in the config the report will show the seed', () => {
const {stderr} = runJest(dir, [
'--seed',
'1234',
'--config',
'different-config.json',
]);
test(`if ${option} is present in the config the report will show the seed`, () => {
const {stderr} = runJest(dir, [
'--seed',
'1234',
'--config',
`${option}-config.json`,
]);

const {summary} = extractSummary(stderr);
const {summary} = extractSummary(stderr);

expect(summary).toMatch(seedValueRegExp);
});
expect(summary).toMatch(seedValueRegExp);
});

test('--seed --showSeed will show the seed in the report', () => {
const {stderr} = runJest(dir, ['--showSeed', '--seed', '1234']);
test(`--seed --${option} will show the seed in the report`, () => {
const {stderr} = runJest(dir, [`--${option}`, '--seed', '1234']);

const {summary} = extractSummary(stderr);
const {summary} = extractSummary(stderr);

expect(summary).toMatch(seedValueRegExp);
expect(summary).toMatch(seedValueRegExp);
});
});
4 changes: 4 additions & 0 deletions e2e/jest-object/randomize-config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"displayName": "Config from randomize-config.json file",
"randomize": true
}
4 changes: 4 additions & 0 deletions e2e/jest-object/showSeed-config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"displayName": "Config from showSeed-config.json file",
"showSeed": true
}
31 changes: 31 additions & 0 deletions e2e/randomize/__tests__/__snapshots__/snapshots.test.js.snap
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
// Jest Snapshot v1, https://goo.gl/fbAQLP

exports[`describe1 test4 1`] = `4`;

exports[`describe1 test5 1`] = `5`;

exports[`describe1 test6 1`] = `6`;

exports[`describe2 test4 1`] = `4`;

exports[`describe2 test5 1`] = `5`;

exports[`describe2 test6 1`] = `6`;

exports[`describe3 describe4 test13 1`] = `13`;

exports[`describe3 describe4 test14 1`] = `14`;

exports[`describe3 describe4 test15 1`] = `15`;

exports[`describe3 test10 1`] = `10`;

exports[`describe3 test11 1`] = `11`;

exports[`describe3 test12 1`] = `12`;

exports[`test1 1`] = `1`;

exports[`test2 1`] = `2`;

exports[`test3 1`] = `3`;
28 changes: 28 additions & 0 deletions e2e/randomize/__tests__/each.test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
/**
* Copyright (c) Meta Platforms, Inc. and affiliates.
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree.
*/

it.each([1, 2, 3])('test%d', () => {
expect(true).toBe(true);
});

describe.each([1, 2])('describe%d', () => {
it.each([4, 5, 6])('test%d', () => {
expect(true).toBe(true);
});
});

describe('describe3', () => {
it.each([10, 11, 12])('test%d', () => {
expect(true).toBe(true);
});

describe('describe4', () => {
it.each([13, 14, 15])('test%d', () => {
expect(true).toBe(true);
});
});
});
Loading