Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Removed a bunch of Test262 panics #906

Merged
merged 3 commits into from
Oct 21, 2020
Merged

Removed a bunch of Test262 panics #906

merged 3 commits into from
Oct 21, 2020

Conversation

Razican
Copy link
Member

@Razican Razican commented Oct 21, 2020

This changes the behaviour of the tester when the environment cannot be correctly set up. Instead of panicking, it just fails the test without the panic. I also removed some of the most notable panics (such as the instanceof), and improved the very verbose -vv output.

The number of panics should be greatly reduced with this.

@Razican Razican added enhancement New feature or request test Issues and PRs related to the tests. labels Oct 21, 2020
@Razican Razican added this to the v0.11.0 milestone Oct 21, 2020
@github-actions
Copy link

github-actions bot commented Oct 21, 2020 •
edited
Loading

Test262 conformance changes:

Test result master count PR count difference
Total 78,415 78,415 0
Passed 18,406 18,461 +55
Ignored 15,507 15,547 +40
Failed 44,502 44,407 -95
Panics 11,221 1,129 -10,092
Conformance 23.47 23.54 +0.07%

@codecov
Copy link

codecov bot commented Oct 21, 2020 •
edited
Loading

Codecov Report

Merging #906 into master will decrease coverage by 0.16%.
The diff coverage is 0.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #906      +/-   ##
==========================================
- Coverage   59.36%   59.19%   -0.17%     
==========================================
  Files         166      166              
  Lines       10486    10512      +26     
==========================================
- Hits         6225     6223       -2     
- Misses       4261     4289      +28
Impacted Files Coverage Δ
boa/src/syntax/ast/node/object/mod.rs 37.14% <ø> (ø)
boa/src/syntax/ast/node/operator/bin_op/mod.rs 71.29% <0.00%> (ø)
boa_tester/src/exec.rs 0.00% <0.00%> (ø)
boa_tester/src/read.rs 0.00% <ø> (ø)
boa/src/builtins/regexp/mod.rs 70.12% <0.00%> (-0.84%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3a293b...4475b92. Read the comment docs.

@github-actions
Copy link

Benchmark for 1a069d6

Click to view benchmark
Test PR Benchmark Master Benchmark %
Arithmetic operations (Execution) 369.8±11.62ns 364.7±1.09ns +1.40%
Arithmetic operations (Full) 241.4±0.34µs 237.9±0.58µs +1.47%
Array access (Execution) 8.3±0.42µs 8.1±0.04µs +2.47%
Array access (Full) 274.7±18.14µs 274.3±10.76µs +0.15%
Array creation (Execution) 2.9±0.01ms 2.8±0.00ms +3.57%
Array creation (Full) 3.1±0.02ms 3.1±0.00ms 0.00%
Array pop (Execution) 1095.8±21.23µs 1034.7±3.75µs +5.91%
Array pop (Full) 1495.1±2.92µs 1517.0±15.93µs -1.44%
Boolean Object Access (Execution) 4.8±0.11µs 4.8±0.01µs 0.00%
Boolean Object Access (Full) 254.9±0.69µs 251.8±0.57µs +1.23%
Clean js (Execution) 737.7±14.92µs 714.9±4.86µs +3.19%
Clean js (Full) 1019.8±8.04µs 1012.2±9.57µs +0.75%
Clean js (Parser) 36.7±0.03µs 36.8±0.03µs -0.27%
Create Realm 483.5±41.57ns 473.2±15.17ns +2.18%
Dynamic Object Property Access (Execution) 5.7±0.02µs 5.8±0.23µs -1.72%
Dynamic Object Property Access (Full) 260.8±7.45µs 276.0±26.85µs -5.51%
Expression (Parser) 6.9±0.01µs 7.0±0.14µs -1.43%
Fibonacci (Execution) 843.9±2.43µs 869.5±19.97µs -2.94%
Fibonacci (Full) 1117.2±3.17µs 1170.0±71.97µs -4.51%
For loop (Execution) 22.4±0.04µs 22.6±0.13µs -0.88%
For loop (Full) 281.2±0.61µs 279.9±0.49µs +0.46%
For loop (Parser) 18.5±0.03µs 18.5±0.03µs 0.00%
Goal Symbols (Parser) 12.5±0.02µs 13.0±1.20µs -3.85%
Hello World (Parser) 3.4±0.01µs 3.4±0.01µs 0.00%
Long file (Parser) 765.6±1.13ns 763.8±1.31ns +0.24%
Mini js (Execution) 649.3±3.74µs 648.3±6.30µs +0.15%
Mini js (Full) 925.6±3.00µs 918.0±3.43µs +0.83%
Mini js (Parser) 32.0±0.04µs 31.9±0.03µs +0.31%
Number Object Access (Execution) 3.7±0.01µs 3.7±0.01µs 0.00%
Number Object Access (Full) 251.0±0.66µs 247.8±0.40µs +1.29%
Object Creation (Execution) 5.0±0.18µs 5.0±0.14µs 0.00%
Object Creation (Full) 251.3±0.37µs 252.6±0.53µs -0.51%
RegExp (Execution) 10.9±0.07µs 10.9±0.06µs 0.00%
RegExp (Full) 262.1±1.32µs 263.8±0.49µs -0.64%
RegExp Literal (Execution) 10.9±0.06µs 10.9±0.07µs 0.00%
RegExp Literal (Full) 274.8±14.44µs 259.6±1.24µs +5.86%
RegExp Literal Creation (Execution) 9.9±0.42µs 9.7±0.03µs +2.06%
RegExp Literal Creation (Full) 256.4±0.43µs 256.9±2.04µs -0.19%
Static Object Property Access (Execution) 5.2±0.15µs 5.1±0.01µs +1.96%
Static Object Property Access (Full) 252.3±0.50µs 250.6±0.54µs +0.68%
String Object Access (Execution) 7.1±0.02µs 7.2±0.03µs -1.39%
String Object Access (Full) 258.4±1.39µs 258.3±0.56µs +0.04%
String comparison (Execution) 6.5±0.03µs 6.8±0.25µs -4.41%
String comparison (Full) 258.0±0.37µs 257.9±0.62µs +0.04%
String concatenation (Execution) 5.4±0.16µs 5.3±0.02µs +1.89%
String concatenation (Full) 257.8±5.66µs 251.0±1.24µs +2.71%
String copy (Execution) 4.1±0.01µs 4.1±0.02µs 0.00%
String copy (Full) 246.6±0.42µs 246.8±0.30µs -0.08%
Symbols (Execution) 3.5±0.01µs 3.6±0.12µs -2.78%
Symbols (Full) 240.9±3.15µs 249.0±8.30µs -3.25%

@github-actions
Copy link

Benchmark for 8e8e843

Click to view benchmark
Test PR Benchmark Master Benchmark %
Arithmetic operations (Execution) 344.0±10.41ns 349.2±9.42ns -1.49%
Arithmetic operations (Full) 229.0±5.78µs 224.7±7.79µs +1.91%
Array access (Execution) 7.7±0.22µs 7.8±0.19µs -1.28%
Array access (Full) 251.7±8.06µs 252.0±5.16µs -0.12%
Array creation (Execution) 2.7±0.09ms 2.6±0.07ms +3.85%
Array creation (Full) 2.9±0.06ms 3.0±0.07ms -3.33%
Array pop (Execution) 1030.7±25.33µs 982.6±27.37µs +4.90%
Array pop (Full) 1422.7±38.21µs 1440.8±42.44µs -1.26%
Boolean Object Access (Execution) 4.5±0.14µs 4.5±0.12µs 0.00%
Boolean Object Access (Full) 244.5±5.43µs 242.8±5.74µs +0.70%
Clean js (Execution) 691.3±16.88µs 684.1±18.05µs +1.05%
Clean js (Full) 980.4±21.33µs 962.3±22.95µs +1.88%
Clean js (Parser) 35.3±1.36µs 36.0±0.65µs -1.94%
Create Realm 463.6±14.24ns 452.6±12.53ns +2.43%
Dynamic Object Property Access (Execution) 5.4±0.13µs 5.3±0.19µs +1.89%
Dynamic Object Property Access (Full) 247.5±4.66µs 243.4±7.10µs +1.68%
Expression (Parser) 6.7±0.19µs 6.7±0.14µs 0.00%
Fibonacci (Execution) 806.0±19.35µs 795.3±19.81µs +1.35%
Fibonacci (Full) 1055.5±34.23µs 1064.3±30.42µs -0.83%
For loop (Execution) 21.6±0.44µs 21.3±0.56µs +1.41%
For loop (Full) 266.4±7.37µs 268.6±7.43µs -0.82%
For loop (Parser) 18.2±0.24µs 17.4±0.45µs +4.60%
Goal Symbols (Parser) 11.8±0.36µs 12.0±0.38µs -1.67%
Hello World (Parser) 3.3±0.04µs 3.2±0.07µs +3.12%
Long file (Parser) 745.7±16.59ns 732.9±19.86ns +1.75%
Mini js (Execution) 632.2±11.19µs 614.7±15.87µs +2.85%
Mini js (Full) 888.2±17.99µs 870.5±20.23µs +2.03%
Mini js (Parser) 31.2±0.71µs 30.3±0.88µs +2.97%
Number Object Access (Execution) 3.5±0.12µs 3.6±0.07µs -2.78%
Number Object Access (Full) 237.9±5.15µs 235.2±7.23µs +1.15%
Object Creation (Execution) 4.7±0.13µs 4.6±0.15µs +2.17%
Object Creation (Full) 240.9±5.51µs 243.1±5.94µs -0.90%
RegExp (Execution) 10.3±0.27µs 10.3±0.29µs 0.00%
RegExp (Full) 256.7±6.48µs 250.1±8.34µs +2.64%
RegExp Literal (Execution) 10.2±0.34µs 10.5±0.19µs -2.86%
RegExp Literal (Full) 248.6±6.89µs 250.4±7.20µs -0.72%
RegExp Literal Creation (Execution) 9.3±0.29µs 9.1±0.28µs +2.20%
RegExp Literal Creation (Full) 242.5±6.07µs 243.9±8.18µs -0.57%
Static Object Property Access (Execution) 4.8±0.16µs 5.0±0.10µs -4.00%
Static Object Property Access (Full) 241.8±6.59µs 241.6±7.90µs +0.08%
String Object Access (Execution) 6.9±0.14µs 6.8±0.19µs +1.47%
String Object Access (Full) 244.4±7.28µs 250.4±5.29µs -2.40%
String comparison (Execution) 6.2±0.16µs 6.4±0.12µs -3.13%
String comparison (Full) 246.1±7.02µs 248.2±5.45µs -0.85%
String concatenation (Execution) 5.1±0.16µs 5.0±0.12µs +2.00%
String concatenation (Full) 240.2±6.20µs 236.1±6.37µs +1.74%
String copy (Execution) 3.9±0.09µs 3.9±0.11µs 0.00%
String copy (Full) 235.3±7.65µs 233.3±5.32µs +0.86%
Symbols (Execution) 3.3±0.09µs 3.3±0.09µs 0.00%
Symbols (Full) 232.7±5.46µs 228.3±6.11µs +1.93%

@Razican Razican mentioned this pull request Oct 21, 2020
Closed
RageKnify
RageKnify approved these changes Oct 21, 2020
View reviewed changes
Copy link
Contributor

@RageKnify RageKnify left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I found a forgotten dbg!(), the rest looks good to me.

@github-actions
Copy link

Benchmark for c19bf94

Click to view benchmark
Test PR Benchmark Master Benchmark %
Arithmetic operations (Execution) 359.2±2.57ns 362.3±1.53ns -0.86%
Arithmetic operations (Full) 240.4±1.22µs 234.8±0.36µs +2.39%
Array access (Execution) 8.0±0.05µs 8.1±0.04µs -1.23%
Array access (Full) 261.6±1.69µs 256.7±2.72µs +1.91%
Array creation (Execution) 2.8±0.03ms 2.8±0.03ms 0.00%
Array creation (Full) 3.1±0.02ms 3.0±0.09ms +3.33%
Array pop (Execution) 1059.7±9.00µs 1059.7±5.29µs 0.00%
Array pop (Full) 1542.5±47.48µs 1484.1±2.28µs +3.94%
Boolean Object Access (Execution) 4.7±0.01µs 4.7±0.01µs 0.00%
Boolean Object Access (Full) 254.3±0.75µs 249.0±1.29µs +2.13%
Clean js (Execution) 713.4±4.93µs 705.2±8.93µs +1.16%
Clean js (Full) 1025.2±5.38µs 1004.7±9.28µs +2.04%
Clean js (Parser) 37.1±0.14µs 37.1±0.10µs 0.00%
Create Realm 468.7±0.31ns 467.5±0.36ns +0.26%
Dynamic Object Property Access (Execution) 5.5±0.12µs 5.6±0.01µs -1.79%
Dynamic Object Property Access (Full) 252.5±2.17µs 252.4±4.33µs +0.04%
Expression (Parser) 6.9±0.02µs 6.9±0.01µs 0.00%
Fibonacci (Execution) 840.3±4.91µs 844.1±3.76µs -0.45%
Fibonacci (Full) 1106.4±11.21µs 1116.9±2.92µs -0.94%
For loop (Execution) 22.7±0.02µs 22.3±0.15µs +1.79%
For loop (Full) 280.0±1.01µs 276.6±0.52µs +1.23%
For loop (Parser) 18.4±0.03µs 18.3±0.22µs +0.55%
Goal Symbols (Parser) 12.7±0.05µs 12.6±0.04µs +0.79%
Hello World (Parser) 3.4±0.03µs 3.3±0.02µs +3.03%
Long file (Parser) 757.7±1.16ns 765.6±3.61ns -1.03%
Mini js (Execution) 641.3±4.32µs 638.4±4.62µs +0.45%
Mini js (Full) 929.2±4.10µs 903.2±12.85µs +2.88%
Mini js (Parser) 32.5±0.03µs 32.5±0.07µs 0.00%
Number Object Access (Execution) 3.7±0.02µs 3.8±0.01µs -2.63%
Number Object Access (Full) 250.2±0.72µs 242.9±2.53µs +3.01%
Object Creation (Execution) 4.8±0.05µs 4.8±0.01µs 0.00%
Object Creation (Full) 250.8±1.55µs 248.8±1.59µs +0.80%
RegExp (Execution) 10.5±0.06µs 10.9±0.07µs -3.67%
RegExp (Full) 264.7±0.70µs 261.2±2.17µs +1.34%
RegExp Literal (Execution) 10.7±0.05µs 10.9±0.07µs -1.83%
RegExp Literal (Full) 261.7±1.15µs 258.8±0.66µs +1.12%
RegExp Literal Creation (Execution) 9.4±0.07µs 9.7±0.03µs -3.09%
RegExp Literal Creation (Full) 255.9±0.64µs 249.7±3.40µs +2.48%
Static Object Property Access (Execution) 5.0±0.05µs 5.1±0.02µs -1.96%
Static Object Property Access (Full) 252.3±1.14µs 250.0±0.43µs +0.92%
String Object Access (Execution) 7.1±0.03µs 7.2±0.07µs -1.39%
String Object Access (Full) 259.1±0.58µs 253.5±2.00µs +2.21%
String comparison (Execution) 6.6±0.02µs 6.5±0.29µs +1.54%
String comparison (Full) 257.9±0.46µs 252.2±3.28µs +2.26%
String concatenation (Execution) 5.3±0.02µs 5.3±0.02µs 0.00%
String concatenation (Full) 251.4±0.45µs 247.0±0.42µs +1.78%
String copy (Execution) 4.0±0.01µs 4.1±0.01µs -2.44%
String copy (Full) 245.4±0.47µs 242.0±1.61µs +1.40%
Symbols (Execution) 3.4±0.00µs 3.5±0.01µs -2.86%
Symbols (Full) 240.3±0.44µs 235.1±2.55µs +2.21%

@Lan2u Lan2u merged commit 9fa49d7 into master Oct 21, 2020
@HalidOdat HalidOdat deleted the tester_panics branch October 22, 2020 01:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HalidOdat HalidOdat HalidOdat approved these changes

@Lan2u Lan2u Lan2u approved these changes

@RageKnify RageKnify RageKnify approved these changes

@jasonwilliams jasonwilliams Awaiting requested review from jasonwilliams

Assignees
No one assigned
Labels
enhancement New feature or request test Issues and PRs related to the tests.
Projects
None yet
Milestone
v0.11.0
Development

Successfully merging this pull request may close these issues.
4 participants
@Razican @HalidOdat @Lan2u @RageKnify