update test262 +test262.properties file (re)generator (#930)

Cleaning up existing test262.properties file for larger changes to come, removing wildcard matching from the 262Test runner - Removed wildcard matching: now need to be explicit: 'name.js' wont match any file in any subdirectory that ends with `name.js` anymore - added all files with full path for current "wildcard" entries and removed wildcard entries (xxxx.js, matching **/*xxxx.js) - removed exclusions of test files that depend on unsupported features (are filtered out/skipped anyway when running the tests) - made sure test files within a directory are listed in a certain order (as to be able to better compare the current test262.properties with a generated one later on) - marked folders containing only failing tests as excluded (instead of listing all failing files individually) - removed some comments - removed unneeded ! in front of every .js file: every .js file is an exclusion regardless and by removing the !, the filePath becomes a key within the Java Properties format - use ~ instead of # for excluding directories - no need to exclude files under an already excluded directory - add generateTest262properties commandline flag to regenerate the used .properties file based on the test results of running the Test262SuiteTest - regeneration of test262.properties with default regeneration - change default to include tests that are skipped due to relying on unsupported features
mozilla · Jun 23, 2021 · 263655b · 263655b
1 parent da22122
commit 263655b
Show file tree

Hide file tree

Showing 6 changed files with 7,005 additions and 5,616 deletions.
diff --git a/build.gradle b/build.gradle
@@ -87,6 +87,7 @@ test {
         systemProperty 'TEST_OPTLEVEL', System.getProperty('optLevel')
     }
     systemProperty 'test262properties', System.getProperty('test262properties')
+    systemProperty 'updateTest262properties', System.getProperty('updateTest262properties')
     maxHeapSize = "1g"
     testLogging.showStandardStreams = true
     // Many tests do not clean up contexts properly. This makes the tests much

diff --git a/test262 b/test262
diff --git a/testsrc/README.md b/testsrc/README.md
@@ -16,6 +16,9 @@ git submodule update
 
 After doing so, the `./gradlew test` command will also execute all tests that are part of the official ECMAScript Test Suite
 
+As Rhino isn't 100% compliant with the latest ECMAScript standard, there is a mechanism to define which tests to run/skip,
+through the [test262.properties](test262.properties) file, the format of which is discussed in the [test262.properties format](#test262.properties-format) section
+
 ## Optimization levels
 By default all tests are run 3 times, at optimization levels -1, 0 and 9.
 
@@ -32,14 +35,74 @@ This behavior can be changed through different means:
 
 ## Running a specific TestSuite
 ```
-./gradlew test --tests org.mozilla.javascript.tests.Test262SuiteTest 
+./gradlew test --tests org.mozilla.javascript.tests.Test262SuiteTest
+```
+
+## test262.properties format
+The [test262.properties](test262.properties) file is used to specify which tests from the official ECMAScript Test Suite to include/exclude,
+so that the test suite can pass even though Rhino is not yet 100% standards compliant
+
+The [test262.properties](test262.properties) file:
+- lists the subfolders of the [test262](../test262) folder to include or exclude when running tests
+- lists the .js test files that are expected to fail per (sub)folder
+
+**Basics**
+```
+built-ins/Array <- include all the tests in the test262/built-ins/Array folder
+    from/calling-from-valid-1-noStrict.js <- but expect that this tests fails
+```
+If `built-ins/Array/from/calling-from-valid-1-noStrict.js` indeed fails, it wont fail the test suite. If it passes, this will be logged while running the test suite
+
+**Skipping entire folders**
+```
+~built-ins/decodeURI
+    name.js
+    S15.1.3.1_A2.4_T1.js
+    S15.1.3.1_A5.2.js
+```
+Toplevel folders can be prefixed with a `~` to mark the folder to be skipped entirely, for example because it contains all tests for a feature not yet supported by Rhino or because running the tests take a long time.
+Any files listed for a skipped folder will be skipped as well.
+
+**Expecting all files in a (sub)folder to fail**
+```
+built-ins/Array <-- topLevel folder
+    prototype/flatMap <-- subfolder under topLevel folder
 ```
+If all files in a subfolder below a topLevel folderare expected to fail, instead of listing all files explicitly, just the path of the folder needs to be included under the topLevel folder
+
+**Comments**
+The test262.properties file uses the Java Properties format, with the folder/.js file paths being the property key. The value of each 'property' can be used to store a comment
+```
+~built-ins/Array All tests on the built-in Array class
+    prototype/flatMap haven't gotten around to implementing flatMap yet
+```
+
+A Java Properties file can also have entire lines as comments, by prefixing the line with either `!` or `#`.
+While the test262.properties file does support this (because it is a Java Properties file), such line comments will be lost when (re)generating the test262.properties file!
+
+## Updating the test262.properties file
+While the [test262.properties](test262.properties) file could be manually updated, the tooling also comes with a mechanism to (re)generate the file based on the current revision of the test262 submodule and the results of running Test262SuiteTest against this revision on all standard optLevels (-1, 0 & 9)
+
+```
+./gradlew test --tests org.mozilla.javascript.tests.Test262SuiteTest --rerun-tasks -DupdateTest262properties [-Dtest262properties=testsrc/myOwn.properties]
+```
+The .properties file generation can be parameterized to affect the output:
+- rollup: include only a single line for a subfolder that contains only failing tests
+- stats: include stats on the # of failed and total tests
+- unsupported: include files containing tests for unsupported features
+
+These defaults can be overridden by specifying a value for the `generateTest262properties` parameter:
+- all: rollup, stats and unsupported all true (default)
+- none: rollup, stats and unsupported all false
+- [rollup][stats][unsupported]: the ones included will be true
+
+Note: the tests must actually run for the .properties file to be updated. If gradle determines that nothing has changed since the last time the `test` command was run, it won't run the tests. The `--rerun-tasks` argument forces gradle to run all tests 
 
 ## Running specific tests from the official ECMAScript Test Suite (test262)
-As Rhino isn't 100% compliant with the latest ECMAScript standard, there is a mechanism to define which tests to run/skip. 
-The default is [test262.properties](test262.properties).
+The default setup for running the test262 test suite is defined in [test262.properties](test262.properties).
 
 Another .properties file to use can be specified using the `test262properties` commandline property
 ```
 ./gradlew test --tests org.mozilla.javascript.tests.Test262SuiteTest -Dtest262properties=testsrc/myOwn.properties
-```
+```
+This allows the creation of a custom .properties file containing for example just the tests for one specific feature being implemented, which allows for (quickly) running just the tests for that specific feature
diff --git a/testsrc/org/mozilla/javascript/drivers/TestUtils.java b/testsrc/org/mozilla/javascript/drivers/TestUtils.java
@@ -13,7 +13,6 @@
 import java.util.Arrays;
 import java.util.List;
 import java.util.Properties;
-
 import org.mozilla.javascript.ContextFactory;
 
 public class TestUtils {
@@ -31,65 +30,60 @@ public static void setGlobalContextFactory(ContextFactory factory) {
     }
 
     public static File[] recursiveListFiles(File dir, FileFilter filter) {
-        if (!dir.isDirectory())
-            throw new IllegalArgumentException(dir + " is not a directory");
+        if (!dir.isDirectory()) throw new IllegalArgumentException(dir + " is not a directory");
         List<File> fileList = new ArrayList<File>();
         recursiveListFilesHelper(dir, filter, fileList);
         return fileList.toArray(new File[fileList.size()]);
     }
 
-    public static void recursiveListFilesHelper(File dir, FileFilter filter,
-                                                List<File> fileList)
-    {
-        for (File f: dir.listFiles()) {
+    public static void recursiveListFilesHelper(File dir, FileFilter filter, List<File> fileList) {
+        for (File f : dir.listFiles()) {
             if (f.isDirectory()) {
                 recursiveListFilesHelper(f, filter, fileList);
             } else {
-                if (filter.accept(f))
-                    fileList.add(f);
+                if (filter.accept(f)) fileList.add(f);
             }
         }
     }
 
-    public static void addTestsFromFile(String filename, List<String> list)
-            throws IOException {
+    public static void addTestsFromFile(String filename, List<String> list) throws IOException {
         addTestsFromStream(new FileInputStream(new File(filename)), list);
     }
 
-    public static void addTestsFromStream(InputStream in, List<String> list)
-            throws IOException {
+    public static void addTestsFromStream(InputStream in, List<String> list) throws IOException {
         Properties props = new Properties();
         props.load(in);
-        for (Object obj: props.keySet()) {
+        for (Object obj : props.keySet()) {
             list.add(obj.toString());
         }
     }
 
     public static String[] loadTestsFromResource(String resource, String[] inherited)
             throws IOException {
-        List<String> list = inherited == null ?
-                new ArrayList<String>() :
-                new ArrayList<String>(Arrays.asList(inherited));
+        List<String> list =
+                inherited == null
+                        ? new ArrayList<String>()
+                        : new ArrayList<String>(Arrays.asList(inherited));
         InputStream in = JsTestsBase.class.getResourceAsStream(resource);
-        if (in != null)
-            addTestsFromStream(in, list);
+        if (in != null) addTestsFromStream(in, list);
         return list.toArray(new String[0]);
     }
 
     public static boolean matches(String[] patterns, String path) {
-        for (int i=0; i<patterns.length; i++) {
+        for (int i = 0; i < patterns.length; i++) {
             if (path.startsWith(patterns[i])) {
                 return true;
             }
         }
         return false;
     }
 
-    public static final FileFilter JS_FILE_FILTER = new FileFilter() {
-        @Override
-        public boolean accept(File pathname) {
-            return pathname.getAbsolutePath().endsWith(".js");
-        }
-    };
-
+    public static final FileFilter JS_FILE_FILTER =
+            new FileFilter() {
+                @Override
+                public boolean accept(File pathname) {
+                    String path = pathname.getAbsolutePath();
+                    return path.endsWith(".js") && !path.endsWith("_FIXTURE.js");
+                }
+            };
 }