Enable WebIDL harness tests for reference implementation

[MattiasBuelens]

As part of #1035, we added WebIDL harness tests to the web platform tests. However, we currently skip them for the reference implementation because they do not work with wpt-runner. This means that bugs in the WebIDL types (or in the WebIDL tests themselves) would only be uncovered when a browser runs the tests against their implementation (which happens much later). And even then, it would not be clear if the browser or the reference implementation got it wrong...

I would like to see the reference implementations run (and pass) the IDL tests, so we can catch bugs in the specification or the tests before browsers start implementing.

[MattiasBuelens]

The good news: I already managed to get the tests to run locally, and I already found some missing things and broken things in the tests:

• Support async iterable declarations in idlharness web-platform-tests/wpt#24266
• Streams: fix IDL tests web-platform-tests/wpt#24267

Now, I'd like to clean up and upstream my local patches. But first, I'd like your input on my suggested approach before I start making more pull requests. 😁

In essence, there are two things missing to get the tests to run:

1. wpt-runner needs to serve /resources/idlharness.js and /resources/WebIDLParser.js (which is actually rewritten to /resources/webidl2/lib/webidl2.js) from web-platform-tests.
2. idlharness.js needs to fetch WebIDL files from web-platform-tests/interfaces.

For (1), wpt-runner would need to either include its own copy of these files, or would need to know where to find them. The first option seems like a bad idea, since we'd have to regular make new releases just to stay up-to-date with WPT. For the second option, we could add an option where the user can define a list of URLs and file paths that the runner needs to serve. Then, the streams spec would pass in the URLs and paths of the WebIDL harness files:

const failures = await wptRunner(testsPath, {
  rootURL: 'streams/',
  server: {
    // map of URL to file path
    "/resources/idlharness.js": path.resolve(__dirname, "web-platform-tests/resources/idlharness.js"),
    "/resources/WebIDLParser.js": path.resolve(__dirname, "web-platform-tests/resources/webidl2/lib/webidl2.js")
  },
  // ...
});

For (2), unfortunately jsdom does not yet support fetch out-of-the-box (jsdom/jsdom#1724). However, we don't need a full-fledged (...full-fetched?) fetch implementation: we only need to support what idlharness's fetch_spec actually uses. So I think the simplest solution would be to inject a (very basic) fetch shim that only serves text files:

const readFileAsync = promisify(fs.readFile);
const wptPath = path.resolve(__dirname, 'web-platform-tests');

const failures = await wptRunner(testsPath, {
  // ...
  setup(window) {
    window.queueMicrotask = queueMicrotask;
    window.fetch = async function (url) {
      const filePath = path.join(wptPath, url);
      if (!filePath.startsWith(wptPath)) {
        throw new TypeError('Invalid URL');
      }
      return {
        ok: true,
        async text() {
          return await readFileAsync(filePath, { encoding: 'utf8' });
        }
      };
    };
    window.eval(bundledJS);
  },
}

And that's it: with those things in place, plus the fixes to the web platform tests themselves, all IDL tests run smoothly.

So, what do you think? 🙂

[domenic]

Ah, this is excellent; thank you so much for working on this!

For (1), my concern is that wpt-runner currently uses its own copies of testharness.js and testdriver-dummy.js. Indeed, that does require rolling the submodule, which is annoying, but I think we should stay consistent. So I would suggest for now that we continue to use the submodule's copy, but I am also open to the idea of revving wpt-runner to no longer use the submodule and instead require a "path to WPT root" where it can pull the files from.

(That also might be a good time to change the path configuration in general; I'm not so happy about the "mount a subpath" behavior.)

For (2), I agree a fake fetch is the way to go, in pretty much exactly the way you've outlined.

[MattiasBuelens]

Right, I forgot there is a wpt submodule inside the wpt-runner repository. I was just hacking on the wpt-runner code directly inside my node_modules directory. 😅

but I am also open to the idea of revving wpt-runner to no longer use the submodule and instead require a "path to WPT root" where it can pull the files from.

(That also might be a good time to change the path configuration in general; I'm not so happy about the "mount a subpath" behavior.)

+1 to both suggestions. But at least for now, I'll just copy the files. 🙂