Tests: ensure version of Code in .vscode-test matches the one expected

[DanTup]

Currently if you have a .vscode-test folder locally, the test script will use the version inside it even if it's stale/doesn't match the CODE_VERSION.

This is likely a fringe case - people will run tests locally using the launch config and on Travis/AppVeyor they'll likely start with a clean filesystem.

However, if you might want to speed things up by reusing folders (eg. if you use Travis/AppVeyor caches to avoid re-downloading dependencies on all builds) then this fix would be required to avoid getting stuck running tests on an old version.

[gkalpak]

I just ran into this 😒 Quite unexpected behavior 😕

I assume it is not a high priority, since it has been open since March 😁
Maybe at least the docs could be updated to mention that you need to clear .vscode-test/ if you want CODE_VERSION to take effect.

That said, from a quick look at bin/test it looks like an easy fix (famous last words 😁).
I'd be happy to work on it, if this is something you are interested in getting fixed, @bpasero.

[gkalpak]

BTW, the same is true for CODE_DOWNLOAD_URL. It is only used if there is no version already present in .vscode-test/.

[gkalpak]

If anyone is interested, this is the script I currently use to handle testing against a specific version. It seems to work fine so far (on Windows, Linux, and macOS ✌️), but still use at your own rist 😁

_{(For more context on how I am using it, this is the whole commit: gkalpak/aio-docs-utils@c1ee381a4)}

[octref]

This is fixed by #148.

Now VS Code is downloaded to .vscode-test/vscode-1.32 when you run yarn test. When 1.33 is released, new versions will be downloaded to .vscode-test/vscode-1.33 and used for running tests.

For more granular control, use vscode-test directly instead of relying on vscode: https://github.com/microsoft/vscode-test.

[gkalpak]

Thx for working on this, @octref 👍
I spotted some issues and submitted fixes in #155 and microsoft/vscode-test#1.

[DanTup]

@octref I think all the code referencing testRunFolderAbsolute may be dead now and should be removed? If I'm reading the code correctly, this console.log looks suspicious:

function downloadExecutableAndRunTests() {
    console.log('Downloading VS Code into "' + testRunFolderAbsolute); // <-- 

    downloadAndUnzipVSCode().then(executablePath => {
        executable = executablePath
        runTests()
    })

AFAICT downloadAndUnzipVSCode does its own calculation of the download folder (including the version number) and testRunFolderAbsolute is the old path (ending in stable, or w/e)?

[DanTup]

Actually, maybe it's deliberate, since the "old" runTests method is still used:

fs.exists(executable, function (exists) {
    if (exists) {
        runTests();
    } else {
        downloadExecutableAndRunTests();
    }
});

It's not clear why, though I guess probably it's intended. I'll try switch over to using the package directly, as I already have a script for running my tests (but was shelling out to the old script). Hopefully this will simplify things :-)