Figure out why unit tests occasionally time out

[rally25rs]

CI builds often fail due to tests timing out. It also happens (very infrequently) when running tests locally.

Example output (this was from a local test run on a Macbook Pro, OSX Sierra, Node v6.3.0):

Summary of all failing tests
 FAIL  __tests__/commands/check.js (21.465s)
  ● --verify-tree should report wrong version

    Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.

      at tryOnTimeout (timers.js:224:11)
      at Timer.listOnTimeout (timers.js:198:5)

 FAIL  __tests__/commands/install/integration.js (184.171s)
  ● root install with optional deps

    Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.

      at tryOnTimeout (timers.js:224:11)
      at Timer.listOnTimeout (timers.js:198:5)

I'll try to dig into this as best I can, but I would appreciate some help... 🆘

[rally25rs]

From discussion in 3510 with @arcanis :

rally25rs: So all these tests are actually resolving real metadata? that seems fragile...

arcanis: Think so. That's why our tests practically never completely pass: we always end up timeouting at least one of them 😞 Probably should fix this at some point! I thought maybe we could set up some sort of fake registry with only the few endpoints required, that would extract its data from a repository local folder.

If the tests making web requests and hitting live metadata repos, this could be a cause for fragile tests in the future.

It would be good to make a plan to make the tests run as "offline" as possible.

Since Yarn already supports a fully offline install through use of the cache and "offline mirror" features, would it be feasible to load all packages and metadata used for tests into an offline mirror, and then always run tests in an "offline" mode?

[rally25rs]

Alternatively, could we just load the metadata package.json file from the repository into __tests__/fixtures/request-cache/GET/...?

[rally25rs]

@arcanis It looks like some of the tests don't hit real HTTP requests.

There is a mock requester in __tests__/__mocks__/request.js that first checks for a file in __tests__/fixtures/request-cache/GET/...

If the file exists, it just returns it.

If the file does not exist, then it does the actual HTTP request, and writes the response to the request-cache directory (where a dev could then check it in to Git)

The metadata is also saved here. For example for left-pad: https://github.com/yarnpkg/yarn/blob/master/__tests__/fixtures/request-cache/GET/registry.yarnpkg.com/left-pad.bin

So if a new version of left-pad is pushed out to the real online repository, it shouldn't break the tests because we use the cached response.

---

I said "some of the tests" because not all of them seem to actually use it.

If I run the command:

node_modules/.bin/jest tests/commands/upgrade.js

With a network connection:

Test Suites: 1 passed, 1 total
Tests:       20 passed, 20 total
Snapshots:   0 total
Time:        6.155s

Without a network connection:

Test Suites: 1 failed, 1 total
Tests:       4 failed, 16 passed, 20 total
Snapshots:   0 total
Time:        3.433s, estimated 9s

---

A good task for someone might be to run all the tests offline, see which ones fail, and see if we can wire in a mock requester for those too.

[cpojer]

@BYK this is resolved, right? #3667

[BYK]

Yup. Closing for now.