Watchman integration tests fail (sockets, symlinks, more)

[zeh]

Building watchman works well on Windows 10 with bash. watchman itself seems to run fine for the most part too, but I get several failures when running its test suite with make integration.

While watchman already has a native Window version, and while some of the errors might be due to my setup, I think it'd be important to look at the results of this integration test as a) some macOS/linux-specific build workflows depend on watchman, and b) I assume it works well as a smoke test for other missing features on WSL. My case is the former; I have one long build/dev workflow that would be great to run from Win 10, but that fails when firing up watchman.

Steps to reproduce

Start by downloading and compiling:

git clone https://github.com/facebook/watchman.git
cd watchman
./autogen.sh
./configure
make

Run integration tests:

make integration

Expected results

Passing with 0 errors. Was able to get this working on a macOS environment with the same commands, but I couldn't test on a real linux environment to see how it compares.

Actual results

The biggest issues in the error log seem to be socket- or symlink-related:

�[31mFAIL�[0m test_sock_perms.TestSockPerms.test_sock_access_group_change (1.024s)
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/mnt/d/xdiffx/watchman/tests/integration/test_sock_perms.py", line 235, in test_sock_access_group_change
    instance.start()
  ...
SocketConnectError: unable to connect to /tmp/watchmantestpRu0U8/inst8iuxgB/zeh-state/sock: [Errno 2] No such file or directory

...

�[31mFAIL�[0m test_sock_perms.TestSockPerms.test_custom_sock_group (1.022s)
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/mnt/d/xdiffx/watchman/tests/integration/test_sock_perms.py", line 135, in test_custom_sock_group
    instance.start()
  ...
SocketConnectError: unable to connect to /tmp/watchmantestpRu0U8/instWyRD0B/zeh-state/sock: [Errno 2] No such file or directory

...

�[31mFAIL�[0m test_symlink.TestSymlinkCliJson.test_delSymlink (0.033s)
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/mnt/d/xdiffx/watchman/tests/integration/test_symlink.py", line 92, in test_delSymlink
    self.assertFileList(root, ['111', '222' ])
  ...
WatchmanEnvironmentError: I/O error communicating with watchman daemon: errno=3 errmsg=No such process, while executing ('query', '/tmp/watchmantestpRu0U8/test_symlink.TestSymlinkCliJson.test_delSymlink.cli.json/tmpyKwSRp', {'fields': ['name'], 'expression': ['exists']})

...

�[31mFAIL�[0m test_symlink.TestSymlinkCliJson.test_watchSymlinkTargetLinkToLink (0.057s)
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/mnt/d/xdiffx/watchman/tests/integration/test_symlink.py", line 190, in test_watchSymlinkTargetLinkToLink
    self.assertWatchListContains(expected)
  ...
WatchmanEnvironmentError: I/O error communicating with watchman daemon: errno=3 errmsg=No such process, while executing ('watch-list',)

The full log is here, or here when running with strace and sudo.

Versions used

• Windows 10 (Insider Fast Ring) 15007
• Python 2.7.12
• Watchman source 4.8.0

Required packages

Install via apt-get:

• python
• automake
• autoconf
• libpcre (pcre)
• nodejs

[sunilmut]

@zeh - Thanks for reporting this. For the socket related failures, it looks like the test tries to connect to some unix socket /tmp/watchmantestpRu0U8/inst8iuxgB/zeh-state/sock. Do you know who creates that socket? Is it the watchman daemon? If so, you may have to start that manually, because of some lack of support for daemons in WSL currently.

[zeh]

@sunilmut I believe the tests themselves should be trying to create that: it creates them, connects to validate, then disconnects. But we do get this error message at the very beginning of the test...

WatchmanEnvironmentError: I/O error communicating with watchman daemon: errno=3 errmsg=No such process, while executing ('watch-project', '/tmp/watchmantestpRu0U8/test_watch_project.TestWatchProjectCliJson.test_watchProject.cli.json/tmp7rFwMC-aZbZ.hg-aZb-c-True/a/b/c')

...so you're probably right on the money, sockets are not available probably because the daemon is not around.

I'll follow the daemon-related issues (like this one) and re-test in the future as appropriate if anything gets committed and released.

[zeh]

(I'll be updating this issue with results for new releases unless someone minds).

So, just for reference, tested with 15014. Comparing the output of different versions is messy because the test order doesn't seem to be deterministic, so diff'ing between the 15007 and 15014 logs is kind of useless. The end result seems to be the same though: 248 PASS, 49 FAIL, 9 SKIP (or Ran 258, failed 9, skipped 7, concurrency 4). I would say that is expected, since 15014 doesn't implement any changes to daemon or symlink support, the biggest failure vectors in the integration test suite.

Edit: the number of pass/fails seems to be a bit random even in successive runs too. Strange tests, these are. Still, I'll be posting updates below.

• Jan 27th, 15019: 230 PASS, 43 FAIL, 33 SKIP / Ran 258, failed 4, skipped 29, concurrency 4
• Feb 2nd, 15025: 230 PASS, 43 FAIL, 33 SKIP / Ran 258, failed 4, skipped 29, concurrency 4

From this point on, running non-root with concurrency and test shuffling turned off, so logs can be more determistic and diffing works:

• Feb 10th, 15031: 226 PASS, 50 FAIL, 39 SKIP / Ran 265, failed 8, skipped 37, concurrency 1
• Feb 27th, 15042: 226 PASS, 50 FAIL, 39 SKIP / Ran 265, failed 8, skipped 37, concurrency 1 (===)
• Mar 2nd, 15046: 226 PASS, 50 FAIL, 39 SKIP / Ran 265, failed 8, skipped 37, concurrency 1 (===)
• Mar 6th, 15048: 226 PASS, 50 FAIL, 39 SKIP / Ran 265, failed 8, skipped 37, concurrency 1 (===)

[therealkenc]

Similar to #2526. Paraphrasing:

Going to snipe this one since no response for two seventeen months+. The lack of response was not because no one cared (I remember when it was reported). The awkward difficultly is that while this issue# meets the "reproducible" requirement absolutely, it isn't actionable. It isn't a bug. It is n bugs (for some moving value n). WSL fails the golang some watchman tests. It also fails some libc tests and some LTP tests tests. There is a testing related blog entry here and an on unofficial side discussion in #2031 (message).

Constructively, what can be done per Golang#22020 watchman is identify specific test case fails that are blocking any killer golang applications watchman scenarios with widespread adoption. This is not an easy task mind, because you'll find that in most cases the problem will be a dupe. But not always, so please report anything you find following the template.