Virtualized or containerized environments might not report number of CPUs correctly

[thedrow]

This problem is mainly present itself in CI environments like the ones Travis provides.
When you use -n auto xdist will create more workers than what's available for the container/VM since that is what's written in the /proc/ filesystem which multiprocessing.cpu_count() reads the number of CPUs from.
A possible solution would be to use psutil which might be able to read the /proc/ filesystem more correctly but I'm not 100% sure about it.

As a workaround I'd like to suggest adding:

if os.environ.get('TRAVIS') == 'true' and n == 'auto':
  cpu_count = 2

to the code to prevent spawning too many processes and slowing down test runs.
A better workaround would be to provide a hook to allow to reinterpret what auto means and return a different number.

[thedrow]

Based on the tests in #10 it seems that os.cpu_count() doesn't respond with the correct answer so psutil won't either since it implemented the same way as os.cpu_count().

[RonnyPfannschmidt]

i presume the only way to do this correct is to get travis use the lxc-proc filesystem for containers (which provides a corrected view of proc)

[thedrow]

But not all of their environments use lxc.

[RonnyPfannschmidt]

does it also report it wrong on top of a kvm/osx?

[thedrow]

On KVM yes, you get the same results as with LXC. No idea what about OSX.

[thedrow]

I think that providing a hook will workfor now. We can document that under Travis your conftest.py should look like:

def pytest__xdist_auto_workers(cpu_count):
  if os.environ.get('TRAVIS') == 'true':
    return 2
  return cpu_count

Not sure about the hook name though.

[nicoddemus]

pytest-vw contains a pretty good list of known CI services. I think it would be nice for xdist to work out of the box with those.

[nicoddemus]

Fixed by #309.