Fix skipping to operate on a per-environment basis

[astrofrog]

This fixes #602. Example output when running asv run ALL --skip:

· Running 470 total benchmarks (47 commits * 2 environments * 5 benchmarks)
[  2.13%] · For project commit hash 622a34ca:
[  2.13%] ·· No benchmarks to run for conda-py2.7-numpy1.11
[  2.13%] ·· No benchmarks to run for conda-py2.7-numpy1.14
[  4.26%] · For project commit hash d8f30472:
[  4.26%] ·· No benchmarks to run for conda-py2.7-numpy1.11
[  4.26%] ·· No benchmarks to run for conda-py2.7-numpy1.14
[  6.38%] · For project commit hash 5f874ea8:
[  6.38%] ·· No benchmarks to run for conda-py2.7-numpy1.11
[  6.38%] ·· No benchmarks to run for conda-py2.7-numpy1.14
[  8.51%] · For project commit hash ad1e1de8:
[  8.51%] ·· No benchmarks to run for conda-py2.7-numpy1.11
[  8.51%] ·· No benchmarks to run for conda-py2.7-numpy1.14
[ 10.64%] · For project commit hash 689d9503:
[ 10.64%] ·· No benchmarks to run for conda-py2.7-numpy1.11
[ 10.64%] ·· No benchmarks to run for conda-py2.7-numpy1.14
[ 12.77%] · For project commit hash 1cce9e12:
[ 12.77%] ·· No benchmarks to run for conda-py2.7-numpy1.11
[ 12.77%] ·· No benchmarks to run for conda-py2.7-numpy1.14
[ 14.89%] · For project commit hash 7eea79ba:
[ 14.89%] ·· No benchmarks to run for conda-py2.7-numpy1.11
[ 14.89%] ·· No benchmarks to run for conda-py2.7-numpy1.14
[ 17.02%] · For project commit hash 4806274b:
[ 17.02%] ·· No benchmarks to run for conda-py2.7-numpy1.11
[ 17.02%] ·· No benchmarks to run for conda-py2.7-numpy1.14
[ 19.15%] · For project commit hash f28fe3fe:
[ 19.15%] ·· No benchmarks to run for conda-py2.7-numpy1.11
[ 19.15%] ·· No benchmarks to run for conda-py2.7-numpy1.14
[ 21.28%] · For project commit hash 304c512d:
[ 21.28%] ·· No benchmarks to run for conda-py2.7-numpy1.11
[ 21.28%] ·· No benchmarks to run for conda-py2.7-numpy1.14
[ 23.40%] · For project commit hash 46b96c1c:
[ 23.40%] ·· No benchmarks to run for conda-py2.7-numpy1.11
[ 23.40%] ·· No benchmarks to run for conda-py2.7-numpy1.14
[ 25.53%] · For project commit hash 10760777:
[ 25.53%] ·· No benchmarks to run for conda-py2.7-numpy1.11
[ 25.53%] ·· No benchmarks to run for conda-py2.7-numpy1.14
[ 27.66%] · For project commit hash 5da5e01c:
[ 27.66%] ·· No benchmarks to run for conda-py2.7-numpy1.11
[ 27.66%] ·· No benchmarks to run for conda-py2.7-numpy1.14
[ 28.72%] · For project commit hash 5e5b5820:
[ 28.72%] ·· No benchmarks to run for conda-py2.7-numpy1.11
[ 28.72%] ·· Building for conda-py2.7-numpy1.14.
[ 28.72%] ·· Benchmarking conda-py2.7-numpy1.14
[ 28.94%] ··· Running benchmarks.MemSuite.mem_list                                                                                                2.42k
[ 29.15%] ··· Running benchmarks.TimeSuite.time_iterkeys                                                                                     32.4±0.8μs
[ 29.36%] ··· Running benchmarks.TimeSuite.time_keys                                                                                         32.4±0.7μs
[ 29.57%] ··· Running benchmarks.TimeSuite.time_range                                                                                        78.0±0.9μs
[ 29.79%] ··· Running benchmarks.TimeSuite.time_xrange                                                                                         79.0±3μs
[ 30.85%] · For project commit hash 859ef6c6:
[ 30.85%] ·· No benchmarks to run for conda-py2.7-numpy1.11
[ 30.85%] ·· Building for conda-py2.7-numpy1.14..
[ 30.85%] ·· Benchmarking conda-py2.7-numpy1.14
[ 31.06%] ··· Running benchmarks.MemSuite.mem_list                                                                                                2.42k
[ 31.28%] ··· Running benchmarks.TimeSuite.time_iterkeys                                                                                     31.6±0.5μs
[ 31.49%] ··· Running benchmarks.TimeSuite.time_keys                                                                                           33.0±1μs
[ 31.70%] ··· Running benchmarks.TimeSuite.time_range                                                                                        78.4±0.8μs

Prior to this, if one environment had no run benchmarks and the other did, there was no way to run the missing ones for one environment.

[pllim]

Thanks!

[pv]

I think the right thing to do here is to do

active_environments = [env for env in environments 
    if set(six.iterkeys(benchmarks)).difference(skipped_benchmarks[env.name])]

if not active_environments:
    continue

if commit_hash:
...
for subenv in util.iter_chunks(active_environments, parallel):
...

[astrofrog]

I agree, this is cleaner - done!

[astrofrog]

@pv - this is ready for review

[astrofrog]

@pv - does this seem ok now?