Update CI configuration

[pkch]

• Add caching of pip packages to Travis (makes tests faster by ~15 sec x 5 builds = 75 sec)
• Skip builds in AppVeyor when only .rst/.md/docs/etc. are changed
• Remove lint from AppVeyor (it's already run in Travis)

I tried adding caching to AppVeyor but it didn't help according to my benchmarks.

Auto-skipping build is not availalbe as a feature in Travis (apart from manually adding [skip ci] to the commit message that no one will ever remember); if there's interest, I can add a small before_install script (which I found in Facebook React repo).

Note that currently the bottleneck by a long margin is AppVeyor: 1 concurrent VM, 3.5 min/build, 4 builds/commit; total ~14 min/commit. By compariosn, Travis: 5 concurrent VMs, 2.5 min/build, 5 builds/commit; total ~2.5 min/commit. So whatever we do to improve test times should mostly focus on AppVeyor for now.

[gvanrossum]

Thanks! I wonder if we should limit AppVeyor to a single build? (But IIRC @ddfisher depends on it for building wheels.)

Quite separately I wonder if we could have a doc build step in Travis that tries to build the docs when anything under docs/ is changed?

[pkch]

I would imagine that the differences between python/Linux and python/Windows are mostly related to subprocesses and I/O; the differences between Win32 and Win64 are mostly around package discovery, build and installation; and the differences between python 3.3-3.7 are much more extensive and can affect any code.

So I'd say since all python versions are already tested on Travis, a single test of python 3.6 / win64 is probably enough for testing subprocesses and IO behavior. But it totally makes sense that building wheels needs to be tested across more environments.

If @ddfisher thinks it's worth it, I could set up a branch where some rarely-broken tests (like wheels) are automatically tested when the CI service isn't busy with more urgent things.

[pkch]

Quite separately I wonder if we could have a doc build step in Travis that tries to build the docs when anything under docs/ is changed?

I am making a PR for that. The conditional build itself is not hard; but I'll need to do some additional research because it's not always obvious how to determine what changed.

Naively, I thought we could just use TRAVIS_COMMIT_RANGE to indicate the new information in the current push; but that range refers to the lost commit in case of a rebase or a forced push, making it useless for this purpose. It wouldn't be good to always give up in such cases and build the docs anyway, since that would unnecessarily slow down many builds (I personally force-push a lot to clean up typos, etc.).

There are some solutions that other Github projects have come up with, after I feel comfortable with them, I'll submit a PR based on one of them.

I'm also investigating how AppVeyor solves this issue in its skip_commits directive implementation (it doesn't provide an env. var. equivalent to TRAVIS_COMMIT_RANGE). If they take the conservative route and just never skip in case the start commit is unavailable, it's fine for us, since without skip_build we were building on every push anyway.

[gvanrossum]

Thanks so much for looking into this!