Refactor CI configuration to use tox and Azure Pipelines for everything

[pierreglaser]

Just making sure windows CI errors in #228 also exist in master.

UPDATE: this PR fixes a coverage issue that was probably present since loky transitioned from appveyor to azure-pipelines, which made the overall coverage drop from 89% to 79% (first effective coverage drop noticed here: 6a0d37b)

UPDATE 2: this PR transitions the CI infrastructure of loky to azure-pipelines for all platforms (Linux, Mac, Windows). Doing this, it also fixes a bug that was present since 6a0d37b and that prevented the windows coverage reports to be uploaded to codecov.

[ogrisel]

It's likely a problem with the installation of sqlite that is now used by coverage >= 5.0. You can pin to coverage 4 with coverage<5 in the pip requirements to avoid this issue.

[ogrisel]

Weird. Maybe coverage is upgraded later? In anycase, when pinning a version number like this one should always add a comment to explain why or pointing to a github issue so as to know when it's safe to remove the pinning.

[ogrisel]

Indeed the codecov step upgrades coverage to version 5 which triggers the problem at the end.

[pierreglaser]

The Windows CI config should now be fixed. We still need to update the coverage token though:

Error: Missing repository upload token

Tip: See all example repositories: https://github.com/codecov?query=example
Support channels:
  Email:   hello@codecov.io
  IRC:     #codecov
  Gitter:  https://gitter.im/codecov/support
  Twitter: @codecov

Finishing: Upload to codecov

@tomMoral could you give @ogrisel or me the rights to update this token?

[pierreglaser]

Adding for the record that this PR fixes a coverage issue that was probably present since loky transitioned from appveyor to azure-pipelines, which made the overall coverage drop from 89% to 79% (first effective coverage drop noticed here: tomMoral@6a0d37b)

[tomMoral]

What right do you need? azure-pipeline rights or codecov rights?

[ogrisel]

Nice, the bash commands executed on Windows Azure CI are actually the true Windows commands and not ubuntu from the Windows Subsystem for Linux (WSL).

This means that we can use bash everywhere on Azure CI, both for Windows, Linux and macOS.

[tomMoral]

This is great news! This will simplify the CI maintenance.

[ogrisel]

We shall do the same for other projects: joblib, cloudpickle, ...

[ogrisel]

The codecov status has disappeared for the last builds on github commit status markers:

==> Azure Pipelines detected.
    project root: D:/a/1/s
    Yaml found at: .codecov.yml
==> Running gcov in D:/a/1/s (disable via -X gcov)
FIND: Parameter format not correct
==> Python coveragepy not found
==> Searching for coverage reports in:
    + D:/a/1/s
--> No coverage report found.
    Please visit http://docs.codecov.io/docs/supported-languages

[ogrisel]

By adding Python 3.8 test I have found a real bug in loky:

___________ TestsProcessPoolSpawnExecutor.test_no_stale_references ____________

self = <tests.test_process_executor_spawn.TestsProcessPoolSpawnExecutor object at 0x000001CB81013850>

    def test_no_stale_references(self):
        # Issue #16284: check that the executors don't unnecessarily hang onto
        # references.
        my_object = MyObject()
        collect = threading.Event()
        _ = weakref.ref(my_object, lambda obj: collect.set())  # noqa
        # Deliberately discarding the future.
        self.executor.submit(my_object.my_method)
        del my_object
    
        collected = False
        for i in range(5):
            if IS_PYPY:
                gc.collect()
            collected = collect.wait(timeout=1.0)
            if collected:
                return
>       assert collected, "Stale reference not collected within timeout."
E       AssertionError: Stale reference not collected within timeout.

_          = <weakref at 0x000001CB81001C20; dead>
collect    = <threading.Event object at 0x000001CB810671F0>
collected  = False
i          = 4
self       = <tests.test_process_executor_spawn.TestsProcessPoolSpawnExecutor object at 0x000001CB81013850>

@tomMoral any idea? It seems to be cause by a change in Python because this does not happen in py37.

[ogrisel]

For codecov we need to:

• find a way to launch the codecov command from the same tox environment to ensure that we run on the same python / coveragepy version
• configure the token for azure pipelines as explained here: https://hynek.me/articles/simple-python-azure-pipelines/

[ogrisel]

LGTM.

[ogrisel]

@pierreglaser:

79.54% remains the same compared to ceaef5e ...

C'est pas bon en fait. Il doit nous manquer des rapports codecov.

[pierreglaser]

Let me check. A few commits ago we were at 90%, a regression was introduced in the last ~10 commits or so.

[ogrisel]

The problem comes from:

https://dev.azure.com/joblib/loky/_build/results?buildId=315&view=logs&j=1441dfab-214e-55fe-ca89-fcb5711d8c8b&t=196a21c3-4bec-594c-74ad-a3b2994fb689&l=34

[pierreglaser]

Coverage is back up. I fixed the issue by telling coverage where to find the coverage xml files. The reason the coverage decreased today is very obscure: for some reason, azure-pipelines now (in our last commits) uses C:\Program Files\Git\usr\bin\bash.exe to run a git bash instance, whereas previously (yesterday) it would run a C:\Program Files\Git\bin\bash.exe. According to this SO post, those two shells use different versions of FIND because of their PATH, and the one today uses a wrong FIND, causing codecov to crash when publishing.

It seems to be a recurrent issue in many CI providers. (see a related github-actions issue)

[pierreglaser]

I changed the approach of the fix: I instead tell azure which Git bash to use specifically, so that I don't need to disable every single codecov step that relies on FIND (and possibly other tools that could be broken under Git/usr/bin/bash).

[pierreglaser]

Coverage data remains somewhat unpredictable though. 87% after efd9cd8, 89% after f27fe6e, which are exactly the same source code...

[ogrisel]

Sounds good enough to me. Let's merge.