[ci] rebalanced Windows tests

[StrikerRUS]

Now we have the following setting for Windows:

• VSTS: regular (with examples), sdist, bdist under VS 2017 (~10 min)
• Appveyor: regular (with examples) under VS 2015 and MinGW (~10-15 min, was ~40 min)

Also, MinGW version has been updated from 6.3.0 to 7.3.0.

Known bug: conda at Windows cannot handle Python 3.7 properly yet (Importing the multiarray numpy extension module failed. Most likely you are trying to import a failed build of numpy. Original error was: DLL load failed: The specified module could not be found. at VSTS and Command executed with exception: from collections import Sequence at Appveyor).

While working on this PR, I found weird bug. The same symptoms were described in #1475. When I tried to run all examples by pytest %APPVEYOR_BUILD_FOLDER%\tests instead of pytest %APPVEYOR_BUILD_FOLDER%\tests\python_package_test, two C_API tests were passed succesfully, but all other tests failed with OSError: exception: access violation writing 0xFFFFFFFF93400000. At present this bug is avoided by running only Python tests (without C_API). I suppose, it's somehow connected to that the same lib_lightgbm.dll is loaded twice from different places (from installation directory for C_API tests and from python-package directory for Python tests).

[StrikerRUS]

May I leave this artifact for my personal purpose? Due to some reason I have installed only VS 2013 on my machine, but sometimes need to test the latest library file from master.

[guolinke]

sure

[StrikerRUS]

Thanks!

[guolinke]

it seems azure pipeline is release and free for open source projects. maybe we can move on it?

[StrikerRUS]

Wow, how many CIs ! 😃

My opinion is that we need to balance between our current CIs (VSTS, Appveyor, Travis), because we have enough experience in them, and after that give a try to Azure.

BTW, which OS are supported at Azure?

[guolinke]

@StrikerRUS I think Azure Pipeline is actually the vsts-ci, but it is free for open-source project.

[StrikerRUS]

Sorry, am I right, that you want to drop VSTS and move to Azure?

[guolinke]

@StrikerRUS Yeah, I think they share the same config. But Azure Pipeline may is able to add community users to manage, while vsts-ci can't.

[StrikerRUS]

@guolinke Got it!

Seems Azure Pipelines is very young! 😃 They have been announced yesterday: http://disq.us/p/1vm7rh5

I think they share the same config.

Does it mean that I can continue rebalancing (at present I'm working on Linux, practically finished)? And then we'll need just to perform some minor management work, like register an account, rename the config file, etc?

[StrikerRUS]

Yesterday I noticed that the badge has been changed, now the name of service.

[guolinke]

@StrikerRUS yeah, the free version still has 10 concurrent jobs.

[guolinke]

@StrikerRUS just move to it, seems everything is fine.

[StrikerRUS]

@guolinke I've noticed it: 404 error happened just right when I wanted to check my rebalancing for Linux status 😄

GPU failed. Seems it doesn't perform --recursive clone.

[guolinke]

@StrikerRUS now you can admin the LightGBM-CI on Azure Pipeline 😄

[StrikerRUS]

@guolinke Thank you very much! Azure Pipeline seems to be more convenient than VSTS: there are no useless things on the screen, only important tabs and settings, and finally cancel and rebuild buttons!

[StrikerRUS]

@guolinke Hmm... I can rebuild PR from you, but cannot rebuild master.

[StrikerRUS]

@guolinke Also, it seems to be they use outdated gnutls library instead of openssl. Therefor, phantom error while pulling submodule happens.

fatal: unable to access 'https://github.com/Microsoft/LightGBM/': gnutls_handshake() failed: Error in the pull function.

UPD:
Seems to be fixed now.

[davidstaheli]

Hi @StrikerRUS and @guolinke. I'm a PM from the Azure Pipelines team. Thanks a lot for giving it a try. Please let me know if anything is missing that you need. We want to help however we can.

[StrikerRUS]

@davidstaheli Hi! Thank you very much for coming here!

I have several moments to share with you after my small experience with Azure Pipelines.

Conda on macOS doesn't work without the following fix:
https://github.com/Microsoft/LightGBM/blob/e1b5901ffd8772e5c6c4cd78c4ac131b17bfad98/.vsts-ci.yml#L77-L79
It raises error about the lack of permissions needed to create conda environment:
https://lightgbm-ci.visualstudio.com/lightgbm-ci/_build/results?buildId=73&view=logs

Even after @guolinke invited me in the team at Azure DevOps, I have no ability to rebuild branches, only PRs. Please see this my #1660 (comment).

It's a real pity that there is no GitHub deployment task like at Appveyor or Travis.

Thanks again!

[davidstaheli]

We'll look into these and get each of them addressed. Thanks a lot, @StrikerRUS.

[StrikerRUS]

@davidstaheli Many thanks!

I've recollected another inconvenient case which consists in the fact that you need to run chmod before executing shell script (at both Ubuntu and macOS).
https://github.com/Microsoft/LightGBM/blob/6214657aed2c9c682e1937748d39d5bf78698164/.vsts-ci.yml#L94-L100

[davidstaheli]

Thanks @StrikerRUS! Looking into the chmod issue, I've seen reports of people needing to add the execute permission before committing & pushing the file, and then the git fetch will automatically re-apply the execute permission. Here's an example. Do you know if that's already been done for your .sh files? If so, it sounds like something we need to address.

[StrikerRUS]

@davidstaheli Thank you very much! The problem was solved using the solution from your link. It's rather strange that, for example, Travis doesn't suffer from the same problem. Anyway, thank you for the solution and sorry for disturbing you.

[davidstaheli]

Please never hesitate! I want to help however I can, so please reach out to me with anything else that comes up. We'll investigate why this was necessary for Azure Pipelines, but not Travis. Thanks a lot.

[brcrista]

@StrikerRUS I see you are running with V0 of the task. V1 of the task doesn't require you to create an environment to use Conda. So as a workaround, try upgrading to V1 of the task and make sure "Create a custom environment" isn't checked.

[StrikerRUS]

Thanks @brcrista for the reply! But could you please elaborate your workaround? Simple change CondaEnvironment@0 to CondaEnvironment@1 here
https://github.com/Microsoft/LightGBM/blob/64002b2c0de9d2308a5bb90e8e91fca184fcb744/.vsts-ci.yml#L82-L87
broke entire conda setup.

[brcrista]

Look at the YAML snippet at https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/package/conda-environment?view=vsts

I'm saying createCustomEnvironment: false and omit environmentName. Let me know if that works

[StrikerRUS]

@brcrista I've already seen that snippet. Unfortunately, we need named environment in our test scripts, so I'd better leave my current hotfix till you'll release a real fix for the permission error.

[brcrista]

Can you try running on Hosted Linux Preview instead of Hosted Ubuntu 16.04? I believe that should work.

[StrikerRUS]

@brcrista I'm facing this issue on macOS, not Linux:
https://github.com/Microsoft/LightGBM/blob/64002b2c0de9d2308a5bb90e8e91fca184fcb744/.vsts-ci.yml#L62-L87

The same task on Linux works fine (with Hosted Linux Preview, as you've said).

[brcrista]

sorry, never mind then.