add windows ci setting #471

yhmtsai · 2020-03-03T11:32:16Z

~~This PR add the pipeline for windows gitlab shared runner.~~
This PR add the github action for windows
For cuda, it only test compilation. For nocuda, it test compilation and google test.

I only keep gitlab ci in commit.
gitlab ci does not give separate badge to show windows compiliation and gives 2000 min/month in our current gitlab plan.
Github action gives badge based on workflow and give unlimited min for open source.
However, it only gives 14GB for storage, so compiling windows with static and debug is failed.

github action strategy allows to evaluate all combination of matrix.
For example,

build_type: ["Debug", "Release"]
shared: ["ON", "OFF"]

gives four jobs.

I only specify only two config to reduce the list of pull request check.

Gitlab seems to have whole pipeline badge.
In #470 , Github action can give badges on different workflow.
Thus, I will try to see whether it also works on github action.
By doing so, we can separate the status from different platform

Summary:

MSVC: cuda (only compile), ref (build/test/install)
MinGW: omp (build/test/install)
Cygwin: omp (build/test/install)
split install and compile in OSX-build

Note:

I do not find a way to use cygwin as a shell in github action, so I need to set cygwin path only to make everything work.
MSVC/mingw install Ginkgo into C:\Program files(x86)\Ginkgo not C:\Program files\Ginkgo.
I can not use CMAKE_INSTALL_PREFIX_INITIALIZED_TO_DEFAULT to set prefix force, which leads that user can only set CMAKE_INSTALL_PREFIX in the beginning. reference: https://gitlab.kitware.com/cmake/cmake/issues/18312
MSVC nvcc has flags like -Xcompiler="-O2 ...", so I add a macro to modify cuda_flag for GinkgoConfig.pc by " -> \"
Github action create a script containing the commands in the run: and use shell to run the script, so github action only catch the last error code. For example, in workflow, the parilu test is failed in configure step, but the last command make install is successful. Thus, the configure is successful.

TODO:

.gitlab-ci.yml

pratikvn · 2020-03-04T10:02:38Z

It seems that OpenMP is being detected, but you do not compile for it. Is that for any specific reason ?

yhmtsai · 2020-03-04T10:06:15Z

MSVC only supports OpenMP 2, but we use something unavailble in OpenMP 2

upsj

LGTM! In my opinion, you could also move the changes from #473 into this PR

.github/workflows/windows-build.yml

pratikvn

LGTM!

.github/workflows/windows-build.yml

codecov · 2020-03-05T01:57:17Z

Codecov Report

Merging #471 into develop will increase coverage by 0.15%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop     #471      +/-   ##
===========================================
+ Coverage    88.69%   88.85%   +0.15%     
===========================================
  Files          256      256              
  Lines        16478    16436      -42     
===========================================
- Hits         14615    14604      -11     
+ Misses        1863     1832      -31

Impacted Files	Coverage Δ
omp/factorization/par_ilu_kernels.cpp	`100.00% <ø> (ø)`
include/ginkgo/core/matrix/sellp.hpp	`81.81% <0.00%> (-8.43%)`	⬇️
include/ginkgo/core/base/polymorphic_object.hpp	`98.43% <0.00%> (+3.19%)`	⬆️
include/ginkgo/core/matrix/hybrid.hpp	`84.21% <0.00%> (+7.28%)`	⬆️
include/ginkgo/core/matrix/coo.hpp	`100.00% <0.00%> (+29.72%)`	⬆️
include/ginkgo/core/matrix/ell.hpp	`100.00% <0.00%> (+31.25%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fda35be...5027399. Read the comment docs.

tcojean

LGTM. But would be great if we could also check OpenMP. I don't know what are the possibilities for that?

Also, does this compile with MSVC? This isn't obvious.

.github/workflows/windows-build.yml

yhmtsai · 2020-03-11T12:43:09Z

Adding "" is failed in cygwin. in https://github.com/ginkgo-project/ginkgo/actions/runs/52976212
cmake command can handle the variable containing space.
(related ref: https://stackoverflow.com/questions/8925396/why-does-cmake-prefixes-spaces-with-backslashes-when-executing-a-command/8933997#8933997)
cygwin considers "b\ uild" as "b/ uild", so it is failed

The current code has no problem with space ("b uild") on https://github.com/ginkgo-project/ginkgo/actions/runs/53528009
I keep testing "build" not "b uild" in the PR.

tcojean

LGTM. Some minor comments.

.github/workflows/osx.yml

CMakeLists.txt

using variables in powershell cmake is failed.

- remove gitlab ci - copmiling static shared with /bigobj uses too much storage, so it is not available in github-host action - use fail-fast:false in strategy to avoid cancel another job when one is failed

Co-Authored-By: Tobias Ribizel <ribizel@kit.edu>

yhmtsai · 2020-03-17T15:38:05Z

We have many badges now and they can not fit in one line.
I split them into two parts.

tcojean · 2020-03-17T15:40:59Z

@yhmtsai Thanks. I was thinking something similar, but maybe the other way around: split after reliability badge, this way first line is all code quality things, second line is information on the software (CDash dashboard, etc).

yhmtsai · 2020-03-17T16:07:29Z

This is the new layout

pratikvn

LGTM!

sonarqubecloud · 2020-03-17T20:58:53Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities (and 0 Security Hotspots to review)
0 Code Smells

No Coverage information
100.0% Duplication

The Ginkgo team is proud to announce the new minor release of Ginkgo version 1.2.0. This release brings full HIP support to Ginkgo, new preconditioners (ParILUT, ISAI), conversion between double and float for all LinOps, and many more features and fixes. Supported systems and requirements: + For all platforms, cmake 3.9+ + Linux and MacOS + gcc: 5.3+, 6.3+, 7.3+, all versions after 8.1+ + clang: 3.9+ + Intel compiler: 2017+ + Apple LLVM: 8.0+ + CUDA module: CUDA 9.0+ + HIP module: ROCm 2.8+ + Windows + MinGW and CygWin: gcc 5.3+, 6.3+, 7.3+, all versions after 8.1+ + Microsoft Visual Studio: VS 2017 15.7+ + CUDA module: CUDA 9.0+, Microsoft Visual Studio + OpenMP module: MinGW or CygWin. The current known issues can be found in the [known issues page](https://github.com/ginkgo-project/ginkgo/wiki/Known-Issues). # Additions Here are the main additions to the Ginkgo library. Other thematic additions are listed below. + Add full HIP support to Ginkgo [#344](#344), [#357](#357), [#384](#384), [#373](#373), [#391](#391), [#396](#396), [#395](#395), [#393](#393), [#404](#404), [#439](#439), [#443](#443), [#567](#567) + Add a new ISAI preconditioner [#489](#489), [#502](#502), [#512](#512), [#508](#508), [#520](#520) + Add support for ParILUT and ParICT factorization with ILU preconditioners [#400](#400) + Add a new BiCG solver [#438](#438) + Add a new permutation matrix format [#352](#352), [#469](#469) + Add CSR SpGEMM support [#386](#386), [#398](#398), [#418](#418), [#457](#457) + Add CSR SpGEAM support [#556](#556) + Make all solvers and preconditioners transposable [#535](#535) + Add CsrBuilder and CooBuilder for intrusive access to matrix arrays [#437](#437) + Add a standard-compliant allocator based on the Executors [#504](#504) + Support conversions for all LinOp between double and float [#521](#521) + Add a new boolean to the CUDA and HIP executors to control DeviceReset (default off) [#557](#557) + Add a relaxation factor to IR to represent Richardson Relaxation [#574](#574) + Add two new stopping criteria, for relative (to `norm(b)`) and absolute residual norm [#577](#577) ### Example additions + Templatize all examples to simplify changing the precision [#513](#513) + Add a new adaptive precision block-Jacobi example [#507](#507) + Add a new IR example [#522](#522) + Add a new Mixed Precision Iterative Refinement example [#525](#525) + Add a new example on iterative trisolves in ILU preconditioning [#526](#526), [#536](#536), [#550](#550) ### Compilation and library changes + Auto-detect compilation settings based on environment [#435](#435), [#537](#537) + Add SONAME to shared libraries [#524](#524) + Add clang-cuda support [#543](#543) ### Other additions + Add sorting, searching and merging kernels for GPUs [#403](#403), [#428](#428), [#417](#417), [#455](#455) + Add `gko::as` support for smart pointers [#493](#493) + Add setters and getters for criterion factories [#527](#527) + Add a new method to check whether a solver uses `x` as an initial guess [#531](#531) + Add contribution guidelines [#549](#549) # Fixes ### Algorithms + Improve the classical CSR strategy's performance [#401](#401) + Improve the CSR automatical strategy [#407](#407), [#559](#559) + Memory, speed improvements to the ELL kernel [#411](#411) + Multiple improvements and fixes to ParILU [#419](#419), [#427](#427), [#429](#429), [#456](#456), [#544](#544) + Fix multiple issues with GMRES [#481](#481), [#523](#523), [#575](#575) + Optimize OpenMP matrix conversions [#505](#505) + Ensure the linearity of the ILU preconditioner [#506](#506) + Fix IR's use of the advanced apply [#522](#522) + Fix empty matrices conversions and add tests [#560](#560) ### Other core functionalities + Fix complex number support in our math header [#410](#410) + Fix CUDA compatibility of the main ginkgo header [#450](#450) + Fix isfinite issues [#465](#465) + Fix the Array::view memory leak and the array/view copy/move [#485](#485) + Fix typos preventing use of some interface functions [#496](#496) + Fix the `gko::dim` to abide to the C++ standard [#498](#498) + Simplify the executor copy interface [#516](#516) + Optimize intermediate storage for Composition [#540](#540) + Provide an initial guess for relevant Compositions [#561](#561) + Better management of nullptr as criterion [#562](#562) + Fix the norm calculations for complex support [#564](#564) ### CUDA and HIP specific + Use the return value of the atomic operations in our wrappers [#405](#405) + Improve the portability of warp lane masks [#422](#422) + Extract thread ID computation into a separate function [#464](#464) + Reorder kernel parameters for consistency [#474](#474) + Fix the use of `pragma unroll` in HIP [#492](#492) ### Other + Fix the Ginkgo CMake installation files [#414](#414), [#553](#553) + Fix the Windows compilation [#415](#415) + Always use demangled types in error messages [#434](#434), [#486](#486) + Add CUDA header dependency to appropriate tests [#452](#452) + Fix several sonarqube or compilation warnings [#453](#453), [#463](#463), [#532](#532), [#569](#569) + Add shuffle tests [#460](#460) + Fix MSVC C2398 error [#490](#490) + Fix missing interface tests in test install [#558](#558) # Tools and ecosystem ### Benchmarks + Add better norm support in the benchmarks [#377](#377) + Add CUDA 10.1 generic SpMV support in benchmarks [#468](#468), [#473](#473) + Add sparse library ILU in benchmarks [#487](#487) + Add overhead benchmarking capacities [#501](#501) + Allow benchmarking from a matrix list file [#503](#503) + Fix benchmarking issue with JSON and non-finite numbers [#514](#514) + Fix benchmark logger crashers with OpenMP [#565](#565) ### CI related + Improvements to the CI setup with HIP compilation [#421](#421), [#466](#466) + Add MacOSX CI support [#470](#470), [#488](#488) + Add Windows CI support [#471](#471), [#488](#488), [#510](#510), [#566](#566) + Use sanitizers instead of valgrind [#476](#476) + Add automatic container generation and update facilities [#499](#499) + Fix the CI parallelism settings [#517](#517), [#538](#538), [#539](#539) + Make the codecov patch check informational [#519](#519) + Add support for LLVM sanitizers with improved thread sanitizer support [#578](#578) ### Test suite + Add an assertion for sparsity pattern equality [#416](#416) + Add core and reference multiprecision tests support [#448](#448) + Speed up GPU tests by avoiding device reset [#467](#467) + Change test matrix location string [#494](#494) ### Other + Add Ginkgo badges from our tools [#413](#413) + Update the `create_new_algorithm.sh` script [#420](#420) + Bump copyright and improve license management [#436](#436), [#433](#433) + Set clang-format minimum requirement [#441](#441), [#484](#484) + Update git-cmake-format [#446](#446), [#484](#484) + Disable the development tools by default [#442](#442) + Add a script for automatic header formatting [#447](#447) + Add GDB pretty printer for `gko::Array` [#509](#509) + Improve compilation speed [#533](#533) + Add editorconfig support [#546](#546) + Add a compile-time check for header self-sufficiency [#552](#552) # Related PR: #583

yhmtsai added reg:ci-cd This is related to the continuous integration system. plat:windows This is related to the Windows Operating system labels Mar 3, 2020

yhmtsai self-assigned this Mar 3, 2020

upsj reviewed Mar 3, 2020

View reviewed changes

.gitlab-ci.yml Outdated Show resolved Hide resolved

yhmtsai force-pushed the windows_ci branch 6 times, most recently from 52e71e9 to 0d7faa1 Compare March 4, 2020 08:33

yhmtsai mentioned this pull request Mar 4, 2020

Fix windows cuda generic #473

Merged

upsj approved these changes Mar 4, 2020

View reviewed changes

.github/workflows/windows-build.yml Outdated Show resolved Hide resolved

.github/workflows/windows-build.yml Outdated Show resolved Hide resolved

yhmtsai force-pushed the windows_ci branch 2 times, most recently from 89e164b to 84995c3 Compare March 4, 2020 10:31

yhmtsai requested review from pratikvn, tcojean, thoasm and hartwiganzt March 4, 2020 10:52

pratikvn approved these changes Mar 4, 2020

View reviewed changes

.github/workflows/windows-build.yml Show resolved Hide resolved

yhmtsai force-pushed the windows_ci branch from 84995c3 to da480f8 Compare March 4, 2020 22:17

tcojean approved these changes Mar 5, 2020

View reviewed changes

.github/workflows/windows-build.yml Show resolved Hide resolved

yhmtsai force-pushed the windows_ci branch 2 times, most recently from e9eefcf to 7261c2a Compare March 6, 2020 10:23

yhmtsai added the 1:ST:WIP This PR is a work in progress. Not ready for review. label Mar 6, 2020

yhmtsai force-pushed the windows_ci branch 3 times, most recently from e3cd966 to 2b56584 Compare March 8, 2020 00:22

yhmtsai force-pushed the windows_ci branch from d95e197 to 19bdcc6 Compare March 11, 2020 12:35

yhmtsai force-pushed the windows_ci branch 2 times, most recently from dbb5012 to 63797d5 Compare March 12, 2020 11:29

tcojean approved these changes Mar 13, 2020

View reviewed changes

.github/workflows/osx.yml Outdated Show resolved Hide resolved

CMakeLists.txt Outdated Show resolved Hide resolved

yhmtsai force-pushed the windows_ci branch from 2365a59 to 18a2b12 Compare March 13, 2020 15:58

yhmtsai and others added 11 commits March 17, 2020 16:32

add windows gitlab ci setting

bc926e9

using variables in powershell cmake is failed.

add windows github action

9e06a34

- remove gitlab ci - copmiling static shared with /bigobj uses too much storage, so it is not available in github-host action - use fail-fast:false in strategy to avoid cancel another job when one is failed

GINKGO_DEVEL_TOOLS=off is already default setting

4315f7e

Co-Authored-By: Tobias Ribizel <ribizel@kit.edu>

enable /bigobj in debug build

d50d646

add cygwin/mingw github action

f930368

try msvc test_install and fix cygwin test_install

383131d

test_install use correct config in MSVC

db9bbd1

split test_install and ctest of OSX-build

97f05fd

fix config problem " -> \" for windows nvcc

e22bc02

add MSVC condition in test_install cuda

a878cc2

review update

9651eaf

yhmtsai force-pushed the windows_ci branch from 18a2b12 to 9651eaf Compare March 17, 2020 15:35

change the badges layout

5027399

pratikvn approved these changes Mar 17, 2020

View reviewed changes

yhmtsai added 1:ST:ready-to-merge This PR is ready to merge. and removed 1:ST:ready-for-review This PR is ready for review labels Mar 18, 2020

yhmtsai merged commit 793a8a8 into develop Mar 18, 2020

yhmtsai deleted the windows_ci branch March 18, 2020 04:01

tcojean mentioned this pull request Jun 23, 2020

Release/1.2.0 #576

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add windows ci setting #471

add windows ci setting #471

yhmtsai commented Mar 3, 2020 •

edited

Loading

pratikvn commented Mar 4, 2020

yhmtsai commented Mar 4, 2020

upsj left a comment •

edited

Loading

pratikvn left a comment

codecov bot commented Mar 5, 2020 •

edited

Loading

tcojean left a comment •

edited

Loading

yhmtsai commented Mar 11, 2020

tcojean left a comment

yhmtsai commented Mar 17, 2020

tcojean commented Mar 17, 2020

yhmtsai commented Mar 17, 2020

pratikvn left a comment

sonarqubecloud bot commented Mar 17, 2020

add windows ci setting #471

add windows ci setting #471

Conversation

yhmtsai commented Mar 3, 2020 • edited Loading

pratikvn commented Mar 4, 2020

yhmtsai commented Mar 4, 2020

upsj left a comment • edited Loading

Choose a reason for hiding this comment

pratikvn left a comment

Choose a reason for hiding this comment

codecov bot commented Mar 5, 2020 • edited Loading

Codecov Report

tcojean left a comment • edited Loading

Choose a reason for hiding this comment

yhmtsai commented Mar 11, 2020

tcojean left a comment

Choose a reason for hiding this comment

yhmtsai commented Mar 17, 2020

tcojean commented Mar 17, 2020

yhmtsai commented Mar 17, 2020

pratikvn left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Mar 17, 2020

yhmtsai commented Mar 3, 2020 •

edited

Loading

upsj left a comment •

edited

Loading

codecov bot commented Mar 5, 2020 •

edited

Loading

tcojean left a comment •

edited

Loading