Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix failing tests for GitHub Actions CI #1608

Merged
merged 20 commits into from
Jun 16, 2021
Merged

fix failing tests for GitHub Actions CI #1608

Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
85a6dab
fix failing tests for GitHub Actions CI
oskooi Jun 12, 2021
254c0e3
reduce tolerances for python/tests/test_mode_coeffs.py
oskooi Jun 12, 2021
72ac87b
install libctl and MPB from source and use autogen.sh rather configur…
oskooi Jun 13, 2021
4cc4e4e
run autogen.sh for libGDSII source repository
oskooi Jun 13, 2021
d6162dc
add back libtool to common dependencies
oskooi Jun 13, 2021
faea27e
revamp matrix to include C++ and Python tests for serial and MPI
oskooi Jun 14, 2021
afe100f
use env context when defining environment variables in workflow file
oskooi Jun 14, 2021
fba5281
leave a space between curly brackets of workflow variables
oskooi Jun 14, 2021
3166fbd
create workflow environment variables and explicitly define /home/osk…
oskooi Jun 14, 2021
dc2e0f5
define environment variables using an environment file for correct pa…
oskooi Jun 15, 2021
1db2e90
escape double quotes to preserve spaces
oskooi Jun 15, 2021
f6bf161
install dependencies before defining environment variables
oskooi Jun 15, 2021
fe8edb2
define GEN_CTL_IO env variable before building libctl from source
oskooi Jun 15, 2021
77b0b3c
separate step for mpi4py install
oskooi Jun 15, 2021
f54b9df
fix configure option for Meep related to MPI
oskooi Jun 15, 2021
a96dae3
replace out-of-tree with in-tree build
oskooi Jun 15, 2021
a7cd5d4
run Python tests as a separate step and output logs at the end
oskooi Jun 15, 2021
7136b98
remove misplaced - from run lines in Output logs
oskooi Jun 15, 2021
1a44cb8
revert python/tests/test_mode_coeffs.py
oskooi Jun 15, 2021
44f1675
remove output logs step
oskooi Jun 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 82 additions & 36 deletions .github/workflows/build-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,56 +16,102 @@ jobs:
fail-fast: false
matrix:
include:
- name-prefix: "C++ without MPI"
- name-prefix: "Python 3.6 without MPI"
enable-mpi: false
configure-options: "--without-python"
python-version: "none"
- name-prefix: "C++ with MPI"
enable-mpi: true
configure-options: "--without-python --with-mpi"
python-version: "none"
- name-prefix: "Python 3.6"
enable-mpi: false
configure-options: "--with-python --enable-maintainer-mode"
configure-options: "--without-mpi"
python-version: "3.6"
- name-prefix: "Python 3.9"
- name-prefix: "Python 3.9 without MPI"
enable-mpi: false
configure-options: "--with-python --enable-maintainer-mode"
configure-options: "--without-mpi"
python-version: "3.9"
- name-prefix: "Python 3.6 with MPI"
enable-mpi: true
configure-options: "--with-mpi"
python-version: "3.6"
- name-prefix: "Python 3.9 with MPI"
enable-mpi: true
configure-options: "--with-mpi"
python-version: "3.9"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is a true Cartesian product of MPI x Python version, you don't have to rely on include to build the four cases manually: https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix


steps:
- name: Install Dependencies
- name: Define common environment variables
run: |
sudo apt-get update -y
sudo apt-get install -y autoconf automake libaec-dev libctl-dev libfftw3-dev libgdsii-dev libgsl-dev libharminv-dev libhdf5-dev libtool mpb mpb-dev swig
- name: Install MPI
echo "CPPFLAGS=-I${HOME}/local/include" >> $GITHUB_ENV
echo "LDFLAGS=-L${HOME}/local/lib" >> $GITHUB_ENV
echo "HDF5_BASE_CPPFLAGS=-I/usr/include/hdf5" >> $GITHUB_ENV
echo "HDF5_BASE_LDFLAGS=-L/usr/lib/x86_64-linux-gnu/hdf5" >> $GITHUB_ENV
echo "GEN_CTL_IO=${HOME}/local/bin/gen-ctl-io" >> $GITHUB_ENV

- name: Define more common environment variables
run: |
echo "HDF5_SERIAL_CPPFLAGS=${HDF5_BASE_CPPFLAGS}/serial" >> $GITHUB_ENV
echo "HDF5_PARALLEL_CPPFLAGS=${HDF5_BASE_CPPFLAGS}/openmpi" >> $GITHUB_ENV
echo "HDF5_SERIAL_LDFLAGS=${HDF5_BASE_LDFLAGS}/serial" >> $GITHUB_ENV
echo "HDF5_PARALLEL_LDFLAGS=${HDF5_BASE_LDFLAGS}/openmpi" >> $GITHUB_ENV
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can update-alternatives in conjunction with pkg-config not be used here?


- name: Install common dependencies
run: |
sudo apt-get -y update
sudo apt-get -y install autoconf automake libfftw3-dev libgsl-dev liblapack-dev guile-3.0-dev libpng-dev libtool swig
git clone https://github.com/NanoComp/libctl.git libctl-src
(cd libctl-src && git checkout master && sh autogen.sh --prefix=${HOME}/local --enable-shared && make -j 2 && make install)
git clone https://github.com/NanoComp/harminv.git
(cd harminv && git checkout master && sh autogen.sh --prefix=${HOME}/local --enable-shared && make -j 2 && make install)
git clone https://github.com/NanoComp/mpb.git
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider the checkout action instead of cloning the full repo:
https://github.com/actions/checkout

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could also just do a git clone --depth 1 (shallow clone).

(cd mpb && git checkout master && sh autogen.sh --prefix=${HOME}/local --enable-shared LIBS=-ldl --with-libctl=${HOME}/local/share/libctl --with-hermitian-eps && make -j 2 && make install)
git clone https://github.com/HomerReid/libGDSII.git
(cd libGDSII && git checkout master && sh autogen.sh --prefix=${HOME}/local && make install)

- name: Define environment variables for serial build
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-j$(nproc) seems better than hardcoding

if: ${{ matrix.enable-mpi == false }}
run: |
echo "MKCHECKFLAGS=-j 2" >> $GITHUB_ENV
echo "CPPFLAGS=${HDF5_SERIAL_CPPFLAGS} ${CPPFLAGS}" >> $GITHUB_ENV
echo "LDFLAGS=${HDF5_SERIAL_LDFLAGS} ${LDFLAGS}" >> $GITHUB_ENV

- name: Define environment variables for MPI build
if: matrix.enable-mpi
run: |
sudo apt-get install libhdf5-openmpi-dev libopenmpi-dev
sudo update-alternatives --set hdf5.pc /usr/lib/x86_64-linux-gnu/pkgconfig/hdf5-openmpi.pc
echo "MKCHECKFLAGS=" >> $GITHUB_ENV
echo "CPPFLAGS=${HDF5_PARALLEL_CPPFLAGS} ${CPPFLAGS}" >> $GITHUB_ENV
echo "LDFLAGS=${HDF5_PARALLEL_LDFLAGS} ${LDFLAGS}" >> $GITHUB_ENV
echo "CC=mpicc" >> $GITHUB_ENV
echo "CXX=mpic++" >> $GITHUB_ENV
echo "HDF5_MPI=ON" >> $GITHUB_ENV

- name: Install serial dependencies
if: ${{ matrix.enable-mpi == false }}
run: sudo apt-get -y install libhdf5-serial-dev

- name: Install MPI dependencies
if: matrix.enable-mpi
run: sudo apt-get -y install libopenmpi-dev mpi-default-bin openmpi-bin libhdf5-openmpi-dev libaec-dev

- name: Check out repository
uses: actions/checkout@v2

- name: Set up Python (version ${{ matrix.python-version }})
if: ${{ matrix.python-version != 'none' }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}

- name: Install Python dependencies
if: ${{ matrix.python-version != 'none' }}
run: pip install autograd h5py jax jaxlib matplotlib mpi4py numpy parameterized pytest scipy
- name: Autoreconf
run: autoreconf -if
- name: configure
run: ./configure --with-hdf5 --without-scheme --enable-shared ${{ matrix.configure-options }}
- name: make
run: make
- name: Run Python tests
if: ${{ matrix.python-version != 'none' }}
env:
MEEP_SKIP_LARGE_TESTS: 1
run: pip install autograd h5py jax jaxlib matplotlib numpy parameterized pytest scipy

- name: Install mpi4py
if: matrix.enable-mpi
run: pip install mpi4py

- name: Run autoreconf
run: |
export PYTHONPATH="${PWD}/python:${PYTHONPATH}"
pytest python/tests
- name: Run C++ tests
if: ${{ matrix.python-version == 'none' }}
run: make check
autoreconf --verbose --install --symlink --force
MEEP_VERSION=$(./configure -V | grep meep | awk '{print $3}')
echo "MEEP_VERSION=${MEEP_VERSION}" >> $GITHUB_ENV

- name: Run configure
run: |
mkdir -p build && pushd build
../configure --enable-maintainer-mode --prefix=${HOME}/local --with-libctl=${HOME}/local/share/libctl ${{ matrix.configure-options }}

- name: Run make distcheck
run: pushd build && make ${MKCHECKFLAGS} distcheck DISTCHECK_CONFIGURE_FLAGS="--with-libctl=${HOME}/local/share/libctl ${{ matrix.configure-options }}"