Skip to content
Dataset Validator

Update validator.yml to fix maintenance issues (dependencies, disk space, error throwing) #189

Sign in to view logs

Update validator.yml to fix maintenance issues (dependencies, disk space, error throwing)

Update validator.yml to fix maintenance issues (dependencies, disk space, error throwing) #189

Workflow file for this run

.github/workflows/validator.yml at 9ca106d
name: Dataset Validator
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
Validate:
runs-on: ubuntu-latest
steps:
- name: Gain disk space by remapping temp disk (`/dev/sdb1`)
uses: easimon/maximize-build-space@master
# We use the above action instead of removing unwanted packages because it only takes about 2s;
# by comparison, removing lots of small file is very time-consuming.
with:
# After freeing up space we could have as much as 87GB available, which is overkill.
# So, we set root == 10GB to save room for installing dependencies (leaving us with ~77GB)
root-reserve-mb: 10240
swap-size-mb: 1024
- name: Checkout
uses: actions/checkout@v4
with:
# make sure to download directly from the PR's repo, whether that is this repo or a fork
# By default github generates a merge commit for each PR in this repo, but only for the one branch under test
# but `git-annex` needs access to *two* branches: the current branch and `git-annex`
# this might be subtly buggy since it is testing the remote version, not the merged version
#
# *if* this is not a pull request, this will fall back to its default behaviour.
repository: ${{github.event.pull_request.head.repo.full_name}}
ref: ${{ github.event.pull_request.head.ref }}
- name: Update apt packages
run: |
# do we want to do this? it's helpful to avoid testing against surprise out-of-date software, but also so slow.
sudo apt-get update &&
sudo DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade &&
sudo DEBIAN_FRONTEND=noninteractive apt-get autoremove
- name: Install git-annex
run: |
sudo apt-get install -y git-annex
git config --global annex.thin true
git config --global annex.retry 2
- name: Setup NodeJS (for bids-validator)
uses: actions/setup-node@v4
- name: Install bids-validator
run: sudo npm install -g bids-validator
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.11
- name: Install spine-generic for acquisition parameters check
run: pip install spinegeneric@git+https://github.com/spine-generic/spine-generic.git@master
- name: git config
run: |
# this is needed by git-annex so that it can write to the local git-annex branch
# the tracking information about the local copy
git config --global user.name "gh-actions"
git config --global user.email "gh-actions"
# but actually, disable those writes. We don't need them because we're throwing away this copy immediately.
# this might make the previous two lines unnecessary, but it's safer just to leave them both in
git config annex.alwayscommit false
- name: Download dataset
run: |
git fetch --depth=1 origin git-annex:git-annex # actions/checkout@v4 does a shallow checkout, so it is missing this important branch
git annex init
git annex get -J8
- name: Checking BIDS compliance
if: always()
run: |
bids-validator --verbose ./ | tee bids.txt
printf "\nShort summary:\n\n"
# FIXME: This should be replaced with bids-validator settings: A) throw error on warning or B) warning ignores
! grep "\[WARN\]\|\[ERR\]" bids.txt # ! -> Reverse error code so "found == error thrown"
- name: Checking acquisition parameters
if: always()
run: |
sg_params_checker -path-in ./ | tee params.txt
# FIXME: Checking the output like this should be replaced with SG throwing actual errors
! grep "WARNING" params.txt # ! -> Reverse error code so "found == error thrown"
- name: Checking data consistency
if: always()
run: |
sg_check_data_consistency -path-in ./ | tee consistency.txt
# FIXME: Checking the output like this should be replaced with SG throwing actual errors
! grep "Warning\|Missing" consistency.txt
grep "all good" consistency.txt