Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move to ghactions #13

Merged
merged 47 commits into from
Jan 8, 2021
Merged
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
689214f
Remove Travis-specific macros
tjurtsch Dec 9, 2020
61852f5
Replace Travis' env variables with GH Actions counterparts
tjurtsch Dec 9, 2020
71c0d6a
Remove Travis-specific comments
tjurtsch Dec 9, 2020
b4f3246
windows: dump use of winpty
tjurtsch Dec 9, 2020
911fefc
fixup: git reset at desired revision, not checkout
tjurtsch Dec 9, 2020
1721ed8
Rename .travis-output.py to .ci-output.py
tjurtsch Dec 9, 2020
1c94eb1
Update macOS SDK location
tjurtsch Dec 9, 2020
f170a02
install.sh: limit additional channels
tjurtsch Dec 9, 2020
0a4376f
Create directory before generating conda env in it
tjurtsch Dec 9, 2020
92458e3
Update README.md
tjurtsch Dec 15, 2020
7547186
Install pexpect
tjurtsch Dec 16, 2020
5158c1d
Add Github action definition
tjurtsch Dec 21, 2020
865f4c5
Initialize environment only when conda-meta exists
tjurtsch Dec 22, 2020
48d51ac
Revert "install.sh: limit additional channels"
tjurtsch Dec 23, 2020
767d83d
Use GITHUB_REPOSITORY to extract owner name
tjurtsch Dec 23, 2020
1b3dc51
Correct channel labels
tjurtsch Dec 23, 2020
d3ebc64
Force upload to temporary label
tjurtsch Dec 28, 2020
cc37d7a
Remove fixup-git.sh
tjurtsch Dec 28, 2020
45056ab
action.yml: little cleanup
tjurtsch Dec 28, 2020
5642fc7
Add master-package script handler
tjurtsch Dec 28, 2020
70b0e37
master-package: source common.sh
tjurtsch Dec 29, 2020
b078359
Correct remaining Travis' env variables
tjurtsch Dec 29, 2020
9e4a787
Run cleanup-anaconda after master-package
tjurtsch Dec 29, 2020
fcfb19e
cleanup-anaconda: set missing variable
tjurtsch Dec 29, 2020
ca3f9ef
Execute inline build script if it was defined in workflow
tjurtsch Dec 29, 2020
8c7d40f
Ensure that all the other jobs were completed before
tjurtsch Dec 31, 2020
e118870
Export windows related paths conditionally
tjurtsch Dec 31, 2020
fc2991a
cleanup-anaconda: increase verbosity
tjurtsch Dec 31, 2020
8f00165
cleanup-anaconda: correct date switch
tjurtsch Dec 31, 2020
7d89aff
cleanup-anaconda: correct cut delimiter
tjurtsch Dec 31, 2020
1035db3
ghactions-api: raise limit of jobs per page in API call
tjurtsch Jan 4, 2021
23289cf
Return error when anaconda/token doesn't work
tjurtsch Jan 5, 2021
ce25d23
test_anaconda: don't exit when token works
tjurtsch Jan 5, 2021
6aed4b1
Remove unused scripts
tjurtsch Jan 7, 2021
b24e481
wip: check if we are inside correct environment
tjurtsch Jan 7, 2021
ced6347
Simplify path used to check Conda env existance
tjurtsch Jan 7, 2021
173f2df
common.sh: remove code repetition
tjurtsch Jan 7, 2021
b3da438
Update README.md
tjurtsch Jan 7, 2021
25ff508
Return an error when number of jobs exceeds max_jobs
tjurtsch Jan 7, 2021
e35959c
Use ghactions ::group:: mechanism wherever travis_fold was used
tjurtsch Jan 7, 2021
dc0ee40
after_failure: remove commented out code
tjurtsch Jan 7, 2021
3d0779d
Look for failure statuses in workflow jobs and act accordingly
tjurtsch Jan 7, 2021
c404f20
wait-for-statuses: apply more pythonic path joining
tjurtsch Jan 7, 2021
99e8cd3
wait-for-statuses: fix a typo
tjurtsch Jan 7, 2021
735fdba
wait-for-statuses: add missing failure handling
tjurtsch Jan 7, 2021
d188f68
wait-for-statuses: treat everything but success as failure
tjurtsch Jan 7, 2021
7696131
wait-for-statuses: look for failing jobs when all jobs completed
tjurtsch Jan 7, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
File renamed without changes.
17 changes: 1 addition & 16 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,5 @@
# litex-conda-ci

Travis scripts for Litex-Hub/litex-conda-(compilers|eda|misc|prog) .
Scripts for Litex-Hub/litex-conda-(compilers|eda|misc|prog) Continuous Integration.

This repository is not meant to be used on its own, but as submodule for above repositories.

## Setting up Travis for your own fork

To use any of the litex-conda-* repositories in your own fork with Travis you must meet the following requirements:

* enable Travis for your repository
* create an account on https://anaconda.org/
* create an access token on https://anaconda.org/{your-account}/settings/access
* visit https://travis-ci.com/github/{your-account}/litex-conda-{suffix}/settings
* add two environment variables:
* `ANACONDA_USER` - set to your anaconda username
* `ANACONDA_TOKEN` - set to the value of the token you created
tjurtsch marked this conversation as resolved.
Show resolved Hide resolved
* remember to **disable** the `Display value in build log` option for `ANACONDA_TOKEN`
* create a daily or weekly Cron Job on the same settings page, to start on the master branch
* this will ensure removal of temporary labels on your Anaconda account (they are created during the build)
29 changes: 29 additions & 0 deletions action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
name: 'Package build'
description: 'Build a package'
runs:
using: "composite"
steps:
- name: build
run: |
if [ $SKIP ]; then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI - there's no "allow-failure" mechanism known from Travis. @umarcor are you by any chance aware of something better?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, that's a huge missing feature in GitHub Actions. There is no replacement. You can allow subsequent steps and/or jobs to continue execution after one fails, but the result will be red (failure). AFAIAA, it's not possible to expect failures and have the result green.

In most projects, allowing failures is handled through pytest or some other test runner.

exit 0;
fi
if [ $OS_NAME = "windows" ]; then
export PATH="$PATH:/c/ProgramData/Chocolatey/bin/"
export PATH="$PATH:/c/Program Files/Git/usr/bin/"
fi
source .github/scripts/common.sh
bash .github/scripts/download_sdk.sh
bash .github/scripts/install.sh
set -x
if [ $SCRIPT ]; then
bash $SCRIPT
else
bash .github/scripts/script.sh
fi
if [ $? -eq 0 ]; then
source .github/scripts/after_success.sh
else
source .github/scripts/after_failure.sh
fi
shell: bash
5 changes: 1 addition & 4 deletions after_failure.sh
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
#!/bin/bash

source $TRAVIS_BUILD_DIR/.travis/common.sh

# Close the after_failure fold travis has created already.
travis_fold end after_failure
source $GITHUB_WORKSPACE/.github/scripts/common.sh

$SPACER

Expand Down
13 changes: 5 additions & 8 deletions after_success.sh
Original file line number Diff line number Diff line change
@@ -1,29 +1,26 @@
#!/bin/bash

source ./.travis/common.sh
source $GITHUB_WORKSPACE/.github/scripts/common.sh
set -e

# Close the after_success fold travis has created already.
travis_fold end after_success
tjurtsch marked this conversation as resolved.
Show resolved Hide resolved

if [[ $UPLOAD == "no-upload" ]]; then
echo "Job without upload..."
else
echo "Job with Conda upload..."

$SPACER
# Travis will not expose the ANACONDA_TOKEN var for pull requests coming from other forks than the original one
if [ x$ANACONDA_TOKEN != x ]; then
# `anaconda-client` is installed in the `base` environment
conda activate base

start_section "package.upload" "${GREEN}Package uploading...${NC}"
# Test `anaconda` with ANACONDA_TOKEN before uploading
source $TRAVIS_BUILD_DIR/.travis/test_anaconda.sh
anaconda -t $ANACONDA_TOKEN upload --no-progress --user $ANACONDA_USER --label travis-${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}-$TRAVIS_BUILD_ID $CONDA_OUT
source $GITHUB_WORKSPACE/.github/scripts/test_anaconda.sh
branch="$(git rev-parse --abbrev-ref HEAD)"
anaconda -t $ANACONDA_TOKEN upload --force --no-progress --user $ANACONDA_USER --label ci-$branch-$GITHUB_RUN_ID $CONDA_OUT
end_section "package.upload"
else
echo "ANACONDA_TOKEN not found. Please consult README of litex-conda-ci for details on setting up Travis tests properly."
echo "ANACONDA_TOKEN not found. Please consult README of litex-conda-ci for details on setting up tests properly."
echo "Packages cannot be uploaded when tests are running for cross-repository Pull Requests."
fi

Expand Down
5 changes: 1 addition & 4 deletions before_cache.sh
Original file line number Diff line number Diff line change
@@ -1,11 +1,8 @@
#!/bin/bash

source ./.travis/common.sh
source $GITHUB_WORKSPACE/.github/scripts/common.sh
set -e

# Close the after_success.1 fold travis has created already.
travis_fold end before_cache

start_section "conda.clean.1" "${GREEN}Clean status...${NC}"
#conda clean -s --dry-run
end_section "conda.clean.1"
Expand Down
14 changes: 11 additions & 3 deletions cleanup-anaconda.sh
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1,19 +1,27 @@
#!/bin/bash

source $GITHUB_WORKSPACE/.github/scripts/common.sh
# `for ... in $(anaconda ...` fails silently if there's any problem with anaconda
source $TRAVIS_BUILD_DIR/.travis/test_anaconda.sh
source $GITHUB_WORKSPACE/.github/scripts/test_anaconda.sh
set -x

#if the timestamp is older than one week, remove the whole label
ago="7 days ago"

if [ $OS_NAME = 'osx' ]; then
tjurtsch marked this conversation as resolved.
Show resolved Hide resolved
DATE_SWITCH="-r "
else
DATE_SWITCH="--date "
fi

#extract date in milliseconds
limit_date=$(date $DATE_SWITCH "$ago" +'%s%N' | cut -b1-13)

echo "Will remove labels older than $limit_date timestamp"

for label in $(anaconda -t $ANACONDA_TOKEN label --list -o litex-hub 2>&1 | grep ' + ' | cut -f2 -d1+)
for label in $(anaconda -t $ANACONDA_TOKEN label --list -o litex-hub 2>&1 | grep ' + ' | cut -f2 -d+)
do
if [[ $label != travis* ]]
if [[ $label != ci* ]]
then
continue
fi
Expand Down
42 changes: 15 additions & 27 deletions common.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,9 @@ NC='\033[0m' # No Color

SPACER="echo -e ${GRAY} - ${NC}"

export -f travis_nanoseconds
export -f travis_fold
export -f travis_time_start
export -f travis_time_finish
CI_MAX_TIME=50

TRAVIS_MAX_TIME=50

# Override default travis_wait to pipe the output
travis_wait() {
ci_wait() {
local timeout="${1}"

if [[ "${timeout}" =~ ^[0-9]+$ ]]; then
Expand All @@ -27,12 +21,12 @@ travis_wait() {
fi

local cmd=("${@}")
local log_file="travis_wait_${$}.log"
local log_file="ci_wait_${$}.log"

"${cmd[@]}" &
local cmd_pid="${!}"

travis_jigger "${!}" "${timeout}" "${cmd[@]}" &
ci_jigger "${!}" "${timeout}" "${cmd[@]}" &
local jigger_pid="${!}"
local result

Expand All @@ -53,8 +47,7 @@ travis_wait() {
return "${result}"
}

# Override default travis_jigger to print invisible character to keep build alive
travis_jigger() {
ci_jigger() {
local cmd_pid="${1}"
shift
local timeout="${1}"
Expand All @@ -74,7 +67,7 @@ travis_jigger() {
kill -9 "${cmd_pid}"
}

if [ $TRAVIS_OS_NAME = 'osx' ]; then
if [ $OS_NAME = 'osx' ]; then
DATE_SWITCH="-r "
else
DATE_SWITCH="--date=@"
Expand All @@ -88,34 +81,30 @@ if [ -z "$DATE_STR" ]; then
fi

function start_section() {
travis_fold start "$1"
travis_time_start
echo -e "${PURPLE}${PACKAGE}${NC}: - $2${NC}"
echo -e "${GRAY}-------------------------------------------------------------------${NC}"
}

function end_section() {
echo -e "${GRAY}-------------------------------------------------------------------${NC}"
travis_time_finish
travis_fold end "$1"
}

# Disable this warning;
# xxxx/conda_build/environ.py:377: UserWarning: The environment variable
# 'TRAVIS' is being passed through with value 0. If you are splitting
# build and test phases with --no-test, please ensure that this value is
# also set similarly at test time.
export PYTHONWARNINGS=ignore::UserWarning:conda_build.environ

export BASE_PATH="/tmp/really-long-path"
mkdir -p "$BASE_PATH"
if [ $TRAVIS_OS_NAME = 'windows' ]; then
if [ $OS_NAME = 'windows' ]; then
export CONDA_PATH='/c/tools/miniconda3'
export PATH=$CONDA_PATH/Scripts/:$CONDA_PATH/:$PATH

# It is much shorter than '$PWD/workdir/conda-env' which in the end (+conda-bld/...)
# causes some build paths to exceed 255 chars (e.g. during prjtrellis building)
export CONDA_ENV='/c/Users/travis/conda-env'
RUNNER_DIR='/c/Users/runner/'
if [ ! -d "$RUNNER_DIR" ]; then
mkdir "$RUNNER_DIR"
fi

export CONDA_ENV="$RUNNER_DIR/conda-env"
if [ -d 'workdir/conda-env' ]; then
mv 'workdir/conda-env' "$CONDA_ENV"
fi
Expand All @@ -125,7 +114,7 @@ else
export CONDA_ENV='workdir/conda-env'
fi

if [ -d "$CONDA_ENV" ] && which conda &>/dev/null; then
if [ -d "$CONDA_ENV/conda-meta" ] && which conda &>/dev/null; then
tjurtsch marked this conversation as resolved.
Show resolved Hide resolved
# >>> conda initialize >>>
eval "$('conda' 'shell.bash' 'hook' 2> /dev/null)"
# <<< conda initialize <<<
Expand All @@ -141,9 +130,8 @@ if [ -d "workdir/recipe" ]; then
export CONDA_BUILD_ARGS="$EXTRA_BUILD_ARGS workdir/recipe"
export CONDA_OUT="$(conda render --output $CONDA_BUILD_ARGS | grep conda-bld | grep tar.bz2 | tail -n 1 | sed -e's/-[0-9]\+\.tar/*.tar/' -e's/-git//')"

if [ "$TRAVIS_OS_NAME" = 'windows' ]; then
if [ "$OS_NAME" = 'windows' ]; then
# conda render outputs Windows-style path which may contain wildcards;
# 'git bash' used by Travis works well with wildcards only in Unix-style paths
export CONDA_OUT="$(cygpath -u $CONDA_OUT)"
fi
fi
Expand Down
24 changes: 9 additions & 15 deletions conda-env.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,6 @@ if ! which conda; then
export PATH=~/conda/bin:$PATH
fi

# Disable this warning;
# xxxx/conda_build/environ.py:377: UserWarning: The environment variable
# 'TRAVIS' is being passed through with value 0. If you are splitting
# build and test phases with --no-test, please ensure that this value is
# also set similarly at test time.
export PYTHONWARNINGS=ignore::UserWarning:conda_build.environ

if [ -z "$DATE_STR" ]; then
Expand All @@ -22,21 +17,20 @@ if [ -z "$GITREV" ]; then
echo "Setting git revision $GITREV"
fi

export TRAVIS=0
export CI=0

export TRAVIS_EVENT_TYPE="local"
echo "TRAVIS_EVENT_TYPE='${TRAVIS_EVENT_TYPE}'"
export CI_EVENT_TYPE="local"
echo "CI_EVENT_TYPE='${CI_EVENT_TYPE}'"

export TRAVIS_BRANCH="$(git rev-parse --abbrev-ref HEAD)"
echo "TRAVIS_BRANCH='${TRAVIS_BRANCH}'"
export CI_BRANCH="$(git rev-parse --abbrev-ref HEAD)"
echo "CI_BRANCH='${CI_BRANCH}'"

export TRAVIS_COMMIT="$(git rev-parse HEAD)"
echo "TRAVIS_COMMIT='${TRAVIS_COMMIT}'"
export CI_COMMIT="$(git rev-parse HEAD)"
echo "CI_COMMIT='${CI_COMMIT}'"

export TRAVIS_REPO_SLUG="$(git rev-parse --abbrev-ref --symbolic-full-name @{u})"
echo "TRAVIS_REPO_SLUG='${TRAVIS_REPO_SLUG}'"
export CI_REPO_SLUG="$(git rev-parse --abbrev-ref --symbolic-full-name @{u})"
echo "CI_REPO_SLUG='${CI_REPO_SLUG}'"

./.travis/conda-meta-extra.sh
$GITHUB_WORKSPACE/.github/scripts/conda-meta-extra.sh
echo conda $@
conda $@
4 changes: 2 additions & 2 deletions conda-get.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@ set -x
set -e

CONDA_PATH=${1:-~/conda}
if [ $TRAVIS_OS_NAME = 'windows' ]; then
if [ $OS_NAME = 'windows' ]; then
if [ ! -d $CONDA_PATH -o ! -z "$CI" ]; then
choco install openssl.light
choco install miniconda3 --params="/AddToPath:1"
fi
export CONDA_PATH='/c/tools/miniconda3'
export PATH=$CONDA_PATH/bin/:$CONDA_PATH/Scripts/:$PATH
else
if [ $TRAVIS_OS_NAME = 'linux' ]; then
if [ $OS_NAME = 'linux' ]; then
sys_name=Linux
else
sys_name=MacOSX
Expand Down
14 changes: 7 additions & 7 deletions conda-meta-extra.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,14 @@ extra:
maintainers:
- Tim 'mithro' Ansell <mithro@mithis.com>
- HDMI2USB Project - https://hdmi2usb.tv <hdmi2usb@googlegroups.com>
travis:
job_id: $TRAVIS_JOB_ID
job_num: $TRAVIS_JOB_NUMBER
type: $TRAVIS_EVENT_TYPE
ci:
job_id: $GITHUB_RUN_ID
job_num: $GITHUB_RUN_NUMBER
type: $CI_EVENT_TYPE
recipe:
repo: 'https://github.com/$TRAVIS_REPO_SLUG'
branch: $TRAVIS_BRANCH
commit: $TRAVIS_COMMIT
repo: 'https://github.com/$CI_REPO_SLUG'
branch: $CI_BRANCH
commit: $CI_COMMIT
describe: $GITREV
date: $DATESTR
EOF
Expand Down
2 changes: 1 addition & 1 deletion conda_build_config.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
CONDA_BUILD_SYSROOT:
- /Users/travis/sdk/MacOSX10.9.sdk # [osx]
- /Users/runner/sdk/MacOSX10.9.sdk # [osx]

2 changes: 1 addition & 1 deletion download_sdk.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
if [ $TRAVIS_OS_NAME = 'osx' ]; then
if [ $OS_NAME = 'osx' ]; then
if [[ ! -d $HOME/sdk/MacOSX10.9.sdk ]]; then
git clone https://github.com/phracker/MacOSX-SDKs $HOME/sdk
fi
Expand Down
23 changes: 0 additions & 23 deletions fixup-git.sh

This file was deleted.

Loading