Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various CI fixes and cleanup #8289

Merged
merged 21 commits into from
May 5, 2020
Merged

Various CI fixes and cleanup #8289

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
9058e41
Add --fail flag to curl calls
web-flow May 4, 2020
0a94d26
Don't update Python unless the version actually differs
web-flow May 2, 2020
2e8e98f
Make Redis build quieter
web-flow May 4, 2020
85adfdb
Simplify install-strace.sh
web-flow May 2, 2020
06ecb67
Move install-cython-examples.sh into ci.sh
web-flow May 2, 2020
4dec6cb
Move install-ray.sh and install_npm_project into ci.sh
web-flow May 2, 2020
9f8683e
Simplify Go installation
web-flow May 2, 2020
756ba91
Make CI less noisy
web-flow May 2, 2020
13e2bda
Install Bazel to /usr/local/bin, move PYTHON3_BIN_PATH to a patch, an…
web-flow May 2, 2020
e82e7f5
Move lint before build so that it finishes faster
web-flow May 4, 2020
9fafc02
Merge stdout and stderr on GitHub Actions
web-flow May 4, 2020
216c404
Update reload_env, simplify code, and update build documentation
web-flow May 2, 2020
61ea87e
Translate some more GitHub Actions environment variables
web-flow May 4, 2020
73e6073
Rename build sections to allow before_install to come earlier
web-flow May 4, 2020
d1bc395
Update Anaconda and use proper conda activation instead of relying on…
web-flow May 2, 2020
25a6a16
web-flow May 4, 2020
9fbf6bb
web-flow May 4, 2020
e549b20
web-flow May 4, 2020
3756428
Lower CI script line lengths
web-flow May 5, 2020
696d5a8
Register LLVM installation instead of making Bash stubs
web-flow May 5, 2020
cff0919
Merge remote-tracking branch 'remotes/upstream/master' into ci-cleanup
web-flow May 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .bazelrc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
build --enable_platform_specific_config
###############################################################################
# On Windows, provide: USE_CLANG_CL=1, CC=clang, BAZEL_LLVM, BAZEL_SH
# On all platforms, provide: PYTHON2_BIN_PATH, PYTHON3_BIN_PATH
# On all platforms, provide: PYTHON3_BIN_PATH=python
###############################################################################
build --action_env=PATH
build:linux --compilation_mode=opt
Expand Down
6 changes: 5 additions & 1 deletion .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@ name: CI

on: [push, pull_request]

env:
GITHUB_BASE_SHA: ${{ github.event.pull_request.base.sha }}
GITHUB_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
GITHUB_PULL_REQUEST: ${{ github.event.number }}

Comment on lines +7 to +10
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a comment saying where these are used?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, they're used for translating Travis environment variables, I'll update them.

jobs:
build:
name: ${{ matrix.name }}
Expand All @@ -18,7 +23,6 @@ jobs:
- name: macos
os: macos-10.15
env:
GITHUB_PULL_REQUEST: ${{ github.event.number }}
PYTHON: 3.6
TRAVIS_BUILD_DIR: ${{ github.workspace }}
TRAVIS_COMMIT: ${{ github.sha }}
Expand Down
74 changes: 39 additions & 35 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@ language: generic
# Use Ubuntu 16.04
dist: xenial

before_install:
- unset -f cd # Travis defines this on Mac for RVM, but it floods the trace log and isn't relevant for us

matrix:
include:
- os: linux
Expand All @@ -10,9 +13,9 @@ matrix:
- PYTHONWARNINGS=ignore
- RAY_DEFAULT_BUILD=1
- RAY_CYTHON_EXAMPLES=1
before_install:
- . ./ci/travis/ci.sh init RAY_CI_SERVE_AFFECTED,RAY_CI_TUNE_AFFECTED,RAY_CI_PYTHON_AFFECTED
install:
- . ./ci/travis/ci.sh init RAY_CI_SERVE_AFFECTED,RAY_CI_TUNE_AFFECTED,RAY_CI_PYTHON_AFFECTED
before_script:
- . ./ci/travis/ci.sh build

- os: osx
Expand All @@ -22,9 +25,9 @@ matrix:
- PYTHONWARNINGS=ignore
- RAY_DEFAULT_BUILD=1
- RAY_CYTHON_EXAMPLES=1
before_install:
- . ./ci/travis/ci.sh init RAY_CI_SERVE_AFFECTED,RAY_CI_TUNE_AFFECTED,RAY_CI_PYTHON_AFFECTED
install:
- . ./ci/travis/ci.sh init RAY_CI_SERVE_AFFECTED,RAY_CI_TUNE_AFFECTED,RAY_CI_PYTHON_AFFECTED
before_script:
- . ./ci/travis/ci.sh build

- os: linux
Expand All @@ -33,9 +36,9 @@ matrix:
- PYTHON=3.6 PYTHONWARNINGS=ignore
- RAY_INSTALL_JAVA=1
- RAY_GCS_ACTOR_SERVICE_ENABLED=true
before_install:
- . ./ci/travis/ci.sh init RAY_CI_JAVA_AFFECTED
install:
- . ./ci/travis/ci.sh init RAY_CI_JAVA_AFFECTED
before_script:
- . ./ci/travis/ci.sh build
script:
- ./java/test.sh
Expand All @@ -47,9 +50,9 @@ matrix:
- RAY_INSTALL_JAVA=1
- RAY_GCS_ACTOR_SERVICE_ENABLED=true
- PYTHON=3.6 PYTHONWARNINGS=ignore
before_install:
- . ./ci/travis/ci.sh init RAY_CI_STREAMING_PYTHON_AFFECTED,RAY_CI_STREAMING_JAVA_AFFECTED
install:
- . ./ci/travis/ci.sh init RAY_CI_STREAMING_PYTHON_AFFECTED,RAY_CI_STREAMING_JAVA_AFFECTED
before_script:
- . ./ci/travis/ci.sh build
script:
# Streaming cpp test.
Expand All @@ -65,9 +68,9 @@ matrix:
- RAY_INSTALL_JAVA=1
- RAY_GCS_SERVICE_ENABLED=false
- RAY_CYTHON_EXAMPLES=1
before_install:
- . ./ci/travis/ci.sh init RAY_CI_ONLY_RLLIB_AFFECTED
install:
- . ./ci/travis/ci.sh init RAY_CI_ONLY_RLLIB_AFFECTED
before_script:
- . ./ci/travis/ci.sh build
script:
- ./ci/suppress_output bash src/ray/test/run_core_worker_tests.sh
Expand All @@ -82,9 +85,9 @@ matrix:
- RAY_INSTALL_JAVA=1
- RAY_GCS_SERVICE_ENABLED=false
- RAY_CYTHON_EXAMPLES=1
before_install:
- . ./ci/travis/ci.sh init RAY_CI_ONLY_RLLIB_AFFECTED
install:
- . ./ci/travis/ci.sh init RAY_CI_ONLY_RLLIB_AFFECTED
before_script:
- . ./ci/travis/ci.sh build
script:
- ./ci/keep_alive bazel test --config=ci --test_tag_filters=-jenkins_only python/ray/tests/...
Expand All @@ -93,12 +96,13 @@ matrix:
env:
- LINT=1
- PYTHONWARNINGS=ignore
before_install:
- . ./ci/travis/ci.sh init
install:
- . ./ci/travis/ci.sh init
before_script:
- . ./ci/travis/ci.sh lint
- . ./ci/travis/ci.sh build
script:
- . ./ci/travis/ci.sh lint
- true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we are just skipping, we can just remove the script block right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sadly not :( if you do that then it falls back to the default outermost block. I think I can technically put script: [] but it's not clear to me why that works, it's kind of confusing.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Add a comment so people maintaining this in the future knows?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure


# Build Linux wheels.
- os: linux
Expand All @@ -107,9 +111,9 @@ matrix:
- PYTHONWARNINGS=ignore
- RAY_INSTALL_JAVA=1
- RAY_GCS_ACTOR_SERVICE_ENABLED=true
before_install:
- . ./ci/travis/ci.sh init RAY_CI_LINUX_WHEELS_AFFECTED
install:
- . ./ci/travis/ci.sh init RAY_CI_LINUX_WHEELS_AFFECTED
before_script:
- . ./ci/travis/ci.sh build
script:
- . ./ci/travis/ci.sh test_wheels
Expand All @@ -123,9 +127,9 @@ matrix:
- PYTHONWARNINGS=ignore
- RAY_INSTALL_JAVA=1
- RAY_GCS_ACTOR_SERVICE_ENABLED=true
before_install:
- . ./ci/travis/ci.sh init RAY_CI_MACOS_WHEELS_AFFECTED
install:
- . ./ci/travis/ci.sh init RAY_CI_MACOS_WHEELS_AFFECTED
before_script:
- . ./ci/travis/ci.sh build
script:
- . ./ci/travis/ci.sh test_wheels
Expand All @@ -140,9 +144,9 @@ matrix:
- TORCH_VERSION=1.4
- PYTHON=3.6
- PYTHONWARNINGS=ignore
before_install:
- . ./ci/travis/ci.sh init RAY_CI_RLLIB_AFFECTED
install:
- . ./ci/travis/ci.sh init RAY_CI_RLLIB_AFFECTED
before_script:
- . ./ci/travis/ci.sh build
script:
- travis_wait 90 bazel test --config=ci --test_output=streamed --build_tests_only --test_tag_filters=learning_tests_tf rllib/...
Expand All @@ -158,9 +162,9 @@ matrix:
- TORCH_VERSION=1.4
- PYTHON=3.6
- PYTHONWARNINGS=ignore
before_install:
- . ./ci/travis/ci.sh init RAY_CI_RLLIB_FULL_AFFECTED
install:
- . ./ci/travis/ci.sh init RAY_CI_RLLIB_FULL_AFFECTED
before_script:
- . ./ci/travis/ci.sh build
script:
- travis_wait 90 bazel test --config=ci --test_output=streamed --build_tests_only --test_tag_filters=learning_tests_tf rllib/...
Expand All @@ -175,9 +179,9 @@ matrix:
- TORCH_VERSION=1.4
- PYTHON=3.6
- PYTHONWARNINGS=ignore
before_install:
- . ./ci/travis/ci.sh init RAY_CI_RLLIB_AFFECTED
install:
- . ./ci/travis/ci.sh init RAY_CI_RLLIB_AFFECTED
before_script:
- . ./ci/travis/ci.sh build
script:
- travis_wait 90 bazel test --config=ci --test_output=streamed --build_tests_only --test_tag_filters=learning_tests_torch rllib/...
Expand All @@ -193,9 +197,9 @@ matrix:
- TFP_VERSION=0.8
- TORCH_VERSION=1.4
- PYTHONWARNINGS=ignore
before_install:
- . ./ci/travis/ci.sh init RAY_CI_RLLIB_FULL_AFFECTED
install:
- . ./ci/travis/ci.sh init RAY_CI_RLLIB_FULL_AFFECTED
before_script:
- . ./ci/travis/ci.sh build
script:
- travis_wait 60 bazel test --config=ci --build_tests_only --test_tag_filters=quick_train rllib/...
Expand All @@ -213,9 +217,9 @@ matrix:
- TFP_VERSION=0.8
- TORCH_VERSION=1.4
- PYTHONWARNINGS=ignore
before_install:
- . ./ci/travis/ci.sh init RAY_CI_RLLIB_FULL_AFFECTED
install:
- . ./ci/travis/ci.sh init RAY_CI_RLLIB_FULL_AFFECTED
before_script:
- . ./ci/travis/ci.sh build
script:
- ./ci/keep_alive bazel test --config=ci --build_tests_only --test_tag_filters=examples_A,examples_B rllib/...
Expand All @@ -233,9 +237,9 @@ matrix:
- TFP_VERSION=0.8
- TORCH_VERSION=1.4
- PYTHONWARNINGS=ignore
before_install:
- . ./ci/travis/ci.sh init RAY_CI_RLLIB_FULL_AFFECTED
install:
- . ./ci/travis/ci.sh init RAY_CI_RLLIB_FULL_AFFECTED
before_script:
- . ./ci/travis/ci.sh build
script:
- ./ci/keep_alive bazel test --config=ci --build_tests_only --test_tag_filters=tests_dir_A,tests_dir_B,tests_dir_C,tests_dir_D,tests_dir_E,tests_dir_F,tests_dir_G,tests_dir_H,tests_dir_I rllib/...
Expand All @@ -250,9 +254,9 @@ matrix:
- TFP_VERSION=0.8
- TORCH_VERSION=1.4
- PYTHONWARNINGS=ignore
before_install:
- . ./ci/travis/ci.sh init RAY_CI_RLLIB_FULL_AFFECTED
install:
- . ./ci/travis/ci.sh init RAY_CI_RLLIB_FULL_AFFECTED
before_script:
- . ./ci/travis/ci.sh build
script:
- ./ci/keep_alive bazel test --config=ci --build_tests_only --test_tag_filters=tests_dir_J,tests_dir_K,tests_dir_L,tests_dir_M,tests_dir_N,tests_dir_O,tests_dir_P,tests_dir_Q,tests_dir_R,tests_dir_S,tests_dir_T,tests_dir_U,tests_dir_V,tests_dir_W,tests_dir_X,tests_dir_Y,tests_dir_Z rllib/...
Expand All @@ -262,9 +266,9 @@ matrix:
env:
- TESTSUITE=cpp_worker
- PYTHON=3.6
before_install:
- . ./ci/travis/ci.sh init
install:
- . ./ci/travis/ci.sh init
before_script:
- . ./ci/travis/ci.sh build
script:
- . ./ci/travis/ci.sh test_cpp
Expand Down
5 changes: 3 additions & 2 deletions BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1526,11 +1526,12 @@ genrule(
mv -f -- redis-cli.exe $(location redis-cli)
""",
"//conditions:default": """
set -x &&
tmpdir="redis.tmp" &&
path=$(location @com_github_antirez_redis//:file) &&
cp -p -L -R -- "$${path%/*}" "$${tmpdir}" &&
make -s -C "$${tmpdir}" -j"$$(getconf _NPROCESSORS_ONLN || echo 1)" V=0 &&
chmod +x "$${tmpdir}"/deps/jemalloc/configure &&
parallel="$$(getconf _NPROCESSORS_ONLN || echo 1)"
make -s -C "$${tmpdir}" -j"$${parallel}" V=0 CFLAGS="$${CFLAGS-} -DLUA_USE_MKSTEMP -Wno-pragmas -Wno-empty-body" &&
mv "$${tmpdir}"/src/redis-server $(location redis-server) &&
chmod +x $(location redis-server) &&
mv "$${tmpdir}"/src/redis-cli $(location redis-cli) &&
Expand Down
3 changes: 2 additions & 1 deletion bazel/ray_deps_setup.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ def ray_deps_setup():
"//thirdparty/patches:hiredis-windows-sigpipe.patch",
"//thirdparty/patches:hiredis-windows-sockets.patch",
"//thirdparty/patches:hiredis-windows-strerror.patch",
"//thirdparty/patches:redis-quiet.patch",
],
)

Expand Down Expand Up @@ -214,8 +215,8 @@ def ray_deps_setup():
url = "https://github.com/grpc/grpc/archive/4790ab6d97e634a1ede983be393f3bb3c132b2f7.tar.gz",
sha256 = "df83bd8a08975870b8b254c34afbecc94c51a55198e6e3a5aab61d62f40b7274",
patches = [
"//thirdparty/patches:grpc-command-quoting.patch",
"//thirdparty/patches:grpc-cython-copts.patch",
"//thirdparty/patches:grpc-python.patch",
],
)

Expand Down
2 changes: 1 addition & 1 deletion build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ git clone https://github.com/suquark/pickle5-backport
pushd pickle5-backport
git checkout 8ffe41ceba9d5e2ce8a98190f6b3d2f3325e5a72
CC=gcc "$PYTHON_EXECUTABLE" setup.py --quiet bdist_wheel
unzip -o dist/*.whl -d "$ROOT_DIR/python/ray/pickle5_files"
unzip -q -o dist/*.whl -d "$ROOT_DIR/python/ray/pickle5_files"
popd
popd

Expand Down
5 changes: 3 additions & 2 deletions ci/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ _Please double-check file/function/etc. names for changes, as this document may
All dependencies (e.g. `apt`, `pip`) should be installed in `install_dependencies()`, following the same pattern as
those that already exist.

Once a dependency is added/removed, please ensure that if `reload_env` (or similar) is updated if it exists, as CI
Once a dependency is added/removed, please ensure that shell environment variables are persisted appropriately, as CI
systems differ on when `~/.bashrc` et al. are reloaded, if at all. (And they are not necessarily idempotent.)

### Bazel, environment variables, and caching
Expand Down Expand Up @@ -39,10 +39,11 @@ Adding new scripts has a number of pitfalls that easily take hours (even days) t

The following practices can avoid such pitfalls while maintaining intuitive control flow:

- Put all environment-modifying functions in the _same_ shell script, so that their invocation behaves intuitively.
- Put all environment-modifying functions in the _top-level_ shell script, so that their invocation behaves intuitively.
(The sheer length of the script is a secondary concern and can be mitigated by keeping functions modular.)

- Avoid adding new scripts if possible. If it's necessary that you do so, call them instead of sourcing them.
Note that thies implies new scripts should not modify the environment, or the caller will not see such changes!

- Always add code inside a function, not at global scope. Use `local` for variables where it makes sense.
However, be careful and know the shell rules: for example, e.g. `local x=$(false)` succeeds even under `set -e`.
Expand Down
Loading