Skip to content

Commit

Permalink
Make rapids-get-artifact work locally (#59)
Browse files Browse the repository at this point in the history
This PR ensures that `rapids-get-artifact` works locally.

This request was made by @bdice who was attempting to use the script
locally to debug some CUDA 12 issues.

I've moved the logic for downloading, extracting, and printing the
artifact's path to a new script, `_rapids-download-from-s3`.

This script is then called by both `rapids-download-from-s3` and
`rapids-get-artifact`.

In the future, I would like to rework some of the scripts regarding
artifact uploading/downloading, but I think we need to finish reworking
the wheel workflows before that will be feasible.
  • Loading branch information
ajschmidt8 authored Jun 8, 2023
1 parent 3303915 commit 17a072b
Show file tree
Hide file tree
Showing 3 changed files with 30 additions and 21 deletions.
27 changes: 27 additions & 0 deletions tools/_rapids-download-from-s3
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/bin/bash
# A utility script that downloads an artifact archive from S3, untars it,
# and prints the location where it was untarred.
# The script can be used locally and in CI.
# Positional Arguments:
# 1) path to s3 artifact
# 2) location to untar it to
set -euo pipefail
source rapids-constants

s3_dl_path="$1"
untar_dest="$2"

mkdir -p "${untar_dest}"

if [ "${CI:-false}" = "false" ]; then
# shellcheck disable=SC2001
s3_dl_path=$(echo "${s3_dl_path}" | sed "s|s3://${RAPIDS_DOWNLOADS_BUCKET}|https://${RAPIDS_DOWNLOADS_DOMAIN}|")
rapids-echo-stderr "Downloading and decompressing ${s3_dl_path} into ${untar_dest}"
wget -qO- "${s3_dl_path}" | tar xzf - -C "${untar_dest}"
else
rapids-echo-stderr "Downloading and decompressing ${s3_dl_path} into ${untar_dest}"
aws s3 cp --only-show-errors "${s3_dl_path}" - | tar xzf - -C "${untar_dest}"
fi

# echo path to untarred contents
echo -n "${untar_dest}"
17 changes: 1 addition & 16 deletions tools/rapids-download-from-s3
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
# 1) package name to generate s3 path for
# 2) location to untar it to
set -euo pipefail
source rapids-constants
export RAPIDS_SCRIPT_NAME="rapids-download-from-s3"

if [ -z "$1" ] || [ -z "$2" ]; then
Expand All @@ -14,20 +13,6 @@ if [ -z "$1" ] || [ -z "$2" ]; then
fi

s3_dl_path="$(rapids-s3-path)$1"

untar_dest="$2"
mkdir -p "${untar_dest}"

if [ "${CI:-false}" = "false" ]; then
# shellcheck disable=SC2001
s3_dl_path=$(echo "${s3_dl_path}" | sed "s|s3://${RAPIDS_DOWNLOADS_BUCKET}|https://${RAPIDS_DOWNLOADS_DOMAIN}|")
rapids-echo-stderr "Downloading and decompressing ${s3_dl_path} into ${untar_dest}"
wget -qO- "${s3_dl_path}" | tar xzf - -C "${untar_dest}"
else
rapids-echo-stderr "Downloading and decompressing ${s3_dl_path} into ${untar_dest}"
aws s3 cp --only-show-errors "${s3_dl_path}" - | tar xzf - -C "${untar_dest}"
fi


# echo path to untarred contents
echo -n "${untar_dest}"
_rapids-download-from-s3 "$s3_dl_path" "$untar_dest"
7 changes: 2 additions & 5 deletions tools/rapids-get-artifact
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,7 @@
set -euo pipefail
source rapids-constants

artifact_path="s3://${RAPIDS_DOWNLOADS_BUCKET}/${1}"
s3_dl_path="s3://${RAPIDS_DOWNLOADS_BUCKET}/${1}"
untar_dest=$(mktemp -d)

rapids-echo-stderr "Downloading and decompressing ${artifact_path} into ${untar_dest}"
aws s3 cp --only-show-errors "${artifact_path}" - | tar xzf - -C "${untar_dest}"

echo -n "${untar_dest}"
_rapids-download-from-s3 "$s3_dl_path" "$untar_dest"

0 comments on commit 17a072b

Please sign in to comment.