-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Added components for Google Cloud Storage: * Upload * Download * List blobs
- Loading branch information
Showing
6 changed files
with
134 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| name: Download from GCS | ||
| inputs: | ||
| - {name: GCS path, type: URI} | ||
| outputs: | ||
| - {name: Data} | ||
| implementation: | ||
| container: | ||
| image: google/cloud-sdk | ||
| command: | ||
| - bash # Pattern comparison only works in Bash | ||
| - -ex | ||
| - -c | ||
| - | | ||
| if [ -n "${GOOGLE_APPLICATION_CREDENTIALS}" ]; then | ||
| gcloud auth activate-service-account --key-file="${GOOGLE_APPLICATION_CREDENTIALS}" | ||
| fi | ||
| stat_result=$(gsutil ls -d "$0") | ||
| if [[ "$stat_result" == */ ]]; then | ||
| mkdir -p "$1" # When source path is a directory, gsutil requires the destination to also be a directory | ||
| else | ||
| mkdir -p "$(dirname "$1")" | ||
| fi | ||
| gsutil -m rsync -r "$0" "$1" # gsutil cp has different path handling than Linux cp. It always puts the source directory (name) inside the destination directory. gsutil rsync does not have that problem. | ||
| - inputValue: GCS path | ||
| - outputPath: Data |
20 changes: 20 additions & 0 deletions
20
components/google-cloud/storage/download_blob/component.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| name: Download from GCS | ||
| inputs: | ||
| - {name: GCS path, type: URI} | ||
| outputs: | ||
| - {name: Data} | ||
| implementation: | ||
| container: | ||
| image: google/cloud-sdk | ||
| command: | ||
| - sh | ||
| - -ex | ||
| - -c | ||
| - | | ||
| if [ -n "${GOOGLE_APPLICATION_CREDENTIALS}" ]; then | ||
| gcloud auth activate-service-account --key-file="${GOOGLE_APPLICATION_CREDENTIALS}" | ||
| fi | ||
| mkdir -p "$(dirname "$1")" | ||
| gsutil -m cp -r "$0" "$1" | ||
| - inputValue: GCS path | ||
| - outputPath: Data |
20 changes: 20 additions & 0 deletions
20
components/google-cloud/storage/download_dir/component.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| name: Download from GCS | ||
| inputs: | ||
| - {name: GCS path, type: URI} | ||
| outputs: | ||
| - {name: Data} | ||
| implementation: | ||
| container: | ||
| image: google/cloud-sdk | ||
| command: | ||
| - sh | ||
| - -ex | ||
| - -c | ||
| - | | ||
| if [ -n "${GOOGLE_APPLICATION_CREDENTIALS}" ]; then | ||
| gcloud auth activate-service-account --key-file="${GOOGLE_APPLICATION_CREDENTIALS}" | ||
| fi | ||
| mkdir -p "$1" | ||
| gsutil -m cp -r "$0" "$1" | ||
| - inputValue: GCS path | ||
| - outputPath: Data |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| name: List blobs | ||
| inputs: | ||
| - {name: GCS path, type: URI, description: 'GCS path for listing. For recursive listing use the "gs://bucket/path/**" syntax".'} | ||
| outputs: | ||
| - {name: Paths} | ||
| implementation: | ||
| container: | ||
| image: google/cloud-sdk | ||
| command: | ||
| - sh | ||
| - -ex | ||
| - -c | ||
| - | | ||
| if [ -n "${GOOGLE_APPLICATION_CREDENTIALS}" ]; then | ||
| gcloud auth activate-service-account --key-file="${GOOGLE_APPLICATION_CREDENTIALS}" | ||
| fi | ||
| mkdir -p "$(dirname "$1")" | ||
| gsutil ls "$0" > "$1" | ||
| - inputValue: GCS path | ||
| - outputPath: Paths |
23 changes: 23 additions & 0 deletions
23
components/google-cloud/storage/upload_to_explicit_uri/component.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| name: Upload to GCS | ||
| inputs: | ||
| - {name: Data} | ||
| - {name: GCS path, type: URI} | ||
| outputs: | ||
| - {name: GCS path, type: URI} | ||
| implementation: | ||
| container: | ||
| image: google/cloud-sdk | ||
| command: | ||
| - sh | ||
| - -ex | ||
| - -c | ||
| - | | ||
| if [ -n "${GOOGLE_APPLICATION_CREDENTIALS}" ]; then | ||
| gcloud auth activate-service-account --key-file="${GOOGLE_APPLICATION_CREDENTIALS}" | ||
| fi | ||
| gsutil cp -r "$0" "$1" | ||
| mkdir -p "$(dirname "$2")" | ||
| echo "$1" > "$2" | ||
| - inputPath: Data | ||
| - inputValue: GCS path | ||
| - outputPath: GCS path |
24 changes: 24 additions & 0 deletions
24
components/google-cloud/storage/upload_to_unique_uri/component.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| name: Upload to GCS | ||
| description: Upload to GCS with unique URI suffix | ||
| inputs: | ||
| - {name: Data} | ||
| - {name: GCS path prefix, type: URI} | ||
| outputs: | ||
| - {name: GCS path, type: URI} | ||
| implementation: | ||
| container: | ||
| image: google/cloud-sdk | ||
| command: | ||
| - sh | ||
| - -ex | ||
| - -c | ||
| - | | ||
| if [ -n "${GOOGLE_APPLICATION_CREDENTIALS}" ]; then | ||
| gcloud auth activate-service-account --key-file="${GOOGLE_APPLICATION_CREDENTIALS}" | ||
| fi | ||
| gsutil cp -r "$0" "$1" | ||
| mkdir -p "$(dirname "$2")" | ||
| echo "$1" > "$2" | ||
| - inputPath: Data | ||
| - concat: [{inputValue: GCS path prefix}, '{{workflow.uid}}_{{pod.name}}'] | ||
| - outputPath: GCS path |