-
Notifications
You must be signed in to change notification settings - Fork 98
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
update config for 24.1 #1225
Merged
Merged
update config for 24.1 #1225
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
0648121
update config for 24.1
bgruening da7ed17
Merge branch 'master' into release_24.1
bgruening ba2adab
add file_source_template config file
bgruening bb01c94
Add templates for object store and file sources, update playbook, and…
sanjaysrikakulam 0b49201
Update galaxy commit id to 24.1 release
sanjaysrikakulam 246db72
Disable file source and object store config for now in 24.1 rollout
sanjaysrikakulam 605cde4
Undo the commit bb01c94ca30589914217ea6cfb6941bdce6273fc
sanjaysrikakulam File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
31 changes: 31 additions & 0 deletions
31
files/galaxy/config/file_source_templates/production_aws_private_bucket.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| - id: aws_private | ||
| version: 0 | ||
| name: Amazon Web Services Private Bucket | ||
| description: Setup access to a private AWS bucket using a secret access key. | ||
| configuration: | ||
| type: s3fs | ||
| bucket: '{{ bucket }}' | ||
| writable: '{{ writable }}' | ||
| secret: '{{ secret_key }}' | ||
| key: '{{ access_key }}' | ||
| variables: | ||
| access_key: | ||
| label: Access Key ID | ||
| type: string | ||
| help: | | ||
| The "access key ID" as defined in the [Amazon Documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). | ||
| bucket: | ||
| label: Bucket | ||
| type: string | ||
| help: | | ||
| The [Amazon Web Services Bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html) to | ||
| access. This should be a bucket the user described by the Access Key ID has access to. | ||
| writable: | ||
| label: Writable? | ||
| type: boolean | ||
| help: Is this a bucket you have permission to write to? | ||
| secrets: | ||
| secret_key: | ||
| label: Secret Access Key | ||
| help: | | ||
| The "secret access key" as defined in the [Amazon Documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). |
16 changes: 16 additions & 0 deletions
16
files/galaxy/config/file_source_templates/production_aws_public_bucket.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| - id: aws_public | ||
| version: 0 | ||
| name: Amazon Web Services Public Bucket | ||
| description: Setup anonymous access to a public AWS bucket. | ||
| configuration: | ||
| type: s3fs | ||
| bucket: '{{ variables.bucket }}' | ||
| writable: false | ||
| anon: true | ||
| variables: | ||
| bucket: | ||
| label: Bucket | ||
| type: string | ||
| help: | | ||
| The [Amazon Web Services Bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html) to | ||
| anonymously access. |
44 changes: 44 additions & 0 deletions
44
files/galaxy/config/file_source_templates/production_azure.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| - id: azure | ||
| version: 0 | ||
| name: Azure Blob Storage | ||
| description: | | ||
| This template allows connecting to [Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction). | ||
| configuration: | ||
| type: azure | ||
| container_name: '{{ variables.container_name }}' | ||
| account_name: '{{ variables.account_name }}' | ||
| account_key: '{{ secrets.account_key }}' | ||
| namespace_type: "{{ 'hierarchical' if variables.hierarchical else 'flat' }}" | ||
| writable: '{{ variables.writable }}' | ||
| variables: | ||
| container_name: | ||
| label: Container Name | ||
| type: string | ||
| help: | | ||
| The name of your Azure Blob Storage container. More information on containers can be found | ||
| in the [Azure Storage documentation](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers). | ||
| account_name: | ||
| label: Storage Account Name | ||
| type: string | ||
| help: | | ||
| The name of your Azure Blob Storage account. More information on containers can be found in the | ||
| [Azure Storage documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview). | ||
| hierarchical: | ||
| label: Hierarchical? | ||
| type: boolean | ||
| default: true | ||
| help: | | ||
| Is this storage hierarchical (e.g. does it use a Azure Data Lake Storage Gen2 hierarchical namespace)? | ||
| More information on Data Lake Storage namespaces can be found in the | ||
| [Azure Blob Storage documentation](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-namespace). | ||
| writable: | ||
| label: Writable? | ||
| type: boolean | ||
| default: true | ||
| help: Allow Galaxy to write data to this Azure Blob Storage container. | ||
| secrets: | ||
| account_key: | ||
| label: Account Key | ||
| help: | | ||
| The Azure Blob Storage account key to use to access your Azure Blob Storage data. More information | ||
| on account keys can be found in the [Azure Storage documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage). |
39 changes: 39 additions & 0 deletions
39
files/galaxy/config/file_source_templates/production_ftp.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| - id: ftp | ||
| version: 0 | ||
| name: An FTP Server | ||
| description: | | ||
| This template allows connecting to FTP servers. This file source plugin should | ||
| support FTP and FTPS servers. | ||
| configuration: | ||
| type: ftp | ||
| host: '{{ variables.host }}' | ||
| user: '{{ variables.user }}' | ||
| port: '{{ variables.port }}' | ||
| passwd: '{{ secrets.password }}' | ||
| writable: '{{ variables.writable }}' | ||
| variables: | ||
| host: | ||
| label: FTP Host | ||
| type: string | ||
| help: Host of FTP Server to connect to. | ||
| user: | ||
| label: FTP User | ||
| type: string | ||
| help: | | ||
| Username to connect with. Leave this blank to connect to the server | ||
| anonymously (if allowed by target server). | ||
| writable: | ||
| label: Writable? | ||
| type: boolean | ||
| help: Is this an FTP server you have permission to write to? | ||
| port: | ||
| label: FTP Port | ||
| type: integer | ||
| help: Port used to connect to the FTP server. | ||
| default: 21 | ||
| secrets: | ||
| password: | ||
| label: FTP Password | ||
| help: | | ||
| Password to connect to FTP server with. Leave this blank to connect | ||
| to the server anonymously (if allowed by target server). |
57 changes: 57 additions & 0 deletions
57
files/galaxy/config/file_source_templates/production_s3fs.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| - id: s3fs | ||
| version: 0 | ||
| name: S3 Compatible Storage with Credentials | ||
| description: | | ||
| The APIs used to connect to Amazon's S3 (Simple Storage Service) have become something | ||
| of an unofficial standard for cloud storage across a variety of vendors and services. | ||
| Many vendors offer storage APIs compatible with S3. This template configuration allows | ||
| using such service as a Galaxy storage location as long as you are able to find the | ||
| connection details and have the relevant credentials. | ||
|
|
||
| Given the amount of information needed to connect to such a service, this is a bit of an | ||
| advanced template and probably should not be used to connect to a service if a more | ||
| specific template is available. | ||
| variables: | ||
| access_key: | ||
| label: Access Key ID | ||
| type: string | ||
| help: | | ||
| The less secure part of your access tokens or access keys that describe the user | ||
| that is accessing the data. The [Amazon documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html) | ||
| calls these an "access key ID", the [CloudFlare documentation](https://developers.cloudflare.com/r2/examples/aws/boto3/) | ||
| describes these as ``aws_access_key_id``. | ||
| bucket: | ||
| label: Bucket | ||
| type: string | ||
| help: | | ||
| The [bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html) to | ||
| store your datasets in. How to setup buckets for your storage will vary from service to service | ||
| but all S3 compatible storage services should have the concept of a bucket to namespace | ||
| a grouping of your data together with. | ||
| endpoint_url: | ||
| label: S3-Compatible API Endpoint | ||
| type: string | ||
| help: | | ||
| If the documentation for your storage service has something called an ``endpoint_url``, | ||
| For instance, the CloudFlare documentation describes its endpoints as ``https://<accountid>.r2.cloudflarestorage.com``. Here | ||
| you would substitute your CloudFlare account ID into the endpoint url and use that value. | ||
| So if your account ID was ``galactian``, you would enter ``galactian.r2.cloudflarestorage.com``. | ||
| The [MinIO](https://min.io/docs/minio/linux/integrations/aws-cli-with-minio.html) | ||
| documentation describes the endpoint URL for its Play service as ``https://play.min.io:9000``, | ||
| this value would be entered here. | ||
| secrets: | ||
| secret_key: | ||
| label: Secret Access Key | ||
| help: | | ||
| The secret key used to connect to the S3 compatible storage with for the given access key. | ||
|
|
||
| The [Amazon documentation] calls these an "secret access key" and | ||
| the [CloudFlare documentation](https://developers.cloudflare.com/r2/examples/aws/boto3/) | ||
| describes these as ``aws_secret_access_key``. Internally to Galaxy, we often just call | ||
| this the ``secret_key``. | ||
| configuration: | ||
| type: s3fs | ||
| endpoint_url: '{{ variables.endpoint_url }}' | ||
| key: '{{ variables.access_key }}' | ||
| secret: '{{ secrets.secret_key }}' | ||
| bucket: '{{ variables.bucket }}' |
35 changes: 35 additions & 0 deletions
35
files/galaxy/config/object_store_templates/production_aws_s3.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| - id: aws_s3 | ||
| version: 0 | ||
| name: Amazon Web Services S3 Storage | ||
| description: | | ||
| Amazon's Simple Storage Service (S3) is Amazon's primary cloud storage service. | ||
| More information on S3 can be found in [Amazon's documentation](https://aws.amazon.com/s3/). | ||
| variables: | ||
| access_key: | ||
| label: Access Key ID | ||
| type: string | ||
| help: | | ||
| A security credential for interacting with AWS services can be created from your | ||
| AWS web console. Creating an "Access Key" creates a pair of keys used to identify | ||
| and authenticate access to your AWS account - the first part of the pair is | ||
| "Access Key ID" and should be entered here. The second part of your key is the secret | ||
| part called the "Secret Access Key". Place that in the secure part of this form below. | ||
| bucket: | ||
| label: Bucket | ||
| type: string | ||
| help: | | ||
| The [AWS S3 Bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html) to | ||
| store your datasets in. You will need to create a bucket to use in your AWS web console before | ||
| using this form. | ||
| secrets: | ||
| secret_key: | ||
| label: Secret Access Key | ||
| help: | | ||
| See the documentation above used "Access Key ID" for information about access key pairs. | ||
| configuration: | ||
| type: boto3 | ||
| auth: | ||
| access_key: '{{ variables.access_key }}' | ||
| secret_key: '{{ secrets.secret_key }}' | ||
| bucket: | ||
| name: '{{ variables.bucket }}' |
36 changes: 36 additions & 0 deletions
36
files/galaxy/config/object_store_templates/production_aws_s3_legacy.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| - id: aws_s3_legacy | ||
| version: 0 | ||
| name: Amazon Web Services S3 Storage (Legacy) | ||
| description: | | ||
| Amazon's Simple Storage Service (S3) is Amazon's primary cloud storage service. | ||
| More information on S3 can be found in [Amazon's documentation](https://aws.amazon.com/s3/). | ||
| variables: | ||
| access_key: | ||
| label: Access Key ID | ||
| type: string | ||
| help: | | ||
| A security credential for interacting with AWS services can be created from your | ||
| AWS web console. Creating an "Access Key" creates a pair of keys used to identify | ||
| and authenticate access to your AWS account - the first part of the pair is | ||
| "Access Key ID" and should be entered here. The second part of your key is the secret | ||
| part called the "Secret Access Key". Place that in the secure part of this form below. | ||
| bucket: | ||
| label: Bucket | ||
| type: string | ||
| help: | | ||
| The [AWS S3 Bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html) to | ||
| store your datasets in. You will need to create a bucket to use in your AWS web console before | ||
| using this form. | ||
| secrets: | ||
| secret_key: | ||
| label: Secret Access Key | ||
| help: | | ||
| See the documentation above used "Access Key ID" for information about access key pairs. | ||
|
|
||
| configuration: | ||
| type: aws_s3 | ||
| auth: | ||
| access_key: '{{ variables.access_key }}' | ||
| secret_key: '{{ secrets.secret_key }}' | ||
| bucket: | ||
| name: '{{ variables.bucket }}' |
31 changes: 31 additions & 0 deletions
31
files/galaxy/config/object_store_templates/production_azure_blob.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| - id: azure | ||
| version: 0 | ||
| name: Azure Blob Storage | ||
| description: | | ||
| This template allows storing dataset in [Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction). | ||
| configuration: | ||
| type: azure_blob | ||
| auth: | ||
| account_name: '{{ variables.account_name }}' | ||
| account_key: '{{ secrets.account_key}}' | ||
| container: | ||
| name: '{{ variables.container_name }}' | ||
| variables: | ||
| container_name: | ||
| label: Container Name | ||
| type: string | ||
| help: | | ||
| The name of your Azure Blob Storage container. More information on containers can be found | ||
| in the [Azure Storage documentation](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers). | ||
| account_name: | ||
| label: Storage Account Name | ||
| type: string | ||
| help: | | ||
| The name of your Azure Blob Storage account. More information on containers can be found in the | ||
| [Azure Storage documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview). | ||
| secrets: | ||
| account_key: | ||
| label: Account Key | ||
| help: | | ||
| The Azure Blob Storage account key to use to access your Azure Blob Storage data. More information | ||
| on account keys can be found in the [Azure Storage documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage). |
41 changes: 41 additions & 0 deletions
41
files/galaxy/config/object_store_templates/production_gcp_s3.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| # https://cloud.google.com/storage/docs/aws-simple-migration | ||
| - id: gcp_s3_interop | ||
| version: 0 | ||
| name: Google Cloud Storage | ||
| description: | | ||
| This template can be used to connect to your [Google Cloud Storage](https://cloud.google.com/storage). | ||
| To use these templates you will need to generate | ||
| [HMAC Keys](https://cloud.google.com/storage/docs/authentication/hmackeys) - these | ||
| can be linked to your user or a service account. Additionally, you will need to defined | ||
| a [default Google cloud project](https://cloud.google.com/storage/docs/aws-simple-migration#defaultproj) | ||
| to allow Galaxy to access your Google Cloud Storage via the interfaces | ||
| described by this template. | ||
| variables: | ||
| access_key: | ||
| label: Access ID | ||
| type: string | ||
| help: | | ||
| This will be given to you by Google when you generate [HMAC Keys](https://cloud.google.com/storage/docs/authentication/hmackeys) | ||
| to use your storage. | ||
| bucket: | ||
| label: Bucket | ||
| type: string | ||
| help: | | ||
| The name of a [bucket](https://cloud.google.com/storage/docs/buckets) you've created to store your Galaxy data. Documentation for how to create buckets | ||
| can be found in [this part of the Google Cloud Storage documentation](https://cloud.google.com/storage/docs/creating-buckets). | ||
| secrets: | ||
| secret_key: | ||
| label: Secret Key | ||
| help: | | ||
| This will be given to you by Google when you generate [HMAC Keys](https://cloud.google.com/storage/docs/authentication/hmackeys) | ||
| to use your storage. It should be 40 characters long and look something like the example used | ||
| the Google documentation - `bGoa+V7g/yqDXvKRqq+JTFn4uQZbPiQJo4pf9RzJ`. | ||
| configuration: | ||
| type: boto3 | ||
| auth: | ||
| access_key: '{{ variables.access_key }}' | ||
| secret_key: '{{ secrets.secret_key }}' | ||
| bucket: | ||
| name: '{{ variables.bucket }}' | ||
| connection: | ||
| endpoint_url: 'https://storage.googleapis.com/' |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any idea how we remember to add new templates here if we add new ones?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will update our operations here and add additional checkpoints there that this script should updated if and when new templates from upstream are added.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Additionally, we can loop over all the files in our directory, compare them to the files in the respective templates dir in the upstream, and compare them during the upgrade cycle (looping over files might not cover all cases.).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
unfortunately not because the upstream has more templates; we picked only the ones that are "production" ready.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Documentation PR