update config for 24.1 #1225

bgruening · 2024-06-04T19:53:20Z

No description provided.

group_vars/gxconfig.yml

bgruening · 2024-06-04T20:13:29Z

files/galaxy/config/file_source_templates.yml

@@ -0,0 +1,5 @@
+- include: ./lib/galaxy/files/templates/examples/production_azure.yml


we need to decide what we do with those templates ...
Do we copy them into a specific location? How do we handle version updates to those files then? I don't think we need to template them out, but we need to decide where to store them.

a few thoughts:

we might want to add new templates, and those are then most likely not in the Galaxy source tree

upstream might add updates - a new version of the file-source with a new version number

those we could sync like we sync now other config files

I assume those templates need to be accessible by celery and other services - so on a shared filesystem

Maybe a separate folder under mutable-config/file-source-templates and mutable-config/object-store-templates/ ?

@jmchilton any thoughts?

I think that is a good solution. We can have a separate directory and append it to the lists here. We can add a couple of variables there and let the ansible-galaxy role copy everything and handle these files.

Also, rsync already syncs the mutable-config dirs to the NFS, and Celery and other services that need them can access it from the NFS.

I am sorry I missed this ping - I just landed up here from that PR review. I would start by not copying them out - just link use the links to link the ones in the code base. Y'all are going to be way ahead of any other instance on this so if you have fixes for the templates - try to upstream them and we will try to be very open to that.

… galaxy conf accordingly

sanjaysrikakulam · 2024-07-05T07:15:24Z

I have added a new commit that:

Adds object store and file source templates and the catalog for the same.
Updates galaxy upgrade diff script to include the compare for these templates.
Enables the file source and object store template variables in the galaxy conf.
Updates sn06 group vars to allow the creation of the new dirs file_source_templates and object_store_templates under galaxy_mutable_config_dir. Also, copies the file source and object store templates along with the catalog files to the respective locations on the server.

I deployed the same in the ESG instance, and the object store and file source options are now available for the users in the user preferences.

bgruening · 2024-07-05T09:55:49Z

bin/diff-before-update

@@ -51,3 +51,19 @@ compare files/$folder/config/user_preferences_extra_conf.yml	$GALAXY/config/user
 #compare templates/$folder/config/build_sites.yml.j2	$GALAXY/config/build_sites.yml.sample.sample

 compare group_vars/gxconfig${ext}.yml	$GALAXY/config/galaxy.yml.sample
+
+# Compare file source templates
+compare files/$folder/config/file_source_templates/production_aws_private_bucket.yml	$GALAXY/files/templates/examples/production_aws_private_bucket.yml


Any idea how we remember to add new templates here if we add new ones?

I will update our operations here and add additional checkpoints there that this script should updated if and when new templates from upstream are added.

Additionally, we can loop over all the files in our directory, compare them to the files in the respective templates dir in the upstream, and compare them during the upgrade cycle (looping over files might not cover all cases.).

unfortunately not because the upstream has more templates; we picked only the ones that are "production" ready.

Documentation PR

group_vars/gxconfig.yml

sanjaysrikakulam · 2024-07-08T07:25:56Z

BTW: I created an issue in the upstream related to the object store galaxyproject/galaxy#18500

We either have to wait a bit or proceed with the rollout without the object stores and deploy object stores later.

sanjaysrikakulam · 2024-07-08T19:29:31Z

Things to set in the galaxy config for the object store:

object_store_cache_path: We need a Universal path for all user object stores (I guess we need to get a dedicated NFS share or create a dedicated user_object_store_cache dir in one of the JWDs. I don't know how big this should be and how well this feature is going to be used by our users).
object_store_cache_size: Set this to the max size of the object_store_cache_path NFS share.

BTW: I have added version: 0 to all the templates part of this PR, so the templates in this PR are the same as the ones in my PR here

ping @bgruening.

bgruening · 2024-07-08T20:21:27Z

We can disable this feature for the time being.

bgruening · 2024-07-08T20:21:55Z

A new NFS is ordered and we will use that when we advertise this feature broadly.

bgruening · 2024-07-09T08:33:46Z

So lets disable this feature then.

sanjaysrikakulam · 2024-07-09T11:56:29Z

I have now disabled copying the object store and file source template catalogs.

I think the PR is ready for review. Please check.

bgruening · 2024-07-09T14:56:28Z

Good for me. I'm I allowed to do the update this evening or better tomorrow evening?

sanjaysrikakulam · 2024-07-09T15:25:08Z

Tomorrow evening is better.

By the way, this patch fixes the object store issue. When we have the new NFS box, we can deploy the user file source and the object store features.

bgruening · 2024-07-10T13:58:01Z

So any last comments before I deploy it this evening?

Removes all the user file sources and object store conf along with the catalog, templates and ansible copy stuff for now. This can be added when this feature is going to be rolled out.

bgruening and others added 2 commits June 4, 2024 21:47

update config for 24.1

0648121

Merge branch 'master' into release_24.1

da7ed17

bgruening commented Jun 4, 2024

View reviewed changes

group_vars/gxconfig.yml Show resolved Hide resolved

add file_source_template config file

ba2adab

bgruening commented Jun 4, 2024

View reviewed changes

Add templates for object store and file sources, update playbook, and…

bb01c94

… galaxy conf accordingly

Update galaxy commit id to 24.1 release

0b49201

bgruening commented Jul 5, 2024

View reviewed changes

mira-miracoli reviewed Jul 8, 2024

View reviewed changes

group_vars/gxconfig.yml Outdated Show resolved Hide resolved

Disable file source and object store config for now in 24.1 rollout

246db72

bgruening marked this pull request as ready for review July 10, 2024 13:57

Undo the commit bb01c94

605cde4

Removes all the user file sources and object store conf along with the catalog, templates and ansible copy stuff for now. This can be added when this feature is going to be rolled out.

sanjaysrikakulam approved these changes Jul 10, 2024

View reviewed changes

mira-miracoli approved these changes Jul 10, 2024

View reviewed changes

bgruening merged commit 261da2e into master Jul 10, 2024
2 checks passed

bgruening deleted the release_24.1 branch July 10, 2024 18:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update config for 24.1 #1225

update config for 24.1 #1225

bgruening commented Jun 4, 2024

bgruening Jun 4, 2024

bgruening Jun 4, 2024

bgruening Jun 4, 2024

sanjaysrikakulam Jul 3, 2024 •

edited

Loading

jmchilton Jul 9, 2024

sanjaysrikakulam commented Jul 5, 2024 •

edited

Loading

bgruening Jul 5, 2024

sanjaysrikakulam Jul 5, 2024

sanjaysrikakulam Jul 5, 2024 •

edited

Loading

sanjaysrikakulam Jul 8, 2024

sanjaysrikakulam Jul 9, 2024

sanjaysrikakulam commented Jul 8, 2024

sanjaysrikakulam commented Jul 8, 2024

bgruening commented Jul 8, 2024

bgruening commented Jul 8, 2024

bgruening commented Jul 9, 2024

sanjaysrikakulam commented Jul 9, 2024

bgruening commented Jul 9, 2024

sanjaysrikakulam commented Jul 9, 2024 •

edited

Loading

bgruening commented Jul 10, 2024

		@@ -0,0 +1,5 @@
		- include: ./lib/galaxy/files/templates/examples/production_azure.yml

update config for 24.1 #1225

update config for 24.1 #1225

Conversation

bgruening commented Jun 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanjaysrikakulam Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanjaysrikakulam commented Jul 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanjaysrikakulam Jul 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanjaysrikakulam commented Jul 8, 2024

sanjaysrikakulam commented Jul 8, 2024

bgruening commented Jul 8, 2024

bgruening commented Jul 8, 2024

bgruening commented Jul 9, 2024

sanjaysrikakulam commented Jul 9, 2024

bgruening commented Jul 9, 2024

sanjaysrikakulam commented Jul 9, 2024 • edited Loading

bgruening commented Jul 10, 2024

sanjaysrikakulam Jul 3, 2024 •

edited

Loading

sanjaysrikakulam commented Jul 5, 2024 •

edited

Loading

sanjaysrikakulam Jul 5, 2024 •

edited

Loading

sanjaysrikakulam commented Jul 9, 2024 •

edited

Loading