User object store AttributeError: 'Bunch' object has no attribute 'object_store_cache_size'. Did you mean: 'object_store_cache_path'?

[sanjaysrikakulam]

Galaxy Version:

{
"version_major": "24.1",
"version_minor": "2.dev0"
}

I am able to add an S3 object store using the template: Any S3 Compatible Storage (Legacy).

The following vars are also exclusively set in the galaxy.yml

    object_store_cache_path: /data/twd01/s3_object_store_cache
    object_store_cache_size: 10000

Tests:

1. Added our NetApp S3 as my user object store
2. I created a new history
3. Updated the history to use the new netapps3 storage
4. Ran a couple of upload jobs (1: Paste/Fetch URL, 2: Upload local file, and both failed with similar errors)

Error:

Jul 05 08:31:12 galaxyesginstance.esgwps.uno python[258019]: galaxy.queue_worker DEBUG 2024-07-05 08:31:12,125 [pN:handler_galaxyesginstance_0,p:258019,tN:Thread-1] App is not a webapp, not building a search index
Jul 05 08:32:33 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs.handler DEBUG 2024-07-05 08:32:33,908 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Grabbed Job(s): 12
Jul 05 08:32:33 galaxyesginstance.esgwps.uno python[258019]: tpv.rules.gateway INFO 2024-07-05 08:32:33,952 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] loading tpv rules from: ['/opt/galaxy/config/total_perspective_vortex/tool_defaults>
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: tpv.rules.gateway INFO 2024-07-05 08:32:34,210 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Watching for changes in file: /opt/galaxy/config/total_perspective_vortex/tool_defa>
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.util.watcher DEBUG 2024-07-05 08:32:34,210 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Watching for changes to file: /opt/galaxy/config/total_perspective_vortex/tool_d>
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: tpv.rules.gateway INFO 2024-07-05 08:32:34,215 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Watching for changes in file: /opt/galaxy/config/total_perspective_vortex/destinati>
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.util.watcher DEBUG 2024-07-05 08:32:34,215 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Watching for changes to file: /opt/galaxy/config/total_perspective_vortex/destin>
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: tpv.rules.gateway INFO 2024-07-05 08:32:34,216 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Watching for changes in file: /opt/galaxy/config/total_perspective_vortex/tools.yml
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.util.watcher DEBUG 2024-07-05 08:32:34,216 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Watching for changes to file: /opt/galaxy/config/total_perspective_vortex/tools.>
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: tpv.rules.gateway INFO 2024-07-05 08:32:34,223 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Watching for changes in file: /opt/galaxy/config/total_perspective_vortex/object_st>
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.util.watcher DEBUG 2024-07-05 08:32:34,223 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Watching for changes to file: /opt/galaxy/config/total_perspective_vortex/object>
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: tpv.core.entities DEBUG 2024-07-05 08:32:34,309 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Ranking destinations: [runner=condor, dest_name=condor_tpv, min_accepted_cores=Non>
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs.mapper DEBUG 2024-07-05 08:32:34,310 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] (12) Mapped job to destination id: condor_tpv
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs.handler DEBUG 2024-07-05 08:32:34,329 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] (12) Dispatching to condor runner
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs DEBUG 2024-07-05 08:32:34,345 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] (12) Persisting job destination (destination id: condor_tpv)
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,420 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Configuring generic S3 Connection
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,503 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Using cloud object store with bucket 'frct-smoe-bench-ec61-02'
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore DEBUG 2024-07-05 08:32:34,512 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Using preferred backend 'user_objects://7a89a28e-a50d-4808-adf6-28de1239ece1' for>
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,518 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Configuring generic S3 Connection
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,543 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Using cloud object store with bucket 'frct-smoe-bench-ec61-02'
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,607 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Pushing cache file '/data/twd01/s3_object_store_cache/f/c/c/dataset_fccf0813-d>
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,638 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Pushed cache file '/data/twd01/s3_object_store_cache/f/c/c/dataset_fccf0813-db>
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,645 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Configuring generic S3 Connection
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,701 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Using cloud object store with bucket 'my_s3_bucket_dummy'
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,711 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Configuring generic S3 Connection
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,780 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Using cloud object store with bucket 'my_s3_bucket_dummy'
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,786 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Configuring generic S3 Connection
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,811 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Using cloud object store with bucket 'my_s3_bucket_dummy'
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs DEBUG 2024-07-05 08:32:34,818 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] (12) Working directory for job is: /data/twd01/main/000/12
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs.runners DEBUG 2024-07-05 08:32:34,830 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] Job [12] queued (501.162 ms)
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs.handler INFO 2024-07-05 08:32:34,837 [pN:handler_galaxyesginstance_0,p:258019,tN:JobHandlerQueue.monitor_thread] (12) Job dispatched
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,901 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Configuring generic S3 Connection
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,926 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Using cloud object store with bucket 'my_s3_bucket_dummy'
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,937 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Configuring generic S3 Connection
Jul 05 08:32:34 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:34,972 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Using cloud object store with bucket 'my_s3_bucket_dummy'
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs DEBUG 2024-07-05 08:32:35,049 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Job wrapper for Job [12] prepared (209.274 ms)
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs.command_factory INFO 2024-07-05 08:32:35,061 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Built script [/data/twd01/main/000/12/tool_script.sh] for tool command [pytho>
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:35,116 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Configuring generic S3 Connection
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:35,139 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Using cloud object store with bucket 'my_s3_bucket_dummy'
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:35,150 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Configuring generic S3 Connection
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:35,173 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Using cloud object store with bucket 'my_s3_bucket_dummy'
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:35,178 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Pulling key 'f/c/c/dataset_fccf0813-db28-4dfe-a7d1-8e6e2bfa4241.dat' into cache to>
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:35,185 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Pulled key 'f/c/c/dataset_fccf0813-db28-4dfe-a7d1-8e6e2bfa4241.dat' into cache to >
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:35,200 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Configuring generic S3 Connection
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:32:35,234 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Using cloud object store with bucket 'my_s3_bucket_dummy'
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs.runners DEBUG 2024-07-05 08:32:35,314 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] (12) command is: mkdir -p working outputs configs
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: if [ -d _working ]; then
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]:     rm -rf working/ outputs/ configs/; cp -R _working working; cp -R _outputs outputs; cp -R _configs configs
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: else
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]:     cp -R working _working; cp -R outputs _outputs; cp -R configs _configs
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: fi
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: cd working; /bin/bash /data/twd01/main/000/12/tool_script.sh > '../outputs/tool_stdout' 2> '../outputs/tool_stderr'; return_code=$?; echo $return_code > /data/twd01/main/000/12/galaxy_12.ec; cd '/data/twd01>
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: [ "$GALAXY_VIRTUAL_ENV" = "None" ] && GALAXY_VIRTUAL_ENV="$_GALAXY_VIRTUAL_ENV"; _galaxy_setup_environment True; python metadata/set.py; sh -c "exit $return_code"
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs.runners.condor DEBUG 2024-07-05 08:32:35,330 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] (12) submitting file /data/twd01/main/000/12/galaxy_12.sh
Jul 05 08:32:35 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs.runners.condor INFO 2024-07-05 08:32:35,368 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] (12) queued as 12
Jul 05 08:32:51 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs.runners.condor DEBUG 2024-07-05 08:32:51,067 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.monitor_thread] (12/12) job is now running
Jul 05 08:32:59 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs.runners.condor DEBUG 2024-07-05 08:32:59,099 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.monitor_thread] (12/12) job is now running
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs.runners.condor DEBUG 2024-07-05 08:33:11,134 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.monitor_thread] (12/12) job is now running
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs.runners.condor DEBUG 2024-07-05 08:33:11,138 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.monitor_thread] (12/12) job has completed
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs ERROR 2024-07-05 08:33:11,171 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] problem importing job outputs. stdout [] stderr [
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: Traceback (most recent call last):
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:   File "/data/twd01/main/000/12/metadata/set.py", line 5, in <module>
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:     from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:                                                                ^^^^^^^^^^^^^^
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:   File "/opt/galaxy/server/lib/galaxy/metadata/set_metadata.py", line 153, in set_metadata
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:     set_metadata_portable()
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:   File "/opt/galaxy/server/lib/galaxy/metadata/set_metadata.py", line 214, in set_metadata_portable
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:     object_store = get_object_store(
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:                    ^^^^^^^^^^^^^^^^^
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:   File "/opt/galaxy/server/lib/galaxy/metadata/set_metadata.py", line 173, in get_object_store
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:     object_store = build_object_store_from_config(None, config_dict=config_dict, disable_process_management=True)
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:   File "/opt/galaxy/server/lib/galaxy/objectstore/__init__.py", line 1651, in build_object_store_from_config
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:     return objectstore_class(config=config, config_dict=config_dict, **objectstore_constructor_kwds)
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:   File "/opt/galaxy/server/lib/galaxy/objectstore/__init__.py", line 1118, in __init__
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:     backend = build_object_store_from_config(config, config_dict=backend_def, fsmon=fsmon)
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:   File "/opt/galaxy/server/lib/galaxy/objectstore/__init__.py", line 1651, in build_object_store_from_config
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:     return objectstore_class(config=config, config_dict=config_dict, **objectstore_constructor_kwds)
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:   File "/opt/galaxy/server/lib/galaxy/objectstore/s3.py", line 183, in __init__
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:     self.cache_size = cache_dict.get("size") or self.config.object_store_cache_size
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: AttributeError: 'Bunch' object has no attribute 'object_store_cache_size'. Did you mean: 'object_store_cache_path'?
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: ]
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: Traceback (most recent call last):
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:   File "/opt/galaxy/server/lib/galaxy/jobs/__init__.py", line 1920, in finish
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:     import_model_store.perform_import(history=job.history, job=job)
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:   File "/opt/galaxy/server/lib/galaxy/model/store/__init__.py", line 407, in perform_import
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:     datasets_attrs = self.datasets_properties()
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:   File "/opt/galaxy/server/lib/galaxy/model/store/__init__.py", line 1539, in datasets_properties
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:     datasets_attrs = load(open(datasets_attrs_file_name))
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: FileNotFoundError: [Errno 2] No such file or directory: '/data/twd01/main/000/12/metadata/outputs_populated/datasets_attrs.txt'
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs.runners ERROR 2024-07-05 08:33:11,174 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] (12/12) Job wrapper finish method failed
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: Traceback (most recent call last):
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:   File "/opt/galaxy/server/lib/galaxy/jobs/runners/__init__.py", line 676, in _finish_or_resubmit_job
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:     job_wrapper.finish(
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:   File "/opt/galaxy/server/lib/galaxy/jobs/__init__.py", line 1920, in finish
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:     import_model_store.perform_import(history=job.history, job=job)
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:   File "/opt/galaxy/server/lib/galaxy/model/store/__init__.py", line 407, in perform_import
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:     datasets_attrs = self.datasets_properties()
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:   File "/opt/galaxy/server/lib/galaxy/model/store/__init__.py", line 1539, in datasets_properties
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:     datasets_attrs = load(open(datasets_attrs_file_name))
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]:                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: FileNotFoundError: [Errno 2] No such file or directory: '/data/twd01/main/000/12/metadata/outputs_populated/datasets_attrs.txt'
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:33:11,186 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Configuring generic S3 Connection
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:33:11,229 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Using cloud object store with bucket 'my_s3_bucket_dummy'
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.jobs DEBUG 2024-07-05 08:33:11,235 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] fail(): Moved /data/twd01/main/000/12/outputs/dataset_fccf0813-db28-4dfe-a7d1-8e6e2bfa4241.d>
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:33:11,246 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Configuring generic S3 Connection
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:33:11,271 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Using cloud object store with bucket 'my_s3_bucket_dummy'
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:33:11,277 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Configuring generic S3 Connection
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:33:11,300 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Using cloud object store with bucket 'my_s3_bucket_dummy'
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:33:11,304 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Configuring generic S3 Connection
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:33:11,329 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Using cloud object store with bucket 'my_s3_bucket_dummy'
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:33:11,483 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Configuring generic S3 Connection
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:33:11,506 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Using cloud object store with bucket 'my_s3_bucket_dummy'
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:33:11,527 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Configuring generic S3 Connection
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:33:11,551 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Using cloud object store with bucket 'my_s3_bucket_dummy'
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:33:11,556 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Configuring generic S3 Connection
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:33:11,579 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Using cloud object store with bucket 'my_s3_bucket_dummy'
Jul 05 08:33:11 galaxyesginstance.esgwps.uno python[258019]: galaxy.objectstore.s3 DEBUG 2024-07-05 08:33:11,590 [pN:handler_galaxyesginstance_0,p:258019,tN:CondorRunner.work_thread-2] Wanted to push file '/data/twd01/s3_object_store_cache/f/c/c/dataset_fccf0813-db28>