[WIP] Use galaxy 19.09 #7
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ericfranz
reviewed
Nov 14, 2019
| <param id="function">dynamic_cores_time</param> | ||
| </destination> | ||
| <destination id="pbs" runner="pbs"> | ||
| <param id="Resource_List">walltime=5:00:00,nodes=1:ppn=<%= ppn %></param> |
There was a problem hiding this comment.
Maybe we should default the walltime to 1 hour and ppn=1, since this is the job runner. We should assume tools are simple and give the option for users to request more. So my understanding with the parameters below is that we are basically saying you can submit a job to an Owens compute node, and request 1 core or all 28 cores and also change the walltime.
ericfranz
reviewed
Nov 14, 2019
| cd galaxy | ||
| sh install_dependencies.sh | ||
| sh create_db.sh | ||
| sh install-galaxy.sh |
There was a problem hiding this comment.
when building at OSC on webtest02, should I first load a specific Python version? For example I just started with the default python
efranz@webtest02:~$ python --version
Python 2.7.5
But there is also
efranz@webtest02:~$ scl enable rh-python35 -- python --version
Python 3.5.1
And we can install https://www.softwarecollections.org/en/scls/rhscl/rh-python36/
There was a problem hiding this comment.
Oops I was installing on webtest02. I forgot this line in the README:
The Install process runs on the login node
Yeah that makes sense. In that case, which python module should be used on the Owens login node?
|
So I cloned the repo, then ran It is stalled in "Starting" mode. I look at the output log and see the following: Perhaps I skipped a step in the installation? Full output is below: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Galaxy interface app runs on Owens. The users can install, manage and run tools and workflows.
Changes
Known Issues:
admin/rolesis not correctly appended to the URL.pbs-pythonis installed at run time when launching.may need to add something similiar to this to
install-galaxy.shExperiment 1: Passenger App (failed)
Failed to mount the app to
/pun/dev/galaxydue to 404 not found error.Experiment 2: Interactive App (succeed with known issues)
After git clone this repo, run
sh install-galaxy.shto git clone Galaxy release_19.09 to./galaxyfolder and install dependencies in the virtual environment under./galaxy.venvfolder and_condaunder./galaxy/database/dependenciesfolder. This script will also build custom visualization pluginsAfter completing the
sh install-galaxy.sh, galaxy can be launched as an interactive app. Inbefore.sh.erb,galaxy.yml(general configuration),job_resource_params_conf.xml(job resource configuration for users to select),job_conf.xml(job runners configuration) are generated.Galaxy is mounted on
/node/${HOSTNAME}/8080=galaxy.webapps.galaxy.buildapp:uwsgi_app()Database:
Data files are stored in the user’s dataroot (default to
~/.galaxy/configured in Galaxy.ymlAuthentication:
Galaxy.ymltakes in the user email address as the user authentication in the single-user mode. User identification has to be in email format, so${USER}@osc.eduis passed toGalaxy.ymlas a temporary solution. Further authentication can be configured as described [here].(https://galaxyproject.org/admin/config/external-user-auth/)Select Job Runner
The users select the tool runner before starting the app. The developer adds destinations to job config file and assigns the user-selected runner to default.
bc_osc_galaxy/template/before.sh.erb
Lines 51 to 62 in 9afb0b7
Job runner field:
Three types of job runners we consider
1. Run tools locally
Pros:
Cons:
2. Submit tool jobs to the cluster
Pros:
Cons:
3. Users configure the runner before submitting each tool job.
Pros:
The user can choose the default runner or specify resources:
If the user chooses to specify resources:
Cons:
GET DATAsection, the users can't specify resources.Example: configure dynamic running tools with user-defined resources
As an example, I configured BED-to-GFF tool to provide resources selection fields. Steps to configure a tool to use the dynamic runner based on resource parameters selected by the user:
1. Specify the parameters in the job resource configuration file (https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/config/sample/job_resource_params_conf.xml.sample). The following example contains
coresandwalltime. The input field can be an input box or a dropdown with several options.bc_osc_galaxy/template/before.sh.erb
Lines 67 to 76 in 9afb0b7
2. Add rules to https://github.com/galaxyproject/galaxy/tree/dev/lib/galaxy/jobs/rules directory to match job resource parameters entered by the user to destinations. The following example matches the default runner to the default destination. If the user enters cores and walltime, we construct a resource list and run the tool with pbs runner.
bc_osc_galaxy/install-galaxy.sh
Lines 11 to 43 in 9afb0b7
3. Add dynamic job runner to the
<plugins>in job config file.rules_modulefield indicates the location of the files we created at step 2.bc_osc_galaxy/template/before.sh.erb
Lines 47 to 49 in 9afb0b7
4. Inside of
<resources>in the job config file, add a group of parameters we defined at step 1 and define the group id.bc_osc_galaxy/template/before.sh.erb
Lines 38 to 40 in 9afb0b7
5. Inside of
<tools>in the job config file, specify the id="tool_id", destination="destination_id" and resource="resource_group_id"bc_osc_galaxy/template/before.sh.erb
Lines 41 to 43 in 9afb0b7
Tools are defined under
https://github.com/galaxyproject/galaxy/tree/dev/toolsin the xml files. To find tool id, it's defined in the<tool>tag such as<tool id="createInterval" name="Create single interval" version="1.0.0">.