Add a web UI that allows a user to launch and monitor a workflow execution

[pditommaso]

The goals of this enhancement is to provide nextflow with a web graphical user interface that allows a user to:

• provide input parameters
• launch the workflow execution
• monitor the process executions
• stop/kill the workflow execution

Input parameters

Nextflow already implements an input parameters mechanism that allows a user to provide input values as command line options.

The same will be used to display an input form allowing a user to provide a workflow inputs. However in order to render an input field are required extra metadata that currently are not available. The required metadata will be a subset of the HTML5 field input types and validation attributes (to be defined).

UI input fields will be defined in the nextflow.config files, using a built-in method called field (temporary name). For example:

params {
   alpha = field foo, type: 'text', label: 'Alpha value'
   beta = field bar, type: 'file' label: 'Beta value'
   delta = field baz, type: 'choice', label: 'Delta value'  
} 

The first argument will define the field default value (for consistency with the current mechanism). The other named attributes will define the input field metadata.

Web server

Nextflow will need to embed an web server container that will take care of render the user interface and display the workflow/processes runtime informations.

Possible candidates are:

• Jetty
• Spark
• Ratpack
• Native HTTP server
• NanoHttpd

The web server instance will need also to manage a nextflow session, run it in the background, capture the standard output and display in as a web page for monitoring purpose.

[pditommaso]

RFC @emi80 @pbelmann

[pbelmann]

@pditommaso It's great that you created this issue and I would really like to contribute to this feature. But since I'm still new to nextflow it is not clear to me how you want to build such an application. I think there are two ways:

1. The web ui could be a standalone project that embeds nextflow and uses the nextflow api like your Cmd classes/interfaces. This way a user would have to download the packaged gui (e.g. a debian package). The advantage of this approach is that you don't have to change your code.
2. Nextflow command line provides a --gui flag which starts the web ui. If you/we could define an interface that a GUI library should implement, it should be easy to develop the library independent of the nextflow main core code. Furthermore it would allow to exchange GUIs.

If you thought of a different way to implement this, please let me know.

Web server

Nextflow will need to embed an web server container that will take care of render the user interface and display the workflow/processes runtime informations.

Possible candidates are:

Jetty
Spark
Ratpack
Native HTTP server
NanoHttpd

The web server instance will need also to manage a nextflow session, run it in the background, capture the standard output and display in as a web page for monitoring purpose.

If you want to implement it according to the first point, my first thought was using play framework. It is stateless (no servlet container (glassfish,tomcat,...etc)) and it allows you to package the whole application in a debian and different other package formats.

If you want to implement it according to the second point I think all of the above frameworks have sufficient functionality but I think I would choose Spark or Ratpack. However this is an important point so maybe a pros cons list would help.

[pditommaso]

I was thinking to implement it as a subproject, in this way it could take advantage of the implicit download mechanism already implemented in Nextflow. See how works the nextflow console command for example.

It could be done both as a new command (e.g. nextflow launch <pipeline>) or using an option on the run command line as you are suggesting.

I'm more in favour for having a new command because this would simplify to handle specific parameters related to the web server (for example: port number, etc).

I know Play framework but I think is oversized for this use case. Spark and Ratpack are interesting alternative but they requires Java 8. Not sure I want drop the support for Java 7 at this time.

In my opinion it should not be discard the Native HTTP server implemented in the JRE. At the end it only needs to manage a very few requests. Also I guess most part of the UI will be managed on the client side using one of these modern template engine like Ember, Angular or React JS.

Unfortunately I don't know any of them, do you have any experience with these tools?

Anyway I agree that we should list pros/cons both for the server and the template engines.

I will create a new branch, trying to put together a skeleton. Then we could work together on this.

Thanks!

[dtrudg]

Per request on twitter I'm detailing a situation we have at UTSW.

We have a requirement that will result in us developing a web-interface for nextflow - although the web app will not be limited to that. Here's the scenario:

Use Case

We have a variety of groups on campus implementing bioinformatics (and other biomedical) workflows. Typically these groups have a few local bioinformaticians who are putting together existing tools, their own R and python code, with bash scripts to direct the workflow.

Many of these groups are now using our HPC facility to run these workflows. There is a wish to provide a standardized web-based means to run data through many of the workflows, using our cluster. The web-interface must be as simple as possible for users, to support execution of workflows by users who have no experience of, and no expectation of learning any command-line interaction

This is a solution for standardized workflows with minimal parameters. It is not intended to provide complex parameter options / modular workflows that can be built from components etc. We have other solutions for that type of approach.

The infrastructure used to provide the solution needs to be such that the existing bash-scripted workflows can be translated without major effort (rules out approaches needing tool wrapping etc).

Reflecting the experience level of the intended final users, the web app must present documentation about the workflow, and have some facility for providing a simple results visualization if desired.

The approach needs to be generic - not limited to a particular domain within bioinformatics etc.

The approach

At present we are moving toward:

A loose definition of a 'workflow package', which is a git repo containing:

• The workflow itself, implemented in nextflow. Plus dependent scripts, but not binaries (will use HPC module system).
• Documentation in a format e.g. markdown that can be rendered easily in a web app.
• Minimal test-data for the workflow
• An optional R Shiny results visualization app
• A meta-data file describing the above, workflow input params etc.

The web app will then:

• Ingest workflow package repos. which are developed by groups on campus.
• Allow users to upload/download data/results
• Render simple workflow submission forms from the workflow meta-data
• Render the workflow's docs on the web
• Launch the nextflow workflows on our cluster and track completion
• Serve the optional Shiny viz app against the results

For us it's going to be a python/django project.

[pditommaso]

Thanks a lot for your detailed comment. Your use case has a bit broader scope than the goal of this feature, above all because I guess it needs to be customized/integrated with your infrastructure.

In any case do you have any plan to release it as a FOSS project?

[dtrudg]

Yes, had no doubt it would be outside the general scope. it'll be a pretty customized solution.

Open Source release will likely be attempted, and will depend on our Technology Development Office.

[joshuabhk]

@pditommaso

I am very interested in this feature and willing to help if you need help to implement this feature.
Is there any further progress about it?

[pditommaso]

As far as I know @emi80 has made some very little progress thought I think he stopped to work in it. Any contribution is more than welcome. Are you comfortable with Java/Groovy and HTML5 development? Thanks for your interest/support.

[joshuabhk]

I am more than happy to help out and implement this functionality.
I am quite familiar with html5 but not that comfortable with groovy since I
learned it from nextflow. But I think can handle groovy since I developed
Java application some time ago.

[pditommaso]

Having some experience in Java you can be fully proficient with Groovy in a few days. I'm very happy you are interested to contribute to this feature. I will rebase the webui branch to synchronise it with the latest changes. Also let's wait a feedback from @emi80 to update us on any progress he may have done. Feel free to start playing with the code and ask questions if needed.

[joshuabhk]

Yes, I will wait for @emi80. But I want to make it clear that I do not have
any intention step on @emi80's tow. So let me know if you have any
reservation @emi80!

[pditommaso]

Of course!

[emi80]

Hi @joshuabhk! Of course any contribution on this is very welcome. No reservation has been made on the feature.

I just started to exepriment a bit with the components for the interface using Foundation (gist here). Though, after playing a bit with it, I think Bootstrap would probably be a better choice. Of course any suggestion on a better way to implement the components is welcome, as long as we keep things simple.

So, please go ahead and keep us posted on the progress! 👍

[pditommaso]

I've rebased the webui branch to the latest changes.

[joshuabhk]

Thanks! I will take a look at it!

[ewels]

Hi @pditommaso,

Another offer of help here - I can't write Java or Groovy to save my life (yet!), but I have a fair bit of experience of building HTML interfaces. If you can draw what you want on a piece of paper I'd be happy to put together static pages for you.

Phil

[ewels]

@emi80 - I recreated your form using bootstrap, code is here and web preview is here.

Bit flat and boring currently, but hopefully gives a useful starting point. Note that I used the v4 beta version of Bootstrap. Let me know if you have any questions / ideas.

@joshuabhk - apologies, hope I didn't tread on your toes!

[ewels]

ps. @pditommaso - minor suggestion, your commit contains the following line:

if( opts.containsKey('hint'))
  this.hint = opts.hint

Personally, I'd change this to help (long descriptive string to go under the field) and add an additional field called placeholder (example content shown in field when nothing entered). I find hint a bit ambiguous.

Also note that some stuff I put into my static page wouldn't be easy / possible with the nextflow config option. For instance, the example dropdown box and 'free HTML' generally. Also grouping / fieldsets.

[pditommaso]

@ewels Thanks for your contribution and suggestions. Soon or latter we will work on this.

[joshuabhk]

Not a problem at all. Actually I am glad that you can work on it. I am
hitting an unexpected high workload, so my plan to work on the web UI are
essentially ruined and postponed.

Best,

Bong-Hyun

On Wed, Nov 2, 2016 at 7:29 PM Phil Ewels notifications@github.com wrote:

@emi80 https://github.com/emi80 - I recreated your form using
bootstrap, code is here
https://gist.github.com/ewels/6517179684decdd68de0cef4d05108fc and web
preview is here
https://rawgit.com/ewels/6517179684decdd68de0cef4d05108fc/raw/c83b015cca7ddfe10d263d0dbd9470ff26462eae/index.html
.

Bit flat and boring currently, but hopefully gives a useful starting
point. Note that I used the v4 beta http://v4-alpha.getbootstrap.com/
version of Bootstrap. Let me know if you have any questions / ideas.

@joshuabhk https://github.com/joshuabhk - apologies, hope I didn't
tread on your toes!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#88 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGkVDOWRIzLf7DNjj_55ulgUePK2Nvphks5q6RzXgaJpZM4Gj-77
.

[philge]

Is this web UI feature now available in Nextflow?

[pditommaso]

No, unfortunately no.

[nephantes]

we have dolphinnext, if anybody wants to try.
Still working on it and finalizing the publication. It would be great if we can get feedbacks.
You can, create, monitor, launch nextflow pipelines in any server you want.
https://github.com/UMMS-Biocore/dolphinnext

[pditommaso]

Nice! need to find time to try it.

[nephantes]

I would definitely help if you need. We also have a web version dolphinnext.umassmed.edu.
We also started a documentation; https://dolphinnext.readthedocs.io.