Generic GUI for nf-core pipelines

[apeltzer]

Hi everyone,

an idea which was already briefly discussed here:

We could produce a generic tool in tools that can produce a graphical user interface for end users to generate an appropriate configuration for the pipeline:

• I'd like to use Kivy for this https://kivy.org/#home :Anybody has a better idea/candidate?
• I'd like to produce a working nextflow config in the end, that can be used for running the actual pipeline

Definition could be like Phil proposed it here:

nf-core/eager#11 (comment)

Happy for any feedback - I thought about this for EAGER2.0, but would be happy to make it generic for all kinds of nf-core pipelines!

@ewels @andreas-wilm @sven1103 @maxulysse (and others who are interested?)
Let me know what you think!

[sven1103]

1. I like the idea
2. I love the framework's name
3. I would love to participate and exercise some MVP and mocking for GUI elements

Do you want to do a config builder from scratch, or read in from an already existing config file and only manipulate values?

[apeltzer]

I thought about doing it from scratch. The idea was also to be able to create a "reproducible" GUI, that uses a GitHub Revision to produce a dynamic interface.

That would ideally mean, that we can simply use the same generic way for multiple pipelines and for multiple versions of the respective pipelines (which broke my neck in EAGER1.X tbh).

Happy if you want to join in here too!

[ewels]

I'm pretty excited about this and have been thinking on and off about it recently. I hate to spoil the fun, but I wonder if it's worth making a command line wizard tool first as that should be much faster and allow us to iterate a few times to figure out what works..? This should be relatively simple using click I think..

I'm thinking that we actually want two sub-tools:

1. Generic system config - general nextflow stuff that will apply to all pipelines
   • Executor type
   • HPC queue name
   • Resource limits
   • Email address
   • ...
2. Pipeline config - parameters specific to running this pipeline this time
   • Input data
   • Output directory
   • Reference locations
   • params flags

The pipeline params config could be used for both types of tools of course. I made a very small starting effort with the methylseq pipeline in ewels/nf-core-methylseq@922d4db

Other question - this has potential for being more generic than just nf-core. We could write it as a core nextflow feature instead, or a stand-alone tool. This would of course affect our choice of language. Any thoughts @pditommaso?

[maxulysse]

Love the idea!
Happy to join if you want me as well ;-)

[apeltzer]

Linking in @jfy133 too - please comment if you have ideas, hints or suggestions - highly appreciated!

[sven1103]

I hate to spoil the fun, but I wonder if it's worth making a command line wizard tool first as that should be much faster and allow us to iterate a few times to figure out what works.

Not at all a show blocker. The CLI is just another implementation of a view interface. The model behind it stays the same, no matter if we use a GUI or CLI.

Thats the cool thing about MVP :)

But I agree, with the CLI comes less overhead in implementation at first, so we could start with this, and add a view implementation with Kivy afterwards. Just a suggestion, me brain-storming here.

[sven1103]

So the core model would be a Nextflow config builder implementation 🏗️

[apeltzer]

I think that would be the easiest way - the question is whether we want to do this with clickas @ewels suggested it or more nextflowy in groovy/kotlin/java which would make it possible to add this as a core (?) feature to nextflow @pditommaso ?

[ewels]

Sorry - yes, with click I was thinking if we trialled this as a subcommand in the nf-core tools package. If it's standalone including the CLI then we can use whatever language.

[apeltzer]

Lets do this in that way! CLI first, GUI once we "converged" in terms of functionality?

[pditommaso]

I don't think this should be a core nextflow feature, but it could be a useful community tool.

[sven1103]

Lets do this in that way! CLI first, GUI once we "converged" in terms of functionality?

The GUI could be an option of the nf-core subcommand, and would be just an implementation of a view interface (In Python there is no such thing, but abstract class). If we do it properly for the CLI subcommand (proper Model, proper Presenter, CLI as View extending an abstract class nfView), than it should be easy). The GUI would then use the same abstract view class and override the functions.

Given an option --gui, you then just build the GUIView Instead of CLIView, the rest stays the same.

[jfy133]

As a user rather than developer (and one who has to regularly teach command-line novices), a command-line wizard would certainly be enough for an initial tool. It would be easier to explain how to set up a nextflow run that way, rather than trying to explain a config file in lieu of a GUI.

Could someone make a short sketch how the CLI wizard could look like? What would be nice for me in @ewels example EAGER2.0 comment would always be having a help text in each params block.

[ewels]

Very roughly, I could see the flow working something like:

• Is a ~/.nextflow file present?
  • If no, start system config wizard
  • Auto-detect any job schedule systems, user project affiliations, docker / singularity installations, other stuff that we can use for defaults
  • Prompt / confirm for a system config file to save as ~/.nextflow, which will be used for all nextflow pipelines
• Launch wizard for a specific pipeline
  • Run through all pipeline params using bundled parameter descriptors that we add to the pipeline - required or not, types, allowed variables and so on.
  • Generates a nextflow.config file in the current working directory
  • Finish by providing or even launching nextflow. Shouldn't need any command line parameters because everything will be present in the system-wide and cwd config files.

[ewels]

...as a super basic starting point for testing and playing around, we could use nf-core tools to generate a JSON file with default values, then pipe this into cookiecutter for the command line prompts to generate a nextflow.config file. Then we don't even have to write any interface stuff at all to begin with.

[sven1103]

Probably something we could work on during the Hackathon in August? :)

[ewels]

Absolutely 😁 At least some discussion / planning anyway.. I'm fairly keen to focus on getting some pipeline releases done where possible, as fancy tools aren't worth much if we don't have any workflows available, but this would be a super nice thing to get started with and would certainly lower the boundary for using pipelines.

[pditommaso]

Sorry, I have found this conversation only now :/

Yes, the implementation of a nextflow UI is an old topic and I think it cannot be postponed any more.

My idea was to extend the nextflow parameters definition so that it was possible to provide extra metadata that would allow the rendering of a basic UI for the pipeline execution, ie. the parameter type, range, etc.

Originally it was supposed to be a core feature, but more recently I was thinking it would make more sense to implement this is application that could be extended as a separate front-end. Still not sure what's the best approach.

I would propose to have brainstorming on this topic to decide how to move on on this feature. If you agree please choose your best day using this doodle. I'm including @skptic.

[apeltzer]

My idea was to extend the nextflow parameters definition so that it was possible to provide extra metadata that would allow the rendering of a basic UI for the pipeline execution, ie. the parameter type, range, etc.

That sounds exactly as what we'd like to have! I'd rather invest time in (helping?) to develop a generic GUI than having to develop a new one for EAGER2!

Don't really know which way is best either - making it a more core feature would allow to ship it to everyone though, whereas a standalone application would keep nextflow slim ;-)

[pditommaso]

Adding to this thread also @wikiselev and @ODiogoSilva in case they want to join the call on the nextflow front-end.

[wikiselev]

Thanks, Paolo! I am adding @micans here, he is doing most of the the NF pipeline development in our group now. Though he is a very command-line oriented person, he may be able to help here.

My personal feeling is that it can be very useful for the end users, if done in a UX driven way. However, for our group at the moment it is not the most important functionality.

[ODiogoSilva]

Thanks @pditommaso for the invitation. This is an interesting discussion and having some sort of general system for setting up the parameters and other general configuration of a pipeline seems pretty useful. Especially if this tool would only need to read from a standard JSON/yaml file, as @ewels suggested (or from the nextflow params directly), regardless of how the pipeline was built.

I really like Kivy framework (I developed and maintain a kivy GUI tool) and I think it should be fairly simple and straightforward to build a front-end for this. It seems to me that we mostly need some forms with text inputs, dropdowns and toggle elements, which are rendered depending on the metadata provided. In any case, I would be happy to contribute where I can. The only issue I see with the GUI approach, is that I often create the pipelines or modify the parameters directly on a cluster, which lacks X server and X11 forwarding. In that regard, the CLI version seems more useful, even if generally considered (wrongly, IMO) less user-friendly. The trick, I think, is to have a system that makes setting up the configuration much easier than just changing params and directives directly in the nextflow.config file(s).

Moreover, if this could also be extended beyond the parameters would be super nice. I often need to tweak cpus/ram/clusterOptions for some processes and having a more guided and automatic way of doing it would be very nice.

[ewels]

The only issue I see with the GUI approach, is that I often create the pipelines or modify the parameters directly on a cluster, which lacks X server and X11 forwarding.

I had thought the same thing. I think the CLI will be widely used, but also it's possible that we could run a GUI tool locally which generates a nextflow.config file is then transferred to the working directory on the cluster. As long as the pipeline is on GitHub it shouldn't be a problem to generate this in a separate location to where the workflow is run.

[pditommaso]

The client is surely the more portable approach, but also a thing web client can run remotely and access via web browser.

[apeltzer]

@sven1103 can you link our gDocs discussion document in here? Or should we transfer this to a project on Github for the hackathon (or just in general? ) ...

[ewels]

https://docs.google.com/document/d/10XY9qKp59ccHM5ftO-OVkMt5vtoiFYHlz9ovb62XZtg/edit?usp=sharing

[ewels]

I think we can close this issue now as it has matured and moved on elsewhere:

• We have a (prototype) parameters schema ✅
• We have a cli wizard 🧙🏻‍♂️
• Plans are in motion to build a web-based GUI for launching pipelines ⚡️

Remaining is the option of having a local GUI to build the config file, but if there’s still interest in that then I think discussion can move to a new issue.

Phil