Status of GRNsight manuscript submission to PeerJ Computer Science

[kdahlquist]

The GRNsight manuscript was submitted to PeerJ Computer Science on May 18, 2016 with simultaneous submission to the PeerJ Preprint server. Submission ID is 10823.

It will now be checked by PeerJ staff, who will email me if any changes are required.
After passing checks, it will be assigned to an Academic Editor, who will invite reviewers to carry out peer-review.

Co-authors on the manuscript are going to get e-mails--you may need to do something to confirm your authorship.

The PDF that was generated for the review has been pushed to the repository as well.

[dondi]

Yay to us! I've confirmed authorship.

[kdahlquist]

Once it officially enters the review process and is live as a preprint, I'll create a news item about this.

[kdahlquist]

Over the weekend, I received several status update e-mails as follows:

1. The manuscript passed staff checks and was sent on to be assigned an academic editor.
2. I was asked to confirm that we wanted to submit it to the preprint server, to which I replied "yes".
3. I was notified that the manuscript has been assigned to an academic editor (but not which one). It said that it will be sent out for review and we will be notified again when the reviews are in and the editor has made a decision.
4. The preprint is now undergoing staff checks and after it passes, it will be published.

[dondi]

The preprint has been published! https://peerj.com/preprints/2068/

[kdahlquist]

We have received a decision on our manuscript. The letter from the editor and the reviews are found below. I will start parsing this into issues in the next few days.

---

Dear Kam,
Thank you for your submission to PeerJ Computer Science.
It is my opinion as the Academic Editor for your article - GRNsight: a web application and service for visualizing models of small- to medium-scale gene regulatory networks - that it requires a number of Minor Revisions.
Although never a guarantee, if you address the editorial changes below (from myself and the reviewers) there is a good likelihood the article will eventually be accepted.
You will also receive a staff email soon with any remaining technical changes to your submission. You must also address these changes before resubmission.
Although not a hard deadline, we expect you to submit your revision within the next 40 days.
With kind regards,
Shawn Gomez
Academic Editor, PeerJ Computer Science

---

Editor's Comments MINOR REVISIONS

Please address the reviewer critiques, as both provided a number of comments as well as relevant potential additions/modifications (e.g., export capability, package accessibility, etc).

---

Reviewer 1 (Anonymous)

Basic reporting
No Comment

Experimental design
Dahlquist et al present GRNsight, a service for visualising gene regulatory networks of the ‘small-to-medium-scale’. The tool is available via a web application and the source code is also provided. On receiving a manuscript such as this I immediately try and use the application and I compare it to my current preferred tools in the domain. The web application loads quickly and is reasonably intuitive to use. It is not so clear that the force algorithm is disabled on specific nodes once that node has been manually relocated to a new position. This may be described under the help section, but I could not find it although it is mentioned in the manuscript (line 217). The visual style of the network is good. Certain behaviours of the visualisation are not compatible with internet explorer (which the authors do not mention is supported � line 239). However,, the only platform for which mouse over on links revealed the underlying weights was firefox on windows 7. Neither safari, chrome not firefox on OSX or Chrome on windows 7 showed the weights. I would like to see an option on the tool for showing or hiding all weights.

Validity of the findings
My two preferred tools for laying out networks are BioTapestry (http://www.biotapestry.org/) and YeD (http://www.yworks.com/products/yed) . In neither case could I see a simple way to import the information from the excel file specified by the authors as output from the GRNmap package to reproduce the visualisation that they show in GRNsight. The specific issues are weighted networks � neither of these packages provide a direcct method of importing these data from a matrix in excel. However GRNsight itself does not work directly from a matrix � rather it parses that network through JSON. The central constraint for GRNsight is the file format output by GRNmap. I could write a parser for that matrix into a format such as GraphML and then import that in to YeD and have far more tools available to me.

The authors themselves recognise this to some degree and argue that their tool is aimed at doing one thing well. I agree that this tool does present a network from GRNmap well. However, given the existence of 47 other tools which already do something similar it must be an exceptionally good tool. I am familiar with cytoscape, but not Gephi and so tried Gephi to visualise a network similar to that presented here. It is possible, but as the authors state non-trivial.

Whilst I agree that simplicity is key (cognitive load � line 103) II am less clear about the ‘understanding the biological results of the model’ enabled by GRNsight. The authors discuss these on lines 281-289 giving generalised interpretations of issues such as feed forward motifs, highest in-degree and the regulatory chains. Yet as far as I can see, these are all determined by visual inspection. Furthermore none of these rely on the weights, which the distinguishing feature of GRNsight. I would be able to identify these more easily in a system such as YeD and do not require the weight information to do so � a simple sif file format givving the directionality between two nodes is sufficient.

The visualisation of the weight parameters described in lines 290-322 is the key behaviour here. Figure 5 D,E show clearly the impact of the addition of this weight information, yet E is the clearest visualisation and the only node within it that is located in anything close to its original position is Ace2 � almost every other node has been moved by hand. This revveals a flaw in the implementation of the force-spring layout algorithm as applied in GRNsight. Given the small area of the view port and the constraint that all nodes remain within it, the layout is sub optimal as the force-spring cannot reach its most relaxed state. Furthermore when looking at figure 5 I am always drawn to panels C and F as being the most informative view. Thus I would argue that the force-spring algorithm used here does not provide much benefit to the layout of the network other than separating nodes from one another. The useful layout requires manipulation. Taking the network from E and recreating it in YeD revealed that just a hierarchical clustering and layout gets you closer to E without any manual intervention.

Comments for the Author
Ultimately the tool presented here is useful for interpreting the results of GRNmap, but I would be unlikely to use it in any other situation. As it does not accept a standard input file type, the output of any other network analysis package requires conversion in to the matrix format required here. Similarly, the tool provides no export function (the option in the File menu remained stubbornly greyed out) and so I can’t take a network from GRNsight and utilise it elsewhere. I also can’t use GRNsight to convert the GRNmap format to something I might like to use elsewhere. GRNmap itself looks to be a very interesting package and I would like to explore it further, but I would be looking at converting its output into something I could use in a number of other pipelines.

The authors refer to future features coming in version 2 (lines 323-329). I encourage them to consider implementing at least one standard filetype for displaying graph data within their tool. Be it sif, graphml or even gml, it would significantly increase the utility of the tool as it currently exists. Alternatively this tool not be considered stand alone from GRNmap.

My comments are rather focussed on the tool and its usability, less on the manuscript itself. The manuscript is generally well written and clear. It’s weakness lies in the arguments about biological insight derived from the visualisation. If I were to parse the matrix into a graphml file format, I could visualise these networks (complete with weightings and line endings etc) in a wide array of tools and � I would argue �– extract more biological value from the interpretation. I would encourage the authors to expand on this section of the manuscript if possible.

---

Reviewer 2 (Manuel Corpas)

Basic reporting

The article "GRNsight: a web application and service for visualizing models of small- to medium-scale gene regulatory networks" is well written and documented. I have found disappointing the level of review of other similar network visualisation tools that exist out there. They only provide comparison agains Cytoscape and Gephi. They do not refer (at least in the introduction) to the Cytoscape.js web application, only to the console application. This to me raises concerns in an otherwise exhaustive work, providing ground for not having extensively researched other similar tools in the field, which is quite crowded. A search in the BioJS registry (http://biojs.io/; an NPM based repository of biological web application) of the word "network" retrieves 12 components. Not all of them are necessarily relevant for this publication, but at least some of them should be compared against and a case should be made so that it is made apparent how GRNsight provides valuable new functionality that is not redundant.

You do mention Cytoscape.js in line 175 but the justification of creating GRNsight because of the future possibility of implementing other D3.js visualisations is not well founded in my opinion. The http://biojs.io/d/biojs-vis-interactions-d3 network visualisation component does use already D3 for network visualisation.

The supplementary figures are very helpful to the understanding of the article.

Experimental design
N/A

Validity of the findings
I am not sure that the visualisation of 75 nodes or 150 edges is the kind of magnitude that would be valuable to many potential users. I would find it more impressive if the capacity to render nodes was in the order of thousands (even though this may be impractical to visualise and some data reductions might be necessary).

To me the bits that I have found most useful for the tool are:

• The visualisation is pleasing and intuitive
• The documentation is extensive and easy to read
• The demos allow users to quickly grasp the functionality
• The article is clear and the results show the relevance of the functionality
• The emphasis on testing and best practice are well appreciated although not complete, see below
• Networks can be uploaded via an xlsx file

I would have also liked some more emphasis on the "findable" and "reusable" aspects of open source software principles. I do believe some mention to "FAIR" principles could be useful: Findable, Accessible, Interoperable, Reusable. This has been done with data sharing (http://www.nature.com/articles/sdata201618) but this applies to software.

Comments for the Author
Bioinformatics web tools like GRNsight are published in the literature but they are not made accessible via a centralised repository like the BioJS registry. I would thoroughly recommend authors to make accessible their tool via the BioJS registry. The requirements for making a package accessible through the BioJS registry are minimal:

• The source code has be made available via GitHub
• It has to be made available in the Node.js Package Manager (NPM), the package manager for JavaScript
• The "biojs" keyword has to be included in the "package.json" file of NPM

If the authors had searched the BioJS registry for the keyword "network" they would have found components that could have at least compared against. By not checking the BioJS registry and not including GRNsight in it they have missed the opportunity to increase the exposure of their tool and potentially lose valuable engagement with a community of reference who might even keen to contribute to the code. It is thus important to appear that this tool is not coded in isolation.

[kdahlquist]

I received an e-mail today from PeerJ saying that no technical changes were required for our manuscript and that we just need to address the editorial changes found in the previous comment on this issue.

[kdahlquist]

I'm working on breaking out the feedback from the reviewers into individual, actionable issues. My draft of the response to the reviewers where I'm starting with a copy of the reviews will be deposited to the repository shortly.

[kdahlquist]

I've gone through the reviewers comments and made or updated issues related to their comments. They are all priority 0.5 for the moment until we decide what we are going to tackle first at our meeting today. There are other things to address in the manuscript, but I only made issues that would require changes to the code or website. The related issues are:

• Make GRNsight package accessible via BioJS registry #286: make GRNsight package accessible via BioJS registry
• Export data to cytoscape sif and graphml format #287: export functionality (SIF, GraphML)
• Import data from cytoscape sif and graphml #288: import functionality (SIF, GraphML)
• Update documentation page with new sections and descriptions of new features #283: clarify in documentation that nodes can be manually relocated and document new features (basics are at least there for now)
• Mouseover edges not displaying weights on Safari, Chrome, Firefox on OSX #284: bug, mouseover edges not displaying in certain browsers

These issues are beyond the scope of what we can accomplish in the time window for revising our manuscript:

• Option for showing or hiding all weights #285: option for showing or hiding all weights
• Create an option to have a small, medium, and large bounding box for different sized screens #159: bounding box is not large enough to let the force spring algorithm to completely relax
• Investigate hierarchical layout #290: investigate hierarchical layout
• Investigate collapsible nodes #291: investigate collapsible nodes

[kdahlquist]

I have finished an initial pass through the response to reviewers and pushed it to the repository. I have referenced issue numbers throughout and there are a couple questions for @dondi also highlighted in green. I wrote it as if we have implemented all requested features/changes, but we might need to revise based on what we can actually accomplish.

I created two new issues, #290 and #291 that are in response to Reviewer 2, but are probably beyond the scope of what we can reasonably accomplish in the next few weeks.

There is a list of issues having to do with the code and documentation listed in a previous comment. Here is a list of tasks @kdahlquist needs to tackle from within the manuscript:

• of course, write about all new features implemented
• a more thorough discussion of competing tools in the intro and/or discussion. This would include tools specifically mentioned by the reviewers, BioTapestry and yED, tools found in BioJS registry, and the tools mentioned in the review we cited by Pavlopoulos et al. (2015).
• expand the discussion section "GRNsight Facilitates Interpretation of GRN Model Results" to defend visual inspection (via Tufte) and to say more about what visualizing the weights does for us.
• discuss how we comply with FAIR principles: findable, accessible, interoperable, reusable
• discuss deposition in BioJS registry, Elixir, Bioinformatics.org, and bioinformatics.ca
• redo Figure 4 with new screenshot
• update test statistics with Istanbul
• update Google Analytics results

[kdahlquist]

Work tonight: added discussion of F, A, and I of FAIR to manuscript. Wrote executive summary section of response to reviewers, did a little minor reformatting.

Note to self: situated knowledges: https://conceptsinsts.wikispaces.com/Situated+Knowledges+(Schaffer)

[kdahlquist]

The response to reviewers is essentially done and pushed to master. @dondi, please take a look. Three things to look out for:

1. My discussion of GraphML in terms of the technical language
2. A reference to Tufte (highlighted in greeen)
3. I think I captured all of our discussion from yesterday, but especially please look it over for tone.

There is one section that I have not completed which has to do with discussing the "47" other tools. I left that out for the moment because I want to work on the manuscript before returning to that in case I run out of time.

So, I'm moving on to the manuscript for tonight and tomorrow. I still need to finish the FAIR section, edit the biological interpretation section, and deal with the Intro, so quite a bit of work still.

Please do chime in with something from Tufte, soon. Thanks.

[dondi]

There are some Tufte quotes from this section of the Wikipedia page on Data Visualization that may fit the bill (they cite the primary source in case we use them). I particularly like the conclusion of the bulleted quote, which says “Graphics reveal data. Indeed graphics can be more precise and revealing than conventional statistical computations.”

There are also some non-Tufte sources on this Stanford Data Visualization course. Perhaps the campus library access privileges can deliver full-text versions of those. The Card/Mackinlay/Shneiderman reference should be pretty authoritative if there is a good quote from there; those three are pretty seminal folks in visualization/interaction design. (van Wijk is no slouch either, but I just happen to be more familiar with Card/Mackinlay/Shneiderman)

[dondi]

OK, modifications/revisions have been made to the cover letter, reviewer response, and text-only paper. I turned on Track Changes on all documents to make it easy to spot differences:

• For the cover letter, I deleted the unfinished feature regarding bounding box size.
• Most of the revisions are in the reviewer response document. Primarily, I added some GraphML detail, wrote up a Tufte/Card/Mackinlay/Shneiderman-based justification for information visualization. There were other minor edits in spots.
• The only change made to the manuscript was the addition of the Tufte and Card/Mackinlay/Shneiderman references to the list of works cited. They are primarily there for easy cut-and-paste if needed. They can be eliminated if the manuscript turns out to not cite them.

Figure 4 has been updated and pushed to the repository, replacing the prior version.

[kdahlquist]

I've got to go home. I'm not finished, so I'll definitely need tonight and tomorrow to get all of this wrapped up. I'll see what I can do tonight, but most likely I'll be in the office tomorrow. I've pushed what I've got so far. The FAIR discussion is long and rambly still and I need to clean it up. The problem I am having is that if I do it justice, it has to be long. I also added some stuff about web service persistence that I thought was applicable.

[dondi]

OK, I have done a read-through of it, and pushed revisions/wordsmithing/comments via track changes. Ultimately I did not change much; the early portion feels pretty solid and flows well. The latter sections have a little more, but in the end I did not see any dealbreakers, just minor tweaks and wordsmithing. Hope this last push goes well!

[kdahlquist]

Got to the final stage of uploading everything and saw that they require a manuscript with tracked changes. I had overlooked this, thinking that they wanted a rebuttal letter or a tracked changes manuscript. I'll have to generate this manually looking at the two versions. I think I need to go home for dinner; I'll do this later tonight. We've got about 13-14 hours left on the clock...

[kdahlquist]

Forgot to upload new screenshot figure, too.

[dondi]

Arg, so sorry to hear about these last glitches…but at least it looks pretty close now.

[kdahlquist]

My little one is finally asleep so I'm on the home stretch. Next update when it's submitted.

[kdahlquist]

IT IS SUBMITTED!!!

Reviewing PDF pushed to repository here: https://github.com/dondi/GRNsight/raw/master/documents/manuscripts/peerj-computerscience-2016/revisions/peerj-reviewing-10823-v1.pdf

Resisting the urge to close this issue: instead will change label from priority 0 to ongoing.

Forgot to add with a whole 10 hours to spare!

[dondi]

YES! Cheers and a big exhale :)

True, sounds appropriate that we keep this open until it is truly and officially published. But for now, kudos on the accomplishment and now it's out of our hands 💯

[kdahlquist]

The submission passed staff checks (actually as of yesterday) and has been passed to the Academic Editor.

[kdahlquist]

Noticed that updating our preprint with the revised version of the manuscript was not automatic. I've e-mailed to get instructions on how to proceed.

[kdahlquist]

So, submitting revisions to the Preprint are manual. I just submitted the revised version.

[kdahlquist]

The new version of the preprint is live! https://peerj.com/preprints/2068/

[dondi]

I've gotten my basic membership, and I just wanted to report that the payment page that I saw had the full list that you mentioned.

[kdahlquist]

The GRNsight paper has been accepted for publication! Congratulations to us all!

The next step is a list of Production Tasks from the PeerJ staff and after we complete that, the proofs.

A reminder to @bengfitzpatrick to pay the membership fee ASAP; I'll take care of mine today.

[dondi]

Woohoo! 👍 💯 That was pretty good turnaround. Hope the Production Tasks are largely by-the-numbers type work that we can just power through. Release party release party release party 🍻🍷🍸🍹🍾

[kdahlquist]

I confirmed with the publisher that we would like to publish the reviews and response to authors with our article. The production editor says that we should receive the proofs in about 10 business days.

[kdahlquist]

Proofs are here. @dondi and I will review and return ASAP.

[dondi]

OK, the updated architecture diagram is there, in revisions/figures.

[kdahlquist]

Got it, although now I'm wondering whether the export to SIF and GraphML somehow needs to be annotated on this as well. I can ask them for another day if this is too much to deal with right now. I'd rather take the time and get it right than rush it. Let me know.

[dondi]

OK, let me see if I can get to it when I'm back in the office. Shouldn't take too long, though I'll note that in terms of connections, it would be more like the JSON goes back to the server which then produces the exported file. So this will entail new edges, and not relabeling of existing ones.

[kdahlquist]

OK. I thought it might be a little more complicated. Like I said, I can ask for another day if need be. It's more important to get this right than to rush to get it back today.

[dondi]

OK, I managed to put something up there but will need to finalize my course prep now. Maybe it’s good to ask for that extra day anyway because I made some choices on this version that might need revision or discussion. Take a look and see how it strikes you.

[kdahlquist]

At first glance, it looks OK. But, I am going to e-mail PeerJ and ask for the extra day so that we have a chance to talk about it later at the GRNsight meeting.

[dondi]

OK, I finally managed to get some sit-down-and-tweak time with the diagram, and I’ve pushed what I could manage. I couldn't get the edges to curve exactly as in the sketch, but the diagram is still an improvement from before.

As for a revised caption, how about:

GRNsight architecture and component interactions. The server provides a web API that accepts files in Microsoft Excel workbook (.xlsx), SIF (.sif), and GraphML (.graphml) formats and converts them into a unified JSON representation. A converse import function accepts this JSON representation and converts it into either SIF (.sif) or GraphML (.graphml) formats. The web application server provides code and resources for the graphical user interface that displays this JSON representation of the graph.

[kdahlquist]

Thanks, it looks good. I'll get this submitted later today.

[kdahlquist]

The annotated proof, revised Figure 3, and document explaining the changes have been uploaded to PeerJ.

[kdahlquist]

It gives me great pleasure to announce that our article has been published!!!

https://peerj.com/articles/cs-85/

Spread the word about the seminar @kdahlquist is giving on Friday, September 16 at 3:00 in LSB 120 and the Release Party afterwards on the LSB 2nd floor terrace!

With a record 49 comments, I'm closing this issue :)