Review guidelines

[nuest]

Do we need review guidelines? I think the reviewers might appreciate a little input, and it should help using similar standards across al reviews.

Draft for review guidelines

• Reviewers are encouraged to provide a public pull request review, but intermediate interactions with the submitter are of course also welcome, either by commenting on a pull request or using the chat. Please try to limit use of other/private publication channels (email, Skype) to cases where confidentiality is required.
• Reviewers should not concern themselves with the format of the submission.
• Review criteria Please use these criteria to structure the review and use keywords "high", "medium", "low", "not applicable".
  • Relevance and contribution to the workshop call and the topic of reproducible research
  • Quality of the submission (language, organization of content, completeness, clarity)
  • Reproducibility (if submission contains computation)
  • Originality (unless reports on pre-published work)
  • Significance of results (where appropriate)
  • Scientific rigour (references etc.)
• Be professional and fair.

Resources

• http://www.cgl.uwaterloo.ca/smann/Research/review-conference.txt
• http://www.dfi.org/pdf/Reviewer%20Guidelines.pdf
• http://fusion2016.org/Review_guidelines
• http://aom.org/annualmeeting/reviewerguidelines/

[nuest]

Should reviewers simply provide a list using the criteria and rank them on a point system?

review summary: "quality: high", "relevance: medium", "reproducibility: inapplicable", "originality: low", "rigour: high"

[MarkusKonk]

Hier meine Kommentare:

• was hat es mit "multiple comments" auf sich? - das ist mir so nicht geläufig
• muss man erwähnen, dass es kein blind review ist? - Interactions sind ja möglich
• bzgl. Review criteria: ich würde nicht sagen, dass die Wichtigkeit absteigend ist, hängt von dem Paper ab und scientific rigour finde ich genauso wichtig wie z.B. quality.

Zu deiner Frage:
Das wäre ja in etwa so wie die vom OSC das geamcht haben. Fänd ich in Ordnung. Vielleicht das Punkte-System und ein finaler Kommentar (5 Sätze)?

[nuest]

@MarkusKonk Who is OSC?

I clarified the "public" aspect and removed the criteria prioritization.

[foost]

I support Daniel's comment to use pre-defined ratings, e.g. "high", "low", "not applicable", etc., and to encourage reviewers to use the provided criteria and rating categories. I think a more structured review will help us in case we have to drop contributions for which the reviewer recommendation is accept, because there are simply too many of them (see issue 2 step 7).

[MarkusKonk]

OSC = Open Science Conference

[nuest]

moved to wiki: https://github.com/o2r-project/agile-2017/wiki/Guidelines-for-reviewers