[JOSS Review] Questions and suggestions for the Software Paper

[blsqr]

Hi @rogersamso, @enekomartinmartinez, and co-authors 👋

I'm one of the reviewers of your submission to JOSS and have a few questions and suggestions regarding the software paper (currently on branch paper_pysd, PDF version here).

First of all: The paper is well written and structured and I find it easy to understand. While I would probably not be considered a “non-specialist audience”, I think that readers from other backgrounds will be able to understand the key motivation and ideas of PySD and SD modelling. 👍

I have some questions and suggestions regarding the text and the figures. I thought about opening a PR for these but I think we’ll have a quicker turnaround if I just list them here and you decide how to proceed:

1. Typos:
   • Line 27: Stella’s or Vensim’s domain specific languages (both need to end with an 's and language should then be plural)
   • Line 60: causal (not casual) loop diagrams
   • Line 62: thoroughly
2. Superfluous abbreviations (not used again, would suggest to drop them):
   • Line 10: MIT
   • Line 24: GUI
3. Figure 1: Is there a higher-quality figure available, ideally in a vector-based format? I would deem this one not fit for publication.
4. Line 24: Would it be worth highlighting that Stella and Vensim are propriertary and (afais) commercial tools? It’s done in the statement of need, but might be more helpful early on in the text.
5. Line 29: Perhaps rephrase to “… of the extensive data-science capabilities of the Python ecosystem.”?
6. Line 31: “has 26 public releases, 2.2.4 being the most recent”: I would drop that line, as it becomes outdated very quickly and provides little additional value in the context of the software paper – the linked repository is a much better indicator for development speed and contribution history.
7. Figure 2: Any chance to include the term “Parsing Expression Grammar” into the figure or caption to make the PEG abbreviation easier to understand? While you mention the abbreviation in the text, readers that first look at the figures might have an easier time understanding the figure that way. (Not a particularly important remark.)
8. Figure 2 subcaption: I would suggest to drop or rephrase the “NOTE: …” clause. To me, that information feels unnecessary at that point.

And finally, a question regarding the last paragraph before the Statement of Need, where you describe the future development roadmap of PySD: Together with Figure 2 that paragraph represents a large chunk of the paper that does not refer to the submitted version of PySD but to a future one. Are these plans certain enough that you want to include them into the software paper? How far in the future is v3.0?
The roadmap is definitely interesting and gives valuable insight into the current and proposed future workings of the parsing and model building logic. To be clear: I am not proposing to drop or shorten that paragraph, but would primarly want to stress that whatever is written there should be reasonably fixed so that it does not become outdated too easily.

So far from my side. I hope that these comments are helpful – please let me know if there are further questions regarding the above or if there is anything else I can assist with :)

[rogersamso]

Hi @blsqr,
Thank you for taking the time to review the paper, and for the valuable comments. We accepted all your suggestions and they have already been merged into the paper branch. We also made a new version of the first figure in svg format, and added it to the paper.

Regarding your last question, we are working on the v3.0 release in #312. The separation between parsing and building is already complete (all pre-existing test models work again) but we still need to work on the documentation and a few other minor things. We will soon start working on the migration from xarray to numpy. So we are confident that v3.0 will be released before the summer break.

Thanks again for the useful feedback and please let me know if there is anything else we could do to further improve the document.

[blsqr]

Thank you for quickly addressing these points, @rogersamso! :)

Regarding Figure 1: The latest article proof still shows the blurry figure, presumably because it still includes the low-res PNG file? I'm actually not sure if SVG is supported, but PDF definitely is.

Actually, I'm now seeing that figure 2 is a PNG as well. If you have the corresponding SVG file anyway, I would suggest to embed that one as a vector-based graphic as well.

So we are confident that v3.0 will be released before the summer break.

In that case I'm wondering: Have you considered postponing the publication of the JOSS paper until that point? It seems like a major structural change that is not too far in the future. Publishing at a later point would allow to focus on the new structure of PySD rather than describing the old one, making the publication more concise.

I obviously do not know your timeline or other constraints you might have; there are many reasons to publish sooner rather than later. I still wanted to bring it up.

[rogersamso]

Hi again @blsqr,
My bad. The two figures are now at higher res and in pdf.

In that case I'm wondering: Have you considered postponing the publication of the JOSS paper until that point? It seems like a major structural change that is not too far in the future. Publishing at a later point would allow to focus on the new structure of PySD rather than describing the old one, making the publication more concise.

That is a very good point, indeed.

As we mention in the paper, PySD has been mature and functional for some time now (since 1.0). The basic problem PySD solves has not changed either, and will remain the same in further releases. That is what we aim to present in the paper.

However, we keep looking for ways to make the code more maintainable, to make simulations run faster, and to improve the user experience. Though these are not essential developments in most cases (unless performance is a must), the users (including ourselves) always welcome them.

That is what release 3.0 is all about: the average user may see a notable increase in simulation performance, but not much else. However, from a developers perspective, 3.0 will bring non backwards compatible changes which will cause inconveniences to the libraries that rely on PySD. Therefore, the objective of presenting it before it is released, is to catch the attention of developers (who may or may not be willing to write builders for other programming languages), and to give them the possibility to review and future-proof what we propose for the 3.0 release. As I mentioned in my previous comment, the PR may be ready before summer, but if we manage to onboard new contributors, we would be more than willing to delay the 3.0 release a few extra months.

Of course this is not a final decision, and we will adapt to what reviewers think is best.

[blsqr]

Thank you for the detailed response, @rogersamso.

Therefore, the objective of presenting it before it is released, is to catch the attention of developers (who may or may not be willing to write builders for other programming languages), and to give them the possibility to review and future-proof what we propose for the 3.0 release.

This is a very good idea. I see more benefits in that approach compared to a postponed publication that would focus solely on v3.0. 👍

[blsqr]

With #312 nearing a merge, I find the revised software paper more concise now. Very nice! 👍

Two minor details:

• In the bib entry of Forrester1971, I believe the [by] jay w. forrester can be removed? If not, the name should be made title case in the bib file.
• It is a bit unfortunate that the "co-first author" footnote symbols are different and the footnote are repeated. I don't know if JOSS's markdown parser (which I believe is pandoc) supports this, but it may be worth trying if this syntax is working? It would probably mean adding something like [^*] to both author names in the table (* becoming the symbol), and adding [^*]: co-first author at the beginning of the text? If that does not work, I would propose to inquire regarding the best practice of denoting the authorship status. What do you think?

Also, are there representative publications that can be cited for the LOCOMOTION project, despite it being still ongoing? Given that it's not unlikely that a project website breaks in the future, I find it more robust to (additionally) cite a journal publication, if available.

[enekomartinmartinez]

In the bib entry of Forrester1971, I believe the [by] jay w. forrester can be removed? If not, the name should be made title case in the bib file.

Corrected

It is a bit unfortunate that the "co-first author" footnote symbols are different and the footnote are repeated. I don't know if JOSS's markdown parser (which I believe is pandoc) supports this, but it may be worth trying if this syntax is working? It would probably mean adding something like [^*] to both author names in the table (* becoming the symbol), and adding [^*]: co-first author at the beginning of the text? If that does not work, I would propose to inquire regarding the best practice of denoting the authorship status. What do you think?

The example given in the JOSS documentation do as current version of the paper. I didn't manage to find a workaround to that, I tried your suggestion but it didn't work. I know that is possible to have a workaround in latex with cleveref package and something like:

Hello\footnote{\label{fnote:my_note}My foot note.}

World\footnoteref{fnote:my_note}

But I don't know if there is a way to have the same workaround with pandoc.

[blsqr]

The example given in the JOSS documentation do as current version of the paper

I missed that example, sorry. In that case, let's just leave it as it is. Thanks for looking into it!

[pdebuyl]

Hi all. I read the discussion above and I am not sure that I understand fully the authors' choice regarding the version that serves as a reference.

I would find it unusual to refer to a version <3 if 3 is out. Although it is an important release, it seems to build on the same overall package organization, and it would make sense to consider version 3.0.1 the reviewed version. @blsqr and authors, what do you think?

PS: JOSS changed some of its infrastructure recently, hence the mismatch for the co-first author syntax. You can remove the footnotes and add a equal-contrib: true line below the co-first authors instead. Similarly, corresponding authors are indicated by a line corresponding: true below the authors. See the related issue that I opened at JOSS.

[enekomartinmartinez]

Hi @pdebuyl
Thanks for your suggestions, I will include them in the paper.

The current version of the paper is about 3.0.0, we updated it and noticed the reviewers take that into account. Version 3.0.1 includes the suggestion made by @blsqr during the revision process and a minor bug fix.

[blsqr]

it would make sense to consider version 3.0.1 the reviewed version. @blsqr and authors, what do you think?

Would fully agree: some version 3.0.x should be considered the reviewed version.

The version probably has to be adjusted in the review issue to no longer say 2.x?

new infrastructure

That's a very nice improvement! 👍