[REVIEW]: optimade-python-tools: a Python library for serving and consuming materials data via OPTIMADE APIs

[whedon]

Submitting author: @ml-evs (Matthew L Evans)
Repository: https://github.com/Materials-Consortia/optimade-python-tools
Version: 0.16.4
Editor: @jgostick
Reviewer: @hungpham2017, @jamesrhester
Archive: 10.5281/zenodo.5517803

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/ef9a3b0e180c8668093a23b5a069c419"><img src="https://joss.theoj.org/papers/ef9a3b0e180c8668093a23b5a069c419/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/ef9a3b0e180c8668093a23b5a069c419/status.svg)](https://joss.theoj.org/papers/ef9a3b0e180c8668093a23b5a069c419)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@hungpham2017 & @jamesrhester, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @jgostick know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @hungpham2017

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@ml-evs) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution. #3542
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @jamesrhester

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@ml-evs) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @hungpham2017, @jamesrhester it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

PDF failed to compile for issue #3458 with the following error:

 Can't find any papers to compile :-(

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.26 s (869.5 files/s, 150555.3 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
JSON                            18              0              0          17080
Python                         121           3594           3254          11813
Markdown                        65            508              0           1205
YAML                             9             89              6            496
SVG                              1              0              0            232
Bourne Shell                     5             25              5            217
HTML                             1              1              0             50
Dockerfile                       1              6              1             13
INI                              1              0              0             11
CSS                              1              1              0              8
-------------------------------------------------------------------------------
SUM:                           223           4224           3266          31125
-------------------------------------------------------------------------------


Statistical information for the repository '13096f790102ec83f762a1a2' was
gathered on 2021/07/07.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Abhijith Gopakumar               4           158             27            0.31
Adam Fekete                     37          1649            671            3.87
Andrius Merkys                   2             4              4            0.01
Casper Welzel Anders           246         15044           7938           38.37
Diether John Estrada             2            67             24            0.15
Donny Winston                   31          1487           2275            6.28
Fawzi Mohamed                    7           173             74            0.41
Johan                            2           325             57            0.64
Kamal Choudhary                  1           112              2            0.19
Leopold Talirz                   6           558             44            1.01
Markus Scheidgen                 4           390              3            0.66
Matthew Evans                  303         14461           6329           34.71
Michael Wu                      44          3778           2085            9.79
OPTIMADE Developers             30            30             30            0.10
Shyam Dwaraknath                48           513            764            2.13
Thomas Purcell                  11           534            245            1.30
dependabot-preview[b             2             2              2            0.01
dependabot[bot]                  6            16             13            0.05

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Adam Fekete                 384           23.3         19.8                7.55
Andrius Merkys                3           75.0          9.6                0.00
Casper Welzel Anders       8004           53.2         13.5                7.41
Donny Winston               127            8.5         26.0                0.00
Fawzi Mohamed                 7            4.0         24.7                0.00
Johan                       325          100.0          0.2                2.15
Kamal Choudhary              82           73.2         12.8               13.41
Leopold Talirz              119           21.3         23.9                1.68
Markus Scheidgen            112           28.7         18.4                2.68
Matthew Evans              9214           63.7          9.0                7.24
Michael Wu                    4            0.1         32.3                0.00
OPTIMADE Developers           1            3.3          0.0                0.00
Shyam Dwaraknath            183           35.7         13.8                1.09
Thomas Purcell               92           17.2         24.8                6.52
dependabot[bot]               4           25.0          3.8               25.00

[ml-evs]

@whedon generate pdf from branch joss

[whedon]

Attempting PDF compilation from custom branch joss. Reticulating splines etc...

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[ml-evs]

FYI @jgostick, I've just been made aware that James Hester (@jamesrhester) is a collaborator on an ongoing project with one of the authors of this package (@CasperWA) and some of the authors of the OPTIMADE specification more widely.

[jamesrhester]

@jgostick, I have read the COI policy and should declare here that author @CasperWA and myself are both currently involved in the EMMO crystallography project. I have also participated in several zoom meetings of the OPTIMADE consortium, neither of which concerned this work, of which I was not aware until now. Neither my institution nor myself have received any funding for this work and I believe that I can assess this paper objectively.

Please advise if JOSS is prepared to waive any actual or perceived conflicts of interest in this case.

[jamesrhester]

@jgostick @ml-evs just bumping this in case my request for a waiver on COI has been missed.

[whedon]

👋 @jamesrhester, please update us on how your review is going (this is an automated reminder).

[whedon]

👋 @hungpham2017, please update us on how your review is going (this is an automated reminder).

[ml-evs]

Perhaps the managing editor from the pre-review (@danielskatz) could help with this too? I guess the review cannot start otherwise...

[danielskatz]

Given that we have to balance the needed expertise to review a submission and the fact that this often means that potential reviewers will be involved in community projects with the developers, this needs to be judged on a case-by-case basis. I think that, given @jamesrhester's comments, we can waive the COI and proceed, if that seems ok with the editor of this submission, @jgostick

[jgostick]

It seems I am not being notified by github of mentions. I'm very sorry for taking so long to check in, I just do it once a week or so.

Regarding the COI, I would say that the more ably a person can review a submission the closer they are to the COI threshold. I'm fine with things proceeding as they are.

[ml-evs]

@whedon generate pdf from branch joss

[whedon]

Attempting PDF compilation from custom branch joss. Reticulating splines etc...

[whedon]

PDF failed to compile for issue #3458 with the following error:

 Error reading bibliography file paper.bib:
(line 168, column 32):
unexpected ","
expecting white space or "}"
Looks like we failed to compile the PDF

[ml-evs]

@whedon generate pdf from branch joss

[whedon]

Attempting PDF compilation from custom branch joss. Reticulating splines etc...

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[ml-evs]

We have just updated some of our references in the paper. Please let me know if we can be of any help with the review.

[jamesrhester]

A few queries and suggestions below - the remainder I'll raise as issues in the repository as they don't concern the paper itself:

1. Github "contributors" suggests that Michael Wu has contributed significantly more to the code than some of the co-authors. Is there any reason he is not a co-author (I see he is acknowledged)?
2. The Lark reference points to pypi, but the other code references are to Github. Is there any reason the Lark pypi link is preferred over Github?
3. There is no mention in the paper of any alternatives to optimade-python-tools. If there are none, a sentence to this effect could be added, e.g. "The authors are not aware of any other packages with similar functionality".
4. The paper could be made appropriate to a broader audience if a few small
   clarifications are included as follows:

• The term "API" can include APIs for programming languages. While it
  eventually becomes clear to the reader that web APIs are the particular
  intended meaning, I suggest that in the "Summary" when the term "API"
  is first introduced, clarify that a standard for web-based access is
  being discussed, e.g.

"Access to these databases has been made available through bespoke web-based
protocols, commonly termed Application Programming Interfaces (APIs), typically
implemented through specially-constructed stateless HTTP queries."

• Likewise, the term "REST" API is used without definition. I suggest adding
  "(stateless)" after the first use of the acronym "REST".

[ml-evs]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[jgostick]

@whedon recommend-accept

[whedon]

No archive DOI set. Exiting...

[jgostick]

@whedon set 10.5281/zenodo.5517803 as archive

[whedon]

OK. 10.5281/zenodo.5517803 is the archive.

[jgostick]

@whedon set 0.16.4 as version

[whedon]

OK. 0.16.4 is the version.

[jgostick]

@whedon recommend-accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.commatsci.2012.10.028 is OK
- 10.1038/s41597-021-00974-z is OK
- 10.5281/zenodo.4195051 is OK
- 10.1088/1361-648x/aa680e is OK
- 10.1038/s41597-020-00638-4 is OK
- 10.1038/s41524-020-00440-1 is OK
- 10.1038/s41597-020-00637-5 is OK
- 10.1063/1.4812323 is OK
- 10.1038/s41524-017-0048-5 is OK
- 10.1016/j.commatsci.2014.10.037 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

👋 @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof 👉 openjournals/joss-papers#2604

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#2604, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[kthyng]

@ml-evs Please update the metadata in your Zenodo archive so the author list and title exactly match your JOSS article.

[kthyng]

Paper looks fine.

[ml-evs]

@ml-evs Please update the metadata in your Zenodo archive so the author list and title exactly match your JOSS article.

Hi @kthyng, is this a hard requirement? Our automatic Zenodo release archives predate this submission by a couple of years and include all historic contributors to the package.

We could release an individual archive for this version+paper, if necessary.

[kthyng]

@ml-evs No it isn't a hard requirement — it gives a clear association of which software archive matches the JOSS publication to have a nice final package to point to. The individual archive sounds like a nice way to go, but you can choose your preference.

[ml-evs]

@ml-evs No it isn't a hard requirement — it gives a clear association of which software archive matches the JOSS publication to have a nice final package to point to. The individual archive sounds like a nice way to go, but you can choose your preference.

In that case, I think we would prefer to keep the single point of truth as the current archive, without changing the metadata (which would just mean removing people) - this means that the links to later versions are kept - assuming that is okay with you.

[kthyng]

@ml-evs That's fine, you can choose. We're all ready to wrap up then!

[kthyng]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.03458 joss-papers#2605
1. Wait a couple of minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.03458
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[kthyng]

Congrats on your new publication @ml-evs! Thanks to editor @jgostick and reviewers @hungpham2017 and @jamesrhester for your time, hard work, and expertise!

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.03458/status.svg)](https://doi.org/10.21105/joss.03458)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.03458">
  <img src="https://joss.theoj.org/papers/10.21105/joss.03458/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.03458/status.svg
   :target: https://doi.org/10.21105/joss.03458

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss

[ml-evs]

Congrats on your new publication @ml-evs! Thanks to editor @jgostick and reviewers @hungpham2017 and @jamesrhester for your time, hard work, and expertise!

Thanks again everyone, on behalf of all my co-authors!