deploy: 2b2fe02

.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: eced35b325e0b62ce14cebe2dbf58f52
+tags: 645f666f9bcd5a90fca523b33c5a78b7

_sources/details/application.md

@@ -0,0 +1,34 @@
+(details-application)=
+
+# Application
+
+The CI2024 Artifact Evaluation will be an asynchronous event held 23 September to 25 October 2024 where authors will collaborate with the AE committee to evaluate and improve the quality of their artifacts.
+
+We provide further information on the key stages, requirements and registration links for Authors and Reviewers.
+
+## Information for authors
+
+### Stages
+
+We provide below the key stages participants will get involved:
+
+* **Expression of interest:** Participants should fill out a registration form for the challenge and indicate some details of the artifacts they will submit for evaluation.
+* **Submission deadline:** Authors should submit their artifacts via the HotCRP platform.
+* **Artifact evaluation (5 weeks):** Authors will collaborate with the AE committee to evaluate and improve the quality of their artifacts.
+
+### Application form
+
+The application period has now closed. Thank you for your interest.
+
+## Information for reviewers
+
+The role of a reviewer is to sharpen the submission and ensure it is ready for publication in the EDS book following the competition. 
+Reviewers should assess the accuracy, completeness, and clarity of their reproduction process and findings. 
+Here is some information for reviewers:
+
+* **Expression of interest:** Reviewers should fill a registration form for the challenge and indicate some details of their areas of interest, programming proficiency and other relevant skills.
+* **Artifact evaluation (5 weeks):** Reviewers will collaborate with the assigned submissions to evaluate and improve the quality of their artifacts.
+
+### Application form
+
+Registration is now open at this [link](https://docs.google.com/forms/d/1hZBOqT9nnl7MZ0HqtuOdfWkdObs5FA9040Ock0oOrWo/prefill) through 15 Aug @ 23:59 AOE.

_sources/details/timeline-process.md

@@ -0,0 +1,35 @@
+# Timeline
+
+Authors of full-papers that are accepted to the Cambridge University Press (CUP) journal proceedings will be encouraged (but not required) to submit supporting materials for Artifact Evaluation following the Climate Informatics conference. Reviewing will take place over 5 weeks.
+
+These are the key dates (all dates are in the [Anywhere on Earth (AOE / UTC-12) timezone](https://www.timeanddate.com/time/zones/aoe):
+
+* Expression of interest (Authors): Mon 28 June
+* Expression of interest (Reviewers): Thu 15 August
+* Artifact platform opening: Mon 2 September
+* Artifact submission (deadline): Thu 12 September
+* Start of review phase: Mon 23 September
+* Further clarifications until Thu 24 October
+* Final decisions sent to authors: Fri 25 October
+
+# Process
+The evaluation process uses an optional, lightweight single-blind system. Authors and Reviewers will be added to the HotCRP system of the CI2024 Artifact Evaluation, and the authors will submit their artifacts to the system.
+
+## Authors
+Authors may iteratively improve their artifacts during the process to overcome small technical problems, but may not submit new material that would require substantially more reviewing effort.
+
+The detailed submission process is as follows:
+
+1. Register your [interest](details-application) to submit an artifact.
+1. Read the [Submission Guidelines](guidelines-authors) page for details on artifact preparation.
+1. Upload your artifact to Zenodo (recommended), or otherwise make it available via a stable URL (i.e. the URL should not change if you later make updates to the artifact; and ideally, the URL has a good chance of continuing to exist well into the future).
+1. Finalize your submission on HotCRP with a link to your artifact.
+
+## Reviewers
+A committee of reviewers, the Artifact Evaluation Committee (AEC), will review the submitted artifacts against three [criteria](overview-evaluation): is the software available? Is it functional, and can it be used to reproduce the (central) claims or thesis of the paper? The reviewers will interact with the authors to suggest how to improve the reproducibility if any issues are discovered.
+
+The detailed review process is as follows:
+1. Register your [interest](details-application) to review artifacts.
+1. Follow the instructions to access the assigned artifacts on HotCRP.
+1. Review the artifacts and provide feedback to the authors.
+1. Score the artifacts based on the [criteria](overview-evaluation) provided.

_sources/guidelines/authors.md

@@ -0,0 +1,49 @@
+(guidelines-authors)=
+
+# Guidelines for Authors
+
+## What authors should provide in their artifact submission
+
+Artifact submissions require three parts, as described below. 
+
+1. An overview of the artifact.
+
+2. A non-institutional URL (or more, e.g. one for the code and one for the data) pointing to either: a single file containing the artifact (recommended), or the address of a public source control repository. The URL must be non-institutional to protect the anonymity of reviewers. Acceptable URLs can be obtained from Google Drive, Dropbox, Gitlab, Zenodo, and other providers. URLs may be generated using reviewer-only links.
+
+3. A hash certifying the version of the artifact at submission time: either an md5 hash of the single file (use the md5 or md5sum command-line tool to generate the hash), or the full commit hash for the repository (e.g., from git reflog --no-abbrev).
+
+Artifacts do not need to be anonymous as reviewers will be aware of author identities.
+
+The overview should be a document (markdown, text, or PDF) consisting of the following parts:
+
+1. Introduction briefly explaining the purpose of the artifact and how it supports the paper.
+
+2. Hardware requirements to evaluate the artifact. If the artifact requires specific hardware (e.g., many cores, disk space, GPUs, specific processors), please provide instructions on how to gain access to the hardware (it may be required that a video call is scheduled if, e.g., access to a cluster cannot be granted but some aspects can be presented over a call).
+
+3. A Getting Started guide, giving instructions for setup and basic testing. List any software requirements. The instructions should take roughly 30 minutes to complete. Reviewers will follow the guide during an initial kick-the-tires phase and report issues as they arise. The Getting Started Guide should be as simple as possible, and yet it should stress the key elements of your artifact. Anyone who has followed the Getting Started Guide should have no technical difficulties with the rest of your artifact.
+
+4. Step-by-Step Instructions for how you propose to evaluate the functionality of your artifact (with appropriate connections to the relevant sections of your paper) and reproduce any experiments or other activities that support the conclusions in your paper. You should:
+
+* List all claims in the paper and state whether or not each is supported.
+
+    - For supported claims, say how the artifact provides support and how to produce these results. Explain the expected outputs and how to interpret the outputs relative to the paper. If there are any expected warnings or error messages, explain those as well. Ideally, artifacts should include sample outputs and logs for comparison.
+
+    - For unsupported claims, explain why they are omitted. For example, this can be reasonable if it requires very long runs on clusters, or involves data sets that are proprietary. In this case, the authors should provide some mechanism by which claims can be tested by some other mechanism to establish the veracity of the approach, e.g., a smaller run or a different sample data set, with expected outputs.
+
+* Please indicate how long you expect any runs to take that will typically take more than half a minute.
+
+It can be useful to describe how to perform runs on smaller input data sets. Reviewers may choose to run on smaller inputs or larger inputs depending on available resources.
+
+5. A Reusability Guide for how you propose to evaluate the reusability of your artifact. Explain which parts of your artifact constitute the core pieces which should be evaluated for reusability. Explain how to adapt the artifact to new inputs or new use cases. Provide instructions for how to find/generate/read the documentation about the core artifact. Articulate any limitations to the artifact’s reusability.
+
+When packaging your artifact, please keep in mind: a) how accessible you are making your artifact to other researchers, and b) the fact that the artifact committee members have a limited time.
+
+A good way to package artifacts is as a virtual machine (VM). VMs give an easily reproducible environment that is somewhat resistant to bit rot. They also give reviewers confidence that errors or other problems cannot cause harm to their machines. Source code artifacts should use Docker or another build tool to manage all compilation and dependencies. This improves the odds that the reviewers will be able to quickly and painlessly install the artifact — without getting lost in environment issues ([e.g. what Python do I need?!](https://xkcd.com/1987/)).
+
+## Support
+
+There was a 1-hour informal session on 20th May, 2024 for authors who are considering submitting artifacts to give an overview of the approach. The recording is available [here](https://drive.google.com/file/d/1A21CuzRpe7sQcTgc5Hza3Vp5EAWVcmH3/view?usp=sharing).
+
+There will also be a panel about reproducibility at CI2024 which will provide some motivation and inspiration.
+
+The artifact chairs are available to ask any questions about the process or to ask advice about preparing the artifact or if any parts of the process are unclear.

_sources/guidelines/reviewers.md

@@ -0,0 +1,30 @@
+(guidelines-reviewers)=
+
+# Guidelines for Reviewers
+
+Reviewers will be recruited to the Artifact Evaluation Committee (AEC) from the [Climate Informatics community](http://www.climateinformatics.org) (join [here](https://groups.google.com/g/climate-informatics-news)) and [Turing Environment and Sustainability Grand Challenge community](https://cassgvp.kumu.io/alan-turing-institute-environment-and-sustainability) (join [here](https://forms.office.com/pages/responsepage.aspx?id=p_SVQ1XklU-Knx-672OE-ZmEJNLHTHVFkqQ97AaCfn9UMTZKT1IwTVhJRE82UjUzMVE2MThSOU5RMC4u)). Please join either (or both!) communities to receive updates on reviewer training opportunities.
+
+## Requirements for reviewers
+We require a minimum level of practical experience in climate data science to participate as a reviewer. This ensures that the review process can focus on technical detail rather than computational literacy. Reviewers will be required to evidence their experience through their own software publications. 
+
+## Expected workload
+
+- Each reviewer will be assigned 2 submissions
+- There will be a 2-hour on-boarding session to get everyone up to speed
+- Each review can be expected to take up to 2 working days to complete, spread over the ~5 weeks of the review process, including time spent communicating with the submitting author using HotCRP. We will consider an artifact that takes longer than 2.5 working days to review to be “unreproducible” for the purposes of this review process.  
+- Final reviews should be submitted by Thursday 24 October.
+
+## Benefits to reviewers
+
+Reviewers will benefit from hands-on training in high fidelity computational reproducibility, which we anticipate will support their own development of reproducible research artifacts. Reviewers will be able to reference their contribution as evidence of leadership culture change towards reproducibility, and invited to co-author a retrospective report to be published in [Environmental Data Science](https://www.cambridge.org/core/journals/environmental-data-science) after the review process is complete. They will be supported by the Climate Informatics Reproducibility team and AEC throughout, thereby strengthening their connections with this highly skilled team.
+
+##  Evaluation process
+
+Reviewers will assess the artifacts against a checklists provided for each ["badge" level to be awarded](overview-evaluation).
+
+For each point, reviewers will be required to briefly note the evidence for this point, anything they have done to validate that point (e.g., what did you need to reproduce a claim), as well as the outcome (negative, neutral, or positive) and a brief reason for their judgment.
+
+## Feedback to the authors
+
+This process has been designed to be collaborative and iterative. During the first 3 weeks, reviewers will be able to feed-back to authors.
+Authors will be encouraged to push improvements to the artifact whenever possible. For example, if essential documentation is missing, then rather than submitting a negative evaluation or providing only the reviewers with the required information, the authors should provide that information via a versioned updated to the artifact.

_sources/intro.md

@@ -0,0 +1,67 @@
+# Welcome 👋
+
+Welcome to the Climate Informatics 2024 Artifact Evaluation Initiative!
+
+```{note} Key Dates
+* __Artifact submission deadline for authors:__ Thursday 12 September 2024
+* __Final decisions sent to authors:__ Friday 25 October 2024
+```
+
+## The Problem
+
+Climate Informatics, like many other communities and fields, has software at its heart. Underlying most publications is a novel piece of software playing some critical role, e.g., embodying a model, processing or analysing data, or producing a visualisation. 
+
+In order for such software artifacts to have the most impact, they should be **available, functional, and reusable**, such that other researchers can benefit from the work, verify the claims of the paper, and then build upon the software to do more great work. These ideals are summarised by the FAIR principles of data, which can be applied to software: research software should be Findable, Accessible, Interoperable, and Reusable ([FAIR](https://www.nature.com/articles/s41597-022-01710-x)). 
+
+The practicalities of achieving FAIR software are non-trivial, and require resourcing for both authors and reviewers, as well as community consensus on requirements and standards. 
+
+## The Solution
+
+In order to help promote FAIR software in our community, Climate Informatics is embarking, for the first time, on an optional _Artifact Evaluation (AE) phase_ for accepted full paper submissions. Those submissions will published as the conference proceedings in [Environmental Data Science](https://www.cambridge.org/core/journals/environmental-data-science) following the traditional peer-review process. 
+
+AE provides an opportunity to embed the values of reproducibility into the publication process in a lightweight opt-in fashion, thus encouraging authors to make software available and the results of the paper reproducible. Submitted artifacts will be assessed by a skilled team of reviewers, who will work with authors to help them develop and share their materials with the highest practical level of computational reproducibility. 
+
+## What are we doing?
+
+We have adopted the AE process of the [Association for Computing Machinery artifact Review and Badging Version 1.1](https://www.acm.org/publications/artifacts), and developed this to fit the specific context of the Climate Informatics community (with minimal changes to retain alignment with this well accepted standard - we do not need to re-invent the wheel!). We have worked with our partners at Cambridge University Press to deliver this process in a way which complements their publication workflow, and does not delay dissemination of the work. 
+
+The next stage is to design and deliver training and resources to the authors so the AE is a positive learning experience, and recruit and train a committee of reviewers to undertake the work of evaluation, with careful awareness of the workload and incentives to contribute their labour to this effort. 
+
+Take a look at published materials and decisions made to date:
+- [Rationale](overview-rationale): Describes in detail the rationale, process, and evaluation criteria in place;
+- [Evaluation guidelines](overview-evaluation): Provides a checklist which will be used in reviewing and assessing whether the artifacts are available, functional, and reusable.
+- [Issues](https://github.com/alan-turing-institute/climate-informatics-2024-ae/issues) is where we are openly recording our decision making processes.
+
+## What do we need?
+
+We need authors who are keen to submit their artifacts for evaluation, and reviewers who would like to contribute to the growth of reproducibility in our community! 
+
+We will be recruiting reviewers in the coming weeks, and working with them to deliver the training and support they need to undertake this work. Take a look at the [benefits for reviewers](https://github.com/alan-turing-institute/climate-informatics-2024-ae/blob/main/process.md#benefits-to-reviewers) to understand how participating as reviewers will be a valuable opportunity for you, and stay tuned on the [Turing Environment and Sustainability Slack](https://alan-turing-institute.github.io/climate-informatics-2024/contact/#slack) for invitations to join the review committee!
+
+## Who are we?
+
+This work is being led by the [Reproducibility working group](https://alan-turing-institute.github.io/climate-informatics-2024/team#reproducibility) of the Climate Informatics 2024 organisers. We are pleased to connect with you if you would like to participate in the leadership or delivery of this work!
+
+## Contact us
+
+### Slack
+
+Connect with us via [Turing Environment and Sustainability Slack](https://alan-turing-institute.github.io/climate-informatics-2024/contact/#slack) - tag or dm Cassandra Gould van Praag (Turing Environment and Sustainability Senior Research Community Manager), or Alejandro Coca-Castro (CI2024 Reproducibility Chair)
+
+### Email
+
+- Cassandra Gould van Praag (Turing Environment and Sustainability Senior Research Community Manager): cgouldvanpraag@turing.ac.uk
+- Alejandro Coca-Castro (CI2024 Reproducibility Chair): acoca@turing.ac.uk
+
+## Acknowledgements
+
+The AE process is developed following [Association for Computing Machinery artifact Review and Badging Version 1.1](https://www.acm.org/publications/artifacts).
+
+This repo and README follow the best practice for community participation of _The Turing Way_ {cite:ps}`a-ttw_2022`.
+
+## References
+
+```{bibliography}
+:style: alpha
+:keyprefix: a-
+```