preregistration_modelling.qmd

---
title: "Adaptive Preregistration for Modelling"
subtitle: "A template suitable for use in Ecology and Conservation with accompanying rationale and explanation."
description: "Note: Black boxes provide an explanation or justification of a particular section or preregistration item. Text contained in 'preregistration item' boxes requires a response."
authors:
  - Elliot Gould
  - Hannah Fraser
  - Libby Rumpff
  - Jian Yen
  - Megan Good
  - Chris Jones
sidebar: false
bibliography: data/references.bib
toc-location: left
toc-expand: true
number-sections: true
---

# Problem Formulation

::: callout-note
## Rationale & Explanation

This section specifies the decision-making context in which the model will be used or the intended scope and context of conclusions. Important components include the decision maker and stakeholders (including experts) and their view on: i) the nature of the problem or decision addressed and how the scope of the modelling tool fits within the (broader) context (i.e. model purpose; ii) the spatial and temporal scales relevant to the decision context; iii) specified desired outputs; iv) role and inclusion in model development and testing; v) whether they foresee unacceptable outcomes that need to be represented in the model (i.e. as constraints), and; vi) what future scenarios does the model need to account for (noting this may be revised later). It should also provide a summary of the domain of applicability of the model, and reasonable extrapolation limits [@Grimm:2014es].
:::

## Model Context and Purpose

::: callout-note
## Rationale & Explanation

Defining the purpose of the model is critical because the model purpose influences choices at later stages of model development [@Jakeman:2006ii]. Common model purposes in ecology include: gaining a better qualitative understanding of the target system, synthesising and reviewing knowledge, and providing guidance for management and decision-making [@Jakeman:2006ii]. Note that modelling objectives are distinct from the analytical objectives of the model.

The scope of the model includes temporal and spatial resolutions, which should also be defined here [@Mahmoud2009]. Any external limitations on model development, analysis and flexibility should also be outlined in this section [@Jakeman:2006ii].
:::

### Key stakeholders and model users

::: callout-caution
## Preregistration Item

*Identify relevant interest groups:*

-   [ ] Who is the model for?
-   [ ] Who is involved in formulating the model?
-   [ ] How will key stakeholders be involved in model development?
-   [ ] Describe the decision-making context in which the model will be used (if relevant).
:::

### Model purpose, context and problem context

::: callout-caution
## Preregistration Item

*Briefly outline:*

-   [ ] the ecological problem,
-   [ ] the decision problem (if relevant), including the decision-trigger and any regulatory frameworks relevant to the problem,
-   [ ] how the model will address the problem, being clear about the scope of the model i.e. is the model addressing the whole problem, or part of it? Are there any linked problems that your model should consider?
-   [ ] Ensure that you specify any focal taxa and study objectives.
:::

### Analytical objectives {#analyticalobjectives}

::: callout-note
## Explanation

How will the model be analysed, what analytical questions will the model be used to answer? For example, you might be using your model in a scenario analysis to determine which management decision is associated with minimum regret or the highest likelihood of improvement. Other examples from ecological decision-making include: to compare the performance of alternative management actions under budget constraint [@Fraser:2017jf] to search for robust decisions under uncertainty [@McDonald-Madden2008], to choose the conservation policy that minimises uncertainty \[insert ref\]. See other examples in [@Moallemi2019].
:::

::: callout-caution
## Preregistration Item

*Provide detail on the analytical purpose and scope of the model:*

-   [ ] How will the model be analysed and what analytical questions will the model be used to answer?
-   [ ] Candidate decisions should be investigated and are specified a priori. Depending on the modelling context, they may be specified by stakeholders, model users or the analyst [@Moallemi2019].
    -   [ ] Describe the method used to identify relevant management actions and
    -   [ ] specify management actions to be considered included in the model.
    -   [ ] Are there potentially unacceptable management or policy outcomes identified by stakeholders that should be captured in the model, i.e. as constraints?
-   [ ] Are there scenarios that model inputs or outputs that must accommodated? Scenarios should be set a priori, [i.e. before the model is built, @Moallemi2019] and may be stakeholder-defined or driven by the judgement of the modeller or other experts [@Mahmoud2009].
    -   [ ] If relevant, describe what processes you will use to elicit and identify relevant scenarios, e.g. literature review, structured workshops with stakeholders or decision-makers.
    -   [ ] Specify scenarios under which decisions are investigated.
:::

### Logistical Constraints

::: callout-caution
## Preregistration Item

-   [ ] What degree of flexibility is required from the model? Might the model need to be quickly reconfigured to explore new scenarios or problems proposed by clients / managers / model-users?
-   [ ] Are there any limitations on model development analysis and flexibility, such as time or budget constraints, for example, does a model need to be deployed rapidly?
    -   [ ] When must the model be completed by, e.g. to help make a decision?
:::

### Model Scope, Scale and Resolution

::: callout-caution
## Preregistration Item

-   [ ] The choice of a model's boundaries is closely linked to the choice of how finely to aggregate the behaviour within the model [@Jakeman:2006ii] - what is the intended scale, and resolution of the model (temporal, spatial or otherwise)?
-   [ ] Where is the boundary of the modelled system? Everything outside beyond the boundary and not crossing it is to be ignored within the domain of the model, and everything crossing the boundary is to be treated as external forcing (known/unknown), or else as model outputs [observed, or not, @Jakeman:2006ii].
:::

### Intended application of results

::: callout-note
## Explanation

Preregistration Items in this section are relevant to model transferability [@Yates2018] and constraints on generality in model analysis interpretation. How far do can the results be extrapolated based on the study design (data + model + analysis)? For instance, if there are many confounding variables and not enough spatial / environmental replication, then making broader more general claims beyond the stated boundaries of the model (@analyticalobjectives) may not be warranted. However, larger generalisations about results may be acceptable if the data comes from experimentally manipulated or controlled systems.
:::

::: callout-caution
## Preregistration Item

-   [ ] What is the intended domain in which the model is to be applied? Are there any reasonable extrapolation limits beyond which you expect the model should not be applied [@Grimm:2014es]?
:::

## Scenario Analysis Operationalisation

::: callout-caution
## Preregistration Item (delete as necessary)

-  [ ] How will you operationalise any scenarios identified in 1.1.3? For example, how will you operationalise any qualitative changes of interest, such as ‚ 'deterioration' or 'improvement'?
- [ ]  Describe how you will evaluate and distinguish the performance of alternative scenario outcomes
- [ ]  Justify or otherwise explain how you chose these measures and determined performance criteria in relation to the analytical objectives, model purpose and modelling context, such as the risk attitudes of decision-makers and stakeholders within this system
:::

# Define Conceptual Model

::: callout-note
## Explanation

Conceptual models underpin the formal or quantitative model [@Cartwright:2016kr]. The conceptual model describes the biological mechanisms relevant to the ecological problem and should capture basic premises about how the target system works, including any prior knowledge and assumptions about system processes. Conceptual models may be represented in a variety of formats, such as influence diagrams, linguistic model block diagram or bond graphs, and these illustrate how model drivers are linked to both outputs or observed responses, and internal (state) variables [@Jakeman:2006ii].
:::

## Choose elicitation and representation method

::: callout-caution
## Preregistration Item

-   [ ] Describe what method you will use to elicit or identify the conceptual model. Some common methods include interviews, drawings, and mapping techniques including influence diagrams, cognitive maps and Bayesian belief networks [@Moon2019]. It is difficult to decide and justify which method is most appropriate, see Moon et al. [-@Moon2019] for guidance addressing this methodological question.
-   [ ] Finally, how do you intend on representing the final conceptual model? This will likely depend on the method chosen to elicit the conceptual model.
:::

## Explain Critical Conceptual Design Decisions

::: callout-caution
## Preregistration Item

*List and explain critical conceptual design decisions [@Grimm:2014es], including:*

-   [ ] spatial and temporal scales,
-   [ ] selection of entities and processes,
-   [ ] representation of stochasticity and heterogeneity,
-   [ ] consideration of local versus global interactions, environmental drivers, etc.
-   [ ] Explain and justify the influence of particular theories, concepts, or earlier models against alternative conceptual design decisions that might lead to alternative model structures.
:::

## Model assumptions and uncertainties

::: callout-caution
## Preregistration Item

*Specify key assumptions and uncertainties underlying the model design, describing how uncertainty and variation will be represented in the model [@Moallemi2019]. Sources of uncertainty may include:*

-   [ ] exogenous uncertainties affecting the system,
-   [ ] parametric uncertainty in input data and
-   [ ] structural / conceptual nonparametric uncertainty in the model.
:::

## Identify predictor and response variables

::: callout-note
## Explanation

The identification and definition of primary model input variables should be driven by scenario definitions, and by the scope of the model described in the problem formulation phase [@Mahmoud2009].
:::

::: callout-caution
## Preregistration Item

*Identify and define system system variables and structures, referencing scenario definitions, and the scope of the model as described within problem formulation:*

-   [ ] What variables would support taking this action or making this decision?
-   [ ] What additional variables may interact with this system (things we can't control, but can hopefully measure)?
-   [ ] What variables have not been measured, but may interact with the system (often occurs in field or observational studies)?
-   [ ] What variables are index or surrogate measures of variables that we cannot or have not measured?
-   [ ] In what ways do we expect these variables to interact (model structures)?
-   [ ] Explain how any key concepts or terms within problem or decision-making contexts, such as regulatory terms, will be operationalised and defined in a biologically meaningful way to answer the research question appropriately? 
:::

## Define prior knowledge, data specification and evaluation

::: callout-note
## Explanation

*This section specifies the plan for collecting, processing and preparing data available for parameterisation, determining model structure, and for scenario analysis. It also allows the researchers to disclose any prior interaction with the data.*
:::

### Collate available data sources that could be used to parameterise or structure the model

::: callout-caution
## Preregistration Item

**For pre-existing data (delete as appropriate):**

-   [ ] Document the identity, quantity and provenance of any data that will be used to develop, identify and test the model.
-   [ ] For each dataset, is the data open or publicly available?
-   [ ] How can the data be accessed? Provide a link or contact as appropriate, indicating any restrictions on the use of data.
-   [ ] Date of download, access, or expected timing offuture access.
-   [ ] Describe the source of the data - what entity originally collected this data? (National Data Set, Private Organisational Data, Own Lab Collection, Other Lab Collection, External Contractor, Meta-Analysis, Expert Elicitation, Other).
-   [ ] Codebook and meta-data. If a codebook or other meta-data is available, link to it here and / or upload the document(s).
-   [ ] Prior work based on this dataset - Have you published / presented any previous work based on this dataset? Include any publications, conference presentations (papers, posters), or working papers (in-prep, unpublished, preprints) based on this dataset you have worked on.
-   [ ] Unpublished Prior Research Activity - Describe any prior but unpublished research activity using these data. Be specific and transparent.
-   [ ] Prior knowledge of the current dataset - Describe any prior knowledge of or interaction with the dataset before commencing this study. For example, have you read any reports or publications about this data?
-   [ ] Describe how the data is arranged, in terms of replicates and covariates.

**Sampling Plan (for data you will collect, delete as appropriate):**

-   [ ] Data collection procedures - Please describe your data collection process, including how sites and transects or any other physical unit were selected and arranged. Describe any inclusion or exclusion rules, and the study timeline.
-   [ ] Sample Size - Describe the sample size of your study.
-   [ ] Sample Size Rationale - Describe how you determined the appropriate sample size for your study. It could include feasibility constraints, such as time, money or personnel.
-   [ ] If sample size cannot be specified, specify a stopping rule - i.e. how will you decide when to terminate your data collection?
:::

### Data Processing and Preparation

::: callout-caution
## Preregistration Item

-   [ ] Describe any data preparation and processing steps, including manipulation of environmental layers (e.g. standardisation and geographic projection) or variable construction (e.g. Principal Component Analysis).
:::

### Describe any data exploration or preliminary data analyses.

::: callout-note
## Explanation

In most modelling cases, it is necessary to perform preliminary analyses to understand the data and check that assumptions and requirements of the chosen modelling procedures are met. Data exploration prior to model fitting or development may include exploratory analyses to check for collinearity, spatial and temporal coverage, quality and resolution, outliers, or the need for transformations [@Yates2018].
:::

::: callout-caution
## Preregistration Item

*For each separate preliminary or investigatory analysis:*

-   [ ] State what needs to be known to proceed with further decision-making about the modelling procedure, and why the analysis is necessary.
-   [ ] Explain how you will implement this analysis, as well as any techniques you will use to summarise and explore your data.
-   [ ] What method will you use to represent this analysis (graphical, tabular, or otherwise, describe)
-   [ ] Specify exactly which parts of the data will be used
-   [ ] Describe how the results will be interpreted, listing each potential analytic decision, as well as the analysis finding that will trigger each decision, where possible.
:::

### Data evaluation, exclusion and missing data

::: callout-note
## Explanation

Documenting issues with reliability is important because data quality and ecological relevance might be constrained by measurement error, inappropriate experimental design, and heterogeneity and variability inherent in ecological systems [@Grimm:2014es]. Ideally, model input data should be internally consistent across temporal and spatial scales and resolutions, and appropriate to the problem at hand [@Mahmoud2009].
:::

::: callout-caution
## Preregistration Item

-   [ ] Describe how you will determine how reliable the data is for the given model purpose. Ideally, model input data should be internally consistent across temporal and spatial scales and resolutions, and appropriate to the problem at hand
-   [ ] Document any issues with data reliability.
-   [ ] How will you determine what data, if any, will be excluded from your analyses?
-   [ ] How will outliers be handled? Describe rules for identifying outlier data, and for excluding a site, transect, quadrat, year or season, species, trait, etc.
-   [ ] How will you identify and deal with incomplete or missing data?
:::

## Conceptual model evaluation

::: callout-caution
## Preregistration Item

-   [ ] Describe how your conceptual model will be critically evaluated. Evaluation includes both the completeness and suitability of the overall model structure.
-   [ ] How will you critically assess any simplifying assumptions [@Augusiak:2014gz]?
-   [ ] Will this process will include consultation or feedback from a client, manager, or model user.
:::

# Formalise and Specify Model

::: callout-note
## Explanation

In this section describe what quantitative methods you will use to build the model/s, explain how they are relevant to the client/manager/user's purpose.
:::

## Model class, modelling framework and approach

::: callout-note
## Explanation

Modelling approaches can be described as occurring on a spectrum from correlative or phenomenological to mechanistic or process-based [@Yates2018]; where correlative models use mathematical functions fitted to data to describe underlying processes, and mechanistic models explicitly represent processes and details of component parts of a biological system that are expected to give rise to the data [@White2019a]. A model 'class,' 'family'' or 'type' is often used to describe a set of models each of which has a distinct but related sampling distribution [@Liu2008]. The model family is driven by choices about the types of variables covered and the nature of their treatment, as well as structural features of the model, such as link functions, spatial and temporal scales of processes and their interactions [@Jakeman:2006ii].
:::

::: callout-caution
## Preregistration Item

-   [ ] Describe what modelling framework, approach or class of model you will use to implement your model and relate your choice to the model purpose and analytical objectives described in 1.1.2 and 1.1.3.
:::

## Choose model features and family

::: callout-note
## Explanation

All modelling approaches require the selection of model features, which conform with the conceptual model and data specified in previous steps [@Jakeman:2006ii]. The choice of model are determined in conjunction with features are selected. Model features include elements such as the functional form of interactions, data structures, measures used to specify links, any bins or discretisation of continuous variables. It is usually difficult to change fundamental features of a model beyond an early stage of model development, so careful thought and planning here is useful to the modeller [@Jakeman:2006ii]. However, if changes to these fundamental aspects of the model do need to change, document how and why these choices were made, including any results used to support any changes in the model.
:::

### Operationalising Model Variables

::: callout-caution
## Preregistration Item

-   [ ] For each response, predictor, and covariate, *specify how these variables will be operationalised in the model. This should relate directly to the analytical and/or management objectives specified during the problem formulation phase. Operationalisations could include: the extent of a response, an extreme value, a trend, a long-term mean, a probability distribution, a spatial pattern, a time-series, qualitative change, such as a direction of change or, the frequency, location, or probability of some event occuring. Specify any treatment of model variables, including whether they are lumped / distributed, lienar / non-linear, stochastic / deterministic [@Jakeman:2006ii]* .
-   [ ] *Provide a rationale for your choices, including why plausible alternatives under consideration were not chosen, and relate your justification bacj to the purpose, objectives, prior knowledge and or logistical constraints specified in the problem formulation phase [@Jakeman:2006ii].*
:::

### Choose model family

::: callout-caution
## Preregistration Item

-   [ ] Specify which family of statistical distributions you will use in your model, and describe any transformations, or link functions.
-   [ ] Include in your rational for selection, detail about which variables the model outputs are likely sensitive to, what aspects of their behaviour are important, and any associated spatial or temporal dimensions in sampling.
:::

## Describe *approach* for identifying model structure

::: callout-note
## Explanation

This section relates to the process of determining the best/most efficient/parsimonious representation of the system at the appropriate scale of concern [@Jakeman:2006ii] that best meets the analytical objectives specified in the problem formulation phase. Model structure refers to the choice of variables included in the model, and the nature of the relationship among those variables. Approaches to finding model structure and parameters may be knowledge-supported, or data-driven [@Boets:2015gl]. Model selection methods can include traditional inferential approaches such as unconstrained searches of a dataset for patterns that explain variations in the response variable, or use of ensemble-modelling methods [@Barnard2019]. Ensemble modelling procedures might aim to derive a single model, or a multi-model average [@Yates2018]. Refining actions to develop a model could include iteratively dropping parameters or adding them, or aggregating / disaggregating system descriptors, such as dimensionality and processes [@Jakeman:2006ii].
:::

::: callout-caution
## Preregistration Item

-   [ ] Specify what approach and methods you will use to identify model structure and parameters.
-   [ ] If using a knowledge-supported approach to deriving model structure (either in whole or in part), specify model structural features, including:
    -   the functional form of interactions (if any)
    -   data structures,
    -   measures used to specify links,
    -   any bins or discretisation of continuous variables [@Jakeman:2006ii],
    -   any other relevant features of the model structure.
:::

## Describe parameter estimation technique and performance criteria

::: callout-note
## Explanation

Before calibrating the model to the data, the performance criteria for judging the calibration (or model fit) are specified. These criteria and their underlying assumptions should reflect the desired properties of the parameter estimates / structure [@Jakeman:2006ii]. For example, modellers might seek parameter estimates that are robust to outliers, unbiased, and yield appropriate predictive performance. Modellers will need to consider whether the assumptions of the estimation technique yielding those desired properties are suited to the problem at hand. For integrated or sub-divided models, other considerations might include choices about where to disaggregate the model for parameter estimation; e.g. spatial sectioning (streams into reaches) and temporal sectioning (piece-wise linear models) [@Jakeman:2006ii].

:::

### Parameter estimation technique

::: callout-caution
## Preregistration Item

-   [ ] Specify what technique you will use to estimate parameter values, and how you will supply non-parametric variables and/or data (e.g. distributed boundary conditions). For example, will you calibrate all variables simultaneously by optimising fit of model outputs to observations, or will you parameterise the model in a piecemeal fashion by either direct measurement, inference from secondary data, or some combination [@Jakeman:2006ii].
-   [ ] Identify which variables will be parameterised directly, such as by expert elicitation or prior knowledge.
-   [ ] Specify which algorithm(s) you will use for any data-driven parameter estimation, including supervised, or unsupervised machine learning, decision-tree, K-nearest neighbour or cluster algorithms [@Liu2018b].
:::

### Parameter estimation / model fit performance criteria

::: callout-caution
## Preregistration Item

-   [ ] Specify which suite of performance criteria you will use to judge the performance of the model. Examples include correlation scores, coefficient of determination, specificity, sensitivity, AUC, etcetera [@Yates2018].
-   [ ] Relate any underlying assumptions of each criterion to the desired properties of the model, and justify the choice of performance metric in relation
-   [ ] Explain how you will identify which model features or components are significant or meaningful.
:::

## Model assumptions and uncertainties

::: callout-caution
## Preregistration Item

- [ ] Specify assumptions and key uncertainties in the formal model. Describe what gaps exist between the model conception, and the real-world problem, what biases might this introduce and how might this impact any interpretation of the model outputs, and what implications are there for evaluating model-output to inform inferences or decisions?
:::

## Specify formal model(s)

::: callout-note
## Explanation

Once critical decisions have been made about the modelling approach and method of model specification, the conceptual model is translated into the quantitative model.
:::

::: callout-caution
## Preregistration Item

-   [ ] Specify all formal models
    - [ ]  Note, For data-driven approaches to determining model structure and or parameterisation, it may not be possible to respond to this preregistration item. In such cases, explain why this is the case, and how you will document the model(s) used in the final analysis.
-   [ ] For quantitative model selection approaches, including ensemble modelling, specify each model used in the candidate set, including any null or full/global model.
:::

# Model Calibration, Validation & Checking

## Model calibration and validation scheme

::: callout-note
## Explanation

This section pertains to any data calibration, validation or testing schemes that will be implemented. For example, the model may be tested on data independent of those used to parameterise the model (external validation), or the model may be cross-validated on random sub-samples of the data used to parameterise the model (internal cross-validation) [@Yates2018; @Barnard2019]. For some types of models, hyper-parameters are estimated from data, and may be tuned on further independent holdouts of the training data, ("validation data").
:::

::: callout-caution
## Preregistration Item

-   [ ] Describe any data calibration, validation and testing scheme you will implement, including any procedures for tuning or estimating model hyper-parameters (if any).
:::

### Describe calibration/validation data

::: callout-note
## Explanation & Rationale

The following items pertain to properties of the *datasets* used for calibration (training), validation, and testing.
:::

::: callout-caution
## Preregistration Item

*If partitioning data for cross-validation or similar approach (delete as needed):*

-   [ ] Describe the approach specifying the number of folds that will be created, the relative size of each fold, and any stratification methods used for ensuring evenness of groups between folds and between calibration / validation data?

*If using external / independent holdout data for model testing and evaluation (delete as needed):*

-   [ ] Which data will be used as a the testing data? What method will you be used for generating training / test data subsets?
-   [ ] Describe any known differences between the training/validation and testing datasets, the relative size of each, as well as any stratification methods used for ensuring evenness of groups between data sets?
-   [ ] It is preferable that any independent data used for model testing remains unknown to modellers during the process of model development, please describe the relationship modellers have to model validation data, will independent datasets be known or accessible to any modeller or analyst?
:::

## Implementation verification

::: callout-note
## Explanation & Examples

Model implementation verification is the process of ensuring that the model has been correctly implemented, and that the model performs as described by the model description [@Grimm:2014es]. This process is distinct from model checking, which assesses the model's performance in representing the system of interest [@Conn:2018hd].

-   Checks for verification implementation should include i) thoroughly checking for bugs or programming errors, and ii) whether the implemented model performs as described by the model description [@Grimm:2014es].
-   Qualitative tests could include syntax checking of code, and peer-code review [@ivimey2023]. Technical measures include using unit tests, or in-built checks within functions to prevent potential errors.
:::


::: callout-caution
## Preregistration Item

-   [ ] What Quality Assurance measures will you take to verify the model has been correctly implemented? Specifying a priori quality assurance tests for implementation verification may help to avoid selective debugging and silent errors.
:::

## Model checking

::: callout-note
## Rationale & Explanation

"Model Checking" goes by many names ("conditional verification", "quantitative verification", "model output verification" ), and refers to a series of analyses that assess a model's performance in representing the system of interest [@Conn:2018hd]. Model checking aids in diagnosing assumption violations, and reveals where a model might need to be altered to better represent the data, and therefore system [@Conn:2018hd]. Quantitative model checking diagnostics include goodness of fit, tests on residuals or errors, such as for heteroscedascity, cross-correlation, and autocorrelation [@Jakeman:2006ii].
:::

### Quantitative model checking

::: callout-caution
## Preregistration Item

*During this process, observed data, or data and patterns that guided model design and calibration, are compared to model output in order to identify if and where there are any systematic differences.*

-   [ ] Specify any diagnostics or tests you will use during model checking to assess a model's performance in representing the system of interest.
-   [ ] For each test, specify the criteria that will you use to interpret the outcome of the test in assessing the model's ability to sufficiently represent the gathered data used to develop and parameterise the model.
:::

### Qualitative model checking

::: callout-note
## Explanation

This step is largely informal and case-specific, but requires‚ 'face validation' with model users / clients / managers who aren't involved in the development of the model to assess whether the interactions and outcomes of the model are feasible an defensible [@Grimm:2014es]. This process is sometimes called a ‚"laugh test" or a "pub test" and in addition to checking the model's believability, it builds the client's confidence in the model [@Jakeman:2006ii]. Face validation could include structured walk-throughs, or presenting descriptions, visualisations or summaries of model results to experts for assessment.
:::

::: callout-caution
## Preregistration Item

-   [ ] Briefly explain how you will qualitatively check the model, and whether and how you will include users and clients in the process.
:::

### Assumption Violation Checks

::: callout-caution
## Preregistration Item

*The consequences of assumption violations on the interpretation of results should be assessed [@Araujo2019].*

- [ ] Explain how you will demonstrate robustness to model assumptions and check for violations of model assumptions.
- [ ] If you cannot perform quantitative assumption checks, describe what theoretical justifications would justify a lack of violation of or robustness to model assumptions.
- [ ] If you cannot demonstrate or theoretically justify violation or robustness to assumptions, explain why not, and specify whether you will discuss assumption violations and their consequences for interpretation of model outputs.
- [ ] If assumption violations cannot be avoided, explain how you will explore the consequences of assumption violations on the interpretation of results (To be completed in interim iterations of the preregistration, only if there are departures from assumptions as demonstrated in the planned tests above).
:::

# Model Validation and Evaluation

::: callout-note
## Explanation

The model validation & evaluation phase comprises a suite of analyses that collectively inform inferences about whether, and under what conditions, a model is suitable to meet its intended purpose [@Augusiak:2014gz]. Errors in design and implementation of the model and their implication on the model output are assessed. Ideally independent data is used against the model outputs to assess whether the model output behaviour exhibits the required accuracy for the model's intended purpose. The outcomes of these analyses build confidence in the model applications and increase understanding of model strengths and limitations. Model evaluation including, model analysis, should complement model checking. It should evaluate model checking, and consider over-fitting and extrapolation. The higher the proportion of calibrated, or uncertain parameters, "the greater the risk that the model seems to work correctly, but for the wrong reasons" (citaiton). Evaluation thus complements model checking because we can rule out the chance that the model fits the calibration data well, but has not captured the relevant ecological mechanisms of the system pertinent to the research question or the decision problem underpinning the model [@Grimm:2014es]. Evaluation of model outputs against external data in conjunction with the results from model checking provide information about the structural realism and therefore credibility of the model [@Grimm2016].

<!-- #NOTE, explain how model-checking is different to model validation / evaluation. What am I defining as model analysis / application?? the phases of the modelling process underlying this template need exposition somewhere -->
:::

## Model output corroboration

::: callout-note
## Explanation

Ideally, model outputs or predictions are compared to independent data and patterns that were not used to develop, parameterise, or verify the model. Testing against a dataset of response and predictor variables that are spatially and/or temporally independent from the training dataset minimises the risk of artificially inflating model performance measures [@Araujo2019]. Although the corroboration of model outputs against an independent validation dataset is considered the 'gold standard' for showing that a model properly represents the internal organisation of the system, model validation is not always possible because empirical experiments are infeasible or model users are working on rapid-response time-frames, hence, why ecologists often model in the first place [@Grimm:2014es]. Independent predictions might instead be tested on sub-models. Alternatively, patterns in model output that are robust and seem characteristic of the system can be identified and evaluated in consultation with the literature or by experts to judge how accurate the model output is [@Grimm:2014es].
:::

::: callout-caution
## Preregistration Item

-   [ ] State whether you will corroborate the model outputs on external data, and document any independent validation data in step.
-   [ ] It is preferable that any independent data used for model evaluation remains unknown to modellers during the process of model building [@Dwork2015], describe the relationship modellers have to model validation data, e.g. will independent datasets be known to any modeller or analyst involved in the model building process?
-   [ ] If unable to evaluate the model outputs against independent data, explain why and explain what steps you will take to interrogate the model.
:::

## Choose performance metrics and criteria

::: callout-note
## Explanation

Model performance can be quantified by a range of tests, including measures of agreement between predictions and independent observations, or estimates of accuracy, bias, calibration, discrimination refinement, resolution and skill [@Araujo2019]. Note that the performance metrics and criteria in this section are used for evaluating the structured and parameterised models (ideally) on independent holdout data, so this step is additional to any performance criteria used for determining model structure or parameterisation in section X.

<!-- #TODO add section-cross referencing -->
:::

::: callout-caution
## Preregistration Item

-   [ ] Specify what performance measures you will use to evaluate the model and briefly explain how each test relates to different desired properties of a model's performance.
-   [ ] Spatial, temporal and environmental pattern of errors and variance can change the interpretation of model predictions and conservation decisions [@Araujo2019], where relevant and possible, describe how you will characterise and report the spatial, temporal and environmental pattern of errors and variance.
-   [ ] If comparing alternative models, specify what measures of model comparison or out-of-sample performance metrics will you use to find support for alternative models or else to optimise predictive ability. State what numerical threshold or qualities you will use for each of these metrics.
:::

## Model analysis

::: callout-note
## Rationale & Explanation

Uncertainty in models arises due to incomplete system understanding (which processes to include, or which interact), from imprecise, finite and sparese data measurements, and from uncertainty in input conditions and scenarios for model simulations or runs [@Jakeman:2006ii]. Non-technical uncertainties can also be introduced throughout the modellign process, such as uncertainties arising from issues in problem-framing, indeterminicies, and modeller / client values [@Jakeman:2006ii].

The purpose of model analysis is to prevent blind trust in the model by understanding how model outputs have emerged, and to 'challenge' the model by verifying whether the model is still believable and fit for purpose if one or more parameters are changed [@Grimm:2014es].

Model analysis should increase understanding of the model behaviour by identifying which processes and process interactions explain characteristic behaviours of the model system. Model analysis typically consists of sensitivity analyses preceded by uncertainty analyses [@Saltelli2019], and a suite of other simulation or other computational experiments. The aim of such computational experiments is to increase understanding of the model behaviour by identifying which processes and process interactions explain characteristic behaviours of the model system [@Grimm:2014es]. Uncertainty analyses and sensitivity analyses augment one another to draw conclusions about model uncertainty.

Because the results from a full suite of sensitivity analysis and uncertainty analysis can be difficult to interpret due to the number and complexity of causal relations examined [@Jakeman:2006ii], it is useful for the analyst to relate the choice of analysis to the modelling context, purpose and analytical objectives defined in the problem formulation phase, in tandem with any critical uncertainties that have emerged during model development and testing prior to this point.
:::

### Uncertainty Analyses

::: callout-note
## Explanation

Uncertainty can arise from different modelling techniques, response data and predictor variables [@Araujo2019]. Uncertainty analyses characterise the uncertainty in model outputs, and identify how uncertainty in model parameters affects uncertainty in model output, but does not identify which model assumptions are driving this behaviour [@Grimm:2014es; @Saltelli2019]. Uncertainty analyses can include propagating known uncertainties through the model, or by investigating the effect of different model scenarios with different parameters and modelling technique combinations [@Araujo2019], for example. It could also include characterising the output distribution, such as through empirical construction using model output data points. It could also include extracting summary statistics like the mean, median and variance from this distribution, and perhaps constructing confidence intervals on the mean [@Saltelli2019].
:::

::: callout-caution
## Preregistration Item

- [ ] Please describe how you will characterise model and data uncertainties, e.g. propagating known uncertainties through the model, investigating the effect of different model scenarios with different parameters and modelling technique combinations [@Araujo2019], or empirically constructing model distributions from model output data points, and extracting summary statistics, including the mean, median, variance, and constructing confidence intervals [@Saltelli2019].
- [ ] Relate your choice of analysis to the context and purposes of the model described in the problem formulation phase. For instance ‚ discrepancies between model output and observed output may be important for forecasting models, where cost, benefit, an risk over a substantial period must be gauged, but much less critical for decision-making or management models where the user may be satisfied with knowing that the predicted ranking order of impacts of alternative scenarios or management options is likely to be correct, with only a rough indication of their sizes" [@Jakeman:2006ii].
- [ ] Briefly describe how you will summarise the results of these in silico experiments with graphical, tabular, or other devices, such as summary statistics.
- [ ] If the chosen modelling approach is able to explicitly articulate uncertainty due to data, measurements or baseline conditions, such as by providing estimates of uncertainty (typically in the form of probabilistic parameter covariance, [@Jakeman:2006ii]), specify which measure of uncertainty you will use.
:::

### Sensitivity analyses

::: callout-note
## Explanation

*Sensitivity analysis examines how uncertainty in model outputs can be apportioned to different sources of uncertainty in model input [@Saltelli2019].*
:::

::: callout-caution
## Preregistration Item

-   [ ] Describe the sensitivity analysis approach you will take: deterministic sensitivity, stochastic sensitivity (variability in the model), or scenario sensitivity (effect of changes based on scenarios).
-   [ ] Describe any sensitivity analyses you will conduct by specifying which parameters will be held constant, which will be varied, and the range and intervals of values over which those parameters will be varied.
-   [ ] State the primary objective of each sensitivity analysis, for example, to identify which input variables contribute the most to model uncertainty so that these variables can be targeted for further data collection, or alternatively to identify which variables or factors contribute little to overall model outputs, and so can be 'dropped' from future iterations of the model [@Saltelli2019].
:::

### Model application or scenario analysis

::: callout-caution
## Preregistration Item

-   [ ] Specify any input conditions and relevant parameter values for initial environmental conditions and decision-variables under each scenario specified in section 1.
-   [ ] Describe any other relevant technical details of model application, such as methods for how you will implement any simulations or model projections.
-   [ ] What raw and transformed model outputs will you extract from the model simulations or projections, and how will you map, plot, or otherwise display and synthesise the results of scenario and model analyses.
-   [ ] Explain how you will analyse the outputs to answer your analytical objectives. For instance, describe any trade-off or robustness analyses you will undertake to help evaluate and choose between different alternatives in consultation with experts or decision-makers.
:::

### Other simulation experiments / robustness analyses

::: callout-caution
## Preregistration Item

-   [ ] Describe any other simulation experiments, robustness analyses or other analyses you will perform on the model, including any metrics and their criteria / thresholds for interpreting the results of the analysis.
:::

# References {.unnumbered}