Introduce SPORES in v0.7.0 as a generalisable mode

CHANGELOG.md

@@ -2,6 +2,9 @@
 
 ### User-facing changes
 
+|changed| Single data entries defined in YAML indexed parameters will not be automatically broadcast along indexed dimensions.
+To achieve the same functionality as in `<v0.7.dev4`, the user must set the new `init` configuration option `broadcast_param_data` to True (#615).
+
 |changed| Helper functions are now documented on their own page within the "Defining your own math" section of the documentation (#698).
 
 |new| `where(array, condition)` math helper function to apply a where array _inside_ an expression, to enable extending component dimensions on-the-fly, and applying filtering to different components within the expression (#604, #679).

docs/creating/config.md

@@ -41,6 +41,10 @@ To test your model pipeline, `config.init.time_subset` is a good way to limit yo
     Various capabilities are available to adjust the temporal resolution of a model on-the-fly, both by resampling or using externally-provided clustering.
     See our [time adjustment page](../advanced/time.md) for more details.
 
+!!! info "See also"
+    The full set of available configuration options is documented in the [configuration schema][model-configuration-schema].
+    This provides you with a description of each configuration option and the default which will be used if you do not provide a value.
+
 ## Deep-dive into some key configuration options
 
 ### `config.build.backend`

docs/creating/parameters.md

@@ -53,3 +53,29 @@ Which will add the new dimension `my_new_dim` to your model: `model.inputs.my_ne
 
 !!! warning
     The `parameter` section should not be used for large datasets (e.g., indexing over the time dimension) as it will have a high memory overhead on loading the data.
+
+## Broadcasting data along indexed dimensions
+
+If you want to set the same data for all index items, you can set the `init` [configuration option](config.md) `broadcast_param_data` to True and then use a single value in `data`:
+
+=== "Without broadcasting"
+
+    ```yaml
+    my_indexed_param:
+      data: [1, 1, 1, 1]
+      index: [my_index_val1, my_index_val2, my_index_val3, my_index_val4]
+      dims: my_new_dim
+    ```
+
+=== "With broadcasting"
+
+    ```yaml
+    my_indexed_param:
+      data: 1  # All index items will take on this value
+      index: [my_index_val1, my_index_val2, my_index_val3, my_index_val4]
+      dims: my_new_dim
+    ```
+
+!!! warning
+    The danger of broadcasting is that you maybe update `index` as a scenario override without realising that the data will be broadcast over this new index.
+    E.g., if you start with `!#yaml {data: 1, index: monetary, dims: costs}` and update it with `!#yaml {index: [monetary, emissions]}` then the `data` value of `1` will be set for both `monetary` and `emissions` index values.

docs/examples/national_scale/index.md

@@ -1,3 +1,10 @@
+---
+costs:
+    file: "src/calliope/example_models/national_scale/data_tables/costs.csv"
+    header: [0, 1]
+    index_col: 0
+---
+
 # National Scale Example Model
 
 This example consists of two possible power supply technologies,
@@ -25,6 +32,29 @@ It does not contain much data, but the scaffolding with which to construct and r
 
 ## Model definition
 
+### Referencing tabular data
+
+As of Calliope v0.7.0 it is possible to load tabular data completely separately from the YAML model definition.
+To do this we reference data tables under the `data_tables` key:
+
+```yaml
+--8<-- "src/calliope/example_models/national_scale/model.yaml:data-tables"
+```
+
+In the Calliope national scale example model, we load both timeseries and cost data from file.
+As an example, the data in the cost CSV file looks like this:
+
+{{ read_csv(page.meta.costs.file, header=page.meta.costs.header, index_col=page.meta.costs.index_col) }}
+
+You'll notice that in each row there is reference to a technology, and in each column to a cost parameter and a comment on the units being used.
+Therefore, we reference `techs` in our data table _rows_, and `parameters` and `comment` in our data table _columns_.
+The `comment` information is metadata that we don't need in our Calliope model object, so we _drop_ it on loading the table.
+Since all the data refers to the one cost class `monetary`, we don't add that information in the CSV file, but instead add it on as a dimension when loading the file.
+Where there is no data for that combination of technology and cost parameter, the value is Not-a-Number (NaN) and this combination will be ignored on loading the table.
+
+!!! info
+    You can read more about loading data from file in [our dedicated tutorial][loading-tabular-data].
+
 ### Indexed parameters
 
 Before we dive into the technologies and nodes in the model, we have defined some parameters that are independent of both of these:
@@ -110,23 +140,6 @@ The costs are more numerous as well, and include monetary costs for all relevant
 * carrier conversion capacity
 * variable operational and maintenance costs
 
-### Interlude: inheriting from templates
-
-You will notice that the above technologies _inherit_ `cost_dim_setter`.
-Templates allow us to avoid excessive repetition in our model definition.
-In this case, `cost_dim_setter` defines the dimension and index of costs, allowing us to keep our definition of technology costs to only defining `data`.
-By defining `data`, the technologies override the `null` setting applied by `cost_dim_setter`.
-We also use it to set the `interest_rate` for all technologies, which will be used to annualise any investment costs each technology defines.
-
-Technologies and nodes can inherit from anything defined in `templates`.
-items in `templates` can also inherit from each other, so you can create inheritance chains.
-
-`cost_dim_setter` looks like this:
-
-```yaml
---8<-- "src/calliope/example_models/national_scale/model_config/techs.yaml:cost-dim-setter"
-```
-
 ### Storage technologies
 
 The second location allows a limited amount of battery storage to be deployed to better balance the system.
@@ -184,8 +197,16 @@ Transmission technologies look different to other technologies, as they link the
 `free_transmission` allows local power transmission from any of the csp facilities to the nearest location.
 As the name suggests, it applies no cost or efficiency losses to this transmission.
 
+### Interlude: inheriting from templates
+
 We can see that those technologies which rely on `free_transmission` inherit a lot of this information from elsewhere in the model definition.
 `free_transmission` is defined in `templates`, which makes it inheritable.
+[Templates](../../creating/templates.md) allow us to avoid excessive repetition in our model definition.
+
+Technologies and nodes can inherit from anything defined in `templates`.
+items in `templates` can also inherit from each other, so you can create inheritance chains.
+
+The `free_transmission` template looks like this:
 
 ```yaml
 --8<-- "src/calliope/example_models/national_scale/model_config/techs.yaml:free-transmission"

docs/examples/national_scale/spores_run.py

@@ -0,0 +1,136 @@
+# ---
+# jupyter:
+#   jupytext:
+#     text_representation:
+#       extension: .py
+#       format_name: percent
+#       format_version: '1.3'
+#       jupytext_version: 1.16.4
+#   kernelspec:
+#     display_name: Python 3 (ipykernel)
+#     language: python
+#     name: python3
+# ---
+
+# %% [markdown]
+# # Generating SPORES
+# An interactive example of how to generate near-optimal system designs (or SPORES) out of a Calliope v0.7.0 model. This example relies solely on default software functionality and a custom Python function to determine how to assign penalties (scores) to previously explored system design options.
+
+# %%
+# Importing the required packages
+import calliope
+import xarray as xr
+
+# %% [markdown]
+# ## Cost-optimal model run and extraction of SPORES-relevant outputs
+
+# %%
+# Loading model files and building the model
+model = calliope.examples.national_scale(scenario="spores")
+model.build()
+
+# Solving
+model.solve()
+
+# Extracting SPORES-relevant data
+least_feasible_cost = model.results.cost.loc[{"costs": "monetary"}].sum().sum()
+print("The minimum cost for a feasible system design is {}".format(least_feasible_cost.values))
+
+
+# %% [markdown]
+# ## SPORES model run
+# ### Definition of the penalty-assignment methods
+
+# %%
+def scoring_integer(results, backend):
+    # Filter for technologies of interest
+    spores_techs = backend.inputs["spores_tracker"].notnull()
+    # Look at capacity deployment in the previous iteration
+    previous_cap = results.flow_cap 
+    # Make sure that penalties are applied only to non-negligible deployments of capacity
+    min_relevant_size = 0.1 * previous_cap.where(spores_techs).max(
+        ["nodes", "carriers", "techs"]
+    )
+    # Where capacity was deployed more than the minimal relevant size, assign an integer penalty (score)
+    new_score = previous_cap.copy()
+    new_score = new_score.where(spores_techs, other=0)
+    new_score = new_score.where(new_score > min_relevant_size, other=0)
+    new_score = new_score.where(new_score == 0, other=1000)
+    # Transform the score into a "cost" parameter
+    new_score.rename("cost_flow_cap")
+    new_score = new_score.expand_dims(costs=["spores_score"]).copy()
+    new_score = new_score.sum("carriers")
+    # Extract the existing cost parameters from the backend
+    all_costs = backend.get_parameter("cost_flow_cap", as_backend_objs=False)
+    try:
+        all_costs = all_costs.expand_dims(nodes=results.nodes).copy()
+    except:
+        pass
+    # Create a new version of the cost parameters by adding up the calculated scores
+    new_all_costs = all_costs
+    new_all_costs.loc[{"costs":"spores_score"}] += new_score.loc[{"costs":"spores_score"}]
+
+    return new_all_costs
+
+
+# %% [markdown]
+# ### Iterating over the desired number of alternatives
+
+# %%
+# Create some lists to store results as they get generated
+spores = [] # full results
+scores = [] # scores only
+spores_counter = 1
+number_of_spores = 5
+
+# %%
+for i in range(spores_counter, spores_counter + number_of_spores):
+
+    if spores_counter == 1:
+        # Store the cost-optimal results
+        spores.append(model.results.expand_dims(spores=[0]))
+        scores.append(
+            model.backend.get_parameter("cost_flow_cap", as_backend_objs=False)
+            .sel(costs="spores_score")
+            .expand_dims(spores=[0])
+        )
+        # Update the slack-cost backend parameter based on the calculated minimum feasible system design cost
+        model.backend.update_parameter("spores_cost_max", least_feasible_cost)
+        # Update the objective_cost_weights to reflect the ones defined for the SPORES mode
+        model.backend.update_parameter(
+            "objective_cost_weights", model.inputs.spores_objective_cost_weights
+        )
+    else:
+        pass
+
+    # Calculate weights based on a scoring method
+    spores_score = scoring_integer(model.results, model.backend)
+    # Assign a new score based on the calculated penalties
+    model.backend.update_parameter(
+        "cost_flow_cap", spores_score.reindex_like(model.inputs.cost_flow_cap)
+    )
+    # Run the model again to get a solution that reflects the new penalties
+    model.solve(force=True)
+    # Store the results
+    spores.append(model.results.expand_dims(spores=[i]))
+    scores.append(
+        model.backend.get_parameter("cost_flow_cap", as_backend_objs=False)
+        .sel(costs="spores_score")
+        .expand_dims(spores=[i])
+    )
+
+    spores_counter += 1
+
+# Concatenate the results in the storage lists into xarray objects 
+spore_ds = xr.concat(spores, dim="spores")
+score_da = xr.concat(scores, dim="spores")
+
+# %% [markdown]
+# ## Sense-check
+
+# %%
+# Extract the deployed capacities across SPORES, which we want to inspect
+flow_caps = spore_ds.flow_cap.where(
+    model.backend.inputs["spores_tracker"].notnull()).sel(
+        carriers='power').to_series().dropna().unstack("spores")
+flow_caps

docs/examples/urban_scale/index.md

@@ -47,6 +47,8 @@ The import section in our file looks like this:
 --8<-- "src/calliope/example_models/urban_scale/model.yaml:import"
 ```
 
+## Model definition
+
 ### Referencing tabular data
 
 As of Calliope v0.7.0 it is possible to load tabular data completely separately from the YAML model definition.
@@ -56,7 +58,7 @@ To do this we reference data tables under the `data_tables` key:
 --8<-- "src/calliope/example_models/urban_scale/model.yaml:data-tables"
 ```
 
-In the Calliope example models, we only load timeseries data from file, including for [energy demand](#demand-technologies), [electricity export price](#revenue-by-export) and [solar PV resource availability](#supply-technologies).
+In the Calliope urban scale example model, we only load timeseries data from file, including for [energy demand](#demand-technologies), [electricity export price](#revenue-by-export) and [solar PV resource availability](#supply-technologies).
 These are large tables of data that do not work well in YAML files!
 As an example, the data in the energy demand CSV file looks like this:
 
@@ -69,8 +71,6 @@ Since all the data refers to the one parameter `sink_use_equals`, we don't add t
 !!! info
     You can read more about loading data from file in [our dedicated tutorial][loading-tabular-data].
 
-## Model definition
-
 ### Indexed parameters
 
 Before we dive into the technologies and nodes in the model, we have defined some parameters that are independent of both of these:
@@ -127,21 +127,6 @@ The definition of this technology in the example model's configuration looks as
 --8<-- "src/calliope/example_models/urban_scale/model_config/techs.yaml:pv"
 ```
 
-### Interlude: inheriting from templates
-
-You will notice that the above technologies _inherit_ `interest_rate_setter`.
-Templates allow us to avoid excessive repetition in our model definition.
-In this case, `interest_rate_setter` defines an interest rate that will be used to annualise any investment costs the technology defines.
-
-Technologies / nodes can inherit from anything defined in `templates`.
-items in `templates` can also inherit from each other, so you can create inheritance chains.
-
-`interest_rate_setter` looks like this:
-
-```yaml
---8<-- "src/calliope/example_models/urban_scale/model_config/techs.yaml:interest-rate-setter"
-```
-
 ### Conversion technologies
 
 The example model defines two conversion technologies.
@@ -241,7 +226,7 @@ Gas is made available in each node without consideration of transmission.
 --8<-- "src/calliope/example_models/urban_scale/model_config/techs.yaml:transmission"
 ```
 
-To avoid excessive duplication in model definition, our transmission technologies inherit most of the their parameters from _templates_:
+To avoid excessive duplication in model definition, our transmission technologies inherit most of the their parameters from [templates](../../creating/templates.md):
 
 ```yaml
 --8<-- "src/calliope/example_models/urban_scale/model_config/techs.yaml:transmission-templates"

docs/user_defined_math/syntax.md

@@ -210,7 +210,7 @@ If you define a lookup parameter "lookup_techs" as:
 ```yaml
 parameters:
   lookup_techs:
-    data: True
+    data: [True, True]
     index: [tech_1, tech_2]
     dims: [techs]
 ```

src/calliope/config/config_schema.yaml

@@ -58,6 +58,13 @@ properties:
               Unit of transmission link `distance` (m - metres, km - kilometres).
               Automatically derived distances from lat/lon coordinates will be given in this unit.
             enum: [m, km]
+          broadcast_param_data:
+            type: boolean
+            default: false
+            description:
+              If True, single data entries in YAML indexed parameters will be broadcast across all index items.
+              If False, the number of data entries in an indexed parameter needs to match the number of index items.
+              Defaults to False to mitigate unexpected broadcasting when applying overrides.
 
       build:
         type: object

src/calliope/example_models/national_scale/additional_math.yaml

@@ -0,0 +1,7 @@
+constraints:
+  total_system_cost_max:
+    description: >
+      Limit total system cost. Conceived for use in SPORES mode to apply a maximum
+      relaxation to the system cost compared to the least-cost feasible option.
+    equations:
+      - expression: sum(cost[costs=monetary], over=[nodes, techs]) <= spores_cost_max * (1 + spores_slack)

src/calliope/example_models/national_scale/data_tables/costs.csv

@@ -0,0 +1,6 @@
+parameters,cost_flow_cap,cost_storage_cap,cost_area_use,cost_source_cap,cost_flow_in,cost_flow_out
+comment,USD per kW,USD per kWh storage capacity,USD per m2,USD per kW,USD per kWh,USD per kWh
+ccgt,750,,,,0.02,
+csp,1000,50,200,200,,0.002
+battery,,200,,,,
+region1_to_region2,200,,,,,0.002