Merge pull request #19 from sstroemer/improve-docs-release2

Improve docs related to release of `v2.0.0`

docs/pages/dev/updating.md

@@ -8,11 +8,61 @@ The `2.0.0` release follows the breaking change of `IESopt.jl` going to `v2.0.0`
 
 ### Changes to the top-level config
 
-To be written: everything.
+The structure of the `config` section in the top-level configuration file changed. Overall it's highly similar, but:
+
+- A `general` section was added that properly groups "general" settings, that were previous keyword arguments.
+- Small adjustments were made to account for that.
+- The `results` section changed slightly.
+
+See the respective section(s) in the [YAML/Top-level config](../manual/yaml/top_level.md) documentation. The most important changes relate to the following (sub-)sections:
+
+- `name`
+- `version`
+- `high_performance`
+- `constraint_safety` is now called `soft_constraints` and part of the `optimization` section.
 
 ### Changes to keyword arguments
 
-To be written: "parameters" and "config".
+Previously we allowed passing arbitrary keyword arguments to public functions, like {py:func}`iesopt.Model.generate` or {py:func}`iesopt.run`. This can lead to problems with latency on previously unseen types of arguments, especially now that we support arbitrary dictionaries as model parameters. This now works differently:
+
+- There are no arbitrary keyword arguments anymore.
+- Model parameters have to be passed as dictionary, using `parameters = {foo: 1.0, bar: "austria"}`.
+- There is a new and "powerful" `config` keyword argument, explained below.
+
+#### `config`
+
+Besides model parameters, specific keyword arguments existed, e.g., `verbosity`. These are gone and now first-class options in the model's top-level config (see [YAML/Top-level config](../manual/yaml/top_level.md)). With that a way to change them programmatically was in need, which is what the `config` argument does. Take the following example:
+
+```{code-block} python
+:caption: Example usage of `config` keyword argument.
+
+import iesopt
+
+iesopt.run("my_config.iesopt.yaml", config = {
+    "general.verbosity.core": "error",
+    "optimization.snapshots.count": 168,
+    "files.data_in": "scenario17/data.csv",
+})
+```
+
+The passed dictionary modifies the top-level configuration during the parsing step. That means you are able to overwrite / set each configuration option programmatically. The access pattern follows the structure of the `config` section in the top-level configuration file. For example, the verbosity in the YAML looks like:
+
+```{code-block} yaml
+:caption: Verbosity example in the top-level YAML.
+
+config:
+  general:
+    verbosity:
+      core: info
+```
+
+To modify / set this, `general.verbosity.core` is used. As you can see that opens the possibility to modify, e.g., files that are loaded based on (for example) which scenario we are in by modifying the `files` section, using `files.data_in`. Here `data_in` is the "name" used in the model later on (using `some_col@data_in`).
+
+Further, this means you no longer need to use "model parameters" for stuff that is actually part of the `config` section, e.g., the number of Snapshots in your model. Instead of coming up with a parameter, e.g., `my_snapshot_count`, using it as `count: <my_snapshot_count>`, and then passing it as `iesopt.run("...", my_snapshot_count = 168)`, you can now just modify that as shown above by setting it using the accessor `optimization.snapshots.count`.
+
+#### `parameters`
+
+Any dictionary passed to this will be used in the same way as keyword arguments were used before: Parameters given there are replaced in the `parameters` section (which might be based on an external `*.iesopt.param.yaml` file).
 
 ### Changes to result extraction
 
@@ -30,4 +80,47 @@ Everything that was previously part of `IESoptLib.jl` is now integrated into `IE
 
 #### Tags
 
-To be written.
+Component "tags" are back (or accessible again, they were never gone completely). That means you can tag components to later simplify result handling or working with your model. A tag can be added when creating a component:
+
+```{code-block} yaml
+:caption: Adding a tag.
+
+components:
+  my_unit:
+    type: Unit
+    tags: CustomUnit
+```
+
+The attribute `tags` supports single tags (e.g., `CustomUnit`) or lists of tags (e.g., `[CustomUnit]` or `[CustomUnit, FindMe]`).
+
+:::{tip}
+Each core component automatically tags its own type, so each `Unit` will already be tagged with the tag `"Unit"`.
+
+Most importantly, that means you are also able to extract all `Virtual`s (non-existing components related to templates that you use), since each of these also tags its "actual type".
+:::
+
+Using this is possible with {py:func}`iesopt.Model.get_components`:
+
+```{code-block} python
+:caption: Extracting tagged components.
+
+import iesopt
+
+model = iesopt.run("tagged_model.iesopt.config")
+
+my_units = model.get_components("CustomUnit")
+```
+
+When passing a `list` of tags, only components having ALL of these tags are returned.
+
+:::{tip}
+Most importantly, that (and the fact that components "auto-tag" their type) means you are also able to extract all `Virtual`s (non-existing components related to templates that you use), since each of these also tags its "actual type". Consider a template `Battery` that you use to initialize a battery
+
+```yaml
+components:
+  my_bat:
+    type: Battery
+```
+
+All batteries can then be found using `model.get_components("Battery")`. This can also be really helpful in addons, where (using Julia) you can find, e.g., all CHPs in your model by doing `chps = IESopt.get_components(model, "CHP")`.
+:::

docs/pages/manual/yaml/top_level.md

@@ -82,12 +82,13 @@ General configuration settings for the optimization model.
 :caption: Example for the `config` section in the top-level YAML configuration file.
 
 config:
-  version:
-    core: 1.1.0
-    python: 1.4.7
-  name:
-    model: FarayOptIndustry
-    scenario: Base_2022_LOW
+  general:
+    version:
+      core: 1.1.0
+      python: 1.4.7
+    name:
+      model: FarayOptIndustry
+      scenario: Base_2022_LOW
   optimization:
     problem_type: LP
     snapshots:
@@ -105,17 +106,22 @@ config:
 
 The following subsections explain each part of the `config` section in more detail.
 
-### `version`
+### `general`
+
+This contains general settings.
+
+#### `version`
 
 This section allows specifying various IESopt versions, which is important to ensure reproducibility and compatibility: Executing a model based on a `config.iesopt.yaml` file with a different version of IESopt might lead to **unexpected results or errors**.
 
 ```{code-block} yaml
 :caption: Example for the `version` section.
 
 config:
-  version:
-    core: 1.1.0
-    python: 1.4.7
+  general:
+    version:
+      core: 1.1.0
+      python: 1.4.7
 ```
 
 We try to stick to [semantic versioning](https://semver.org/), which means that changes to the minor or patch version should not contain breaking changes, but:
@@ -133,7 +139,7 @@ Therefore: Make sure you **KNOW** what changes between versions, and **TEST** yo
 You may add arbitrary versions to this section, e.g., for personal use in specific addons or other dependencies.
 :::
 
-### `name`
+#### `name`
 
 This section does not directly affect the model but can be used to store the name of the model and the scenario. This can be useful for logging and debugging purposes, as well as for the results output.
 
@@ -155,9 +161,10 @@ Consider the following example for the `name` section:
 :caption: Example for the `name` section.
 
 config:
-  name:
-    model: FarayOptIndustry
-    scenario: Base_2022_LOW_T-$TIME$
+  general:
+    name:
+      model: FarayOptIndustry
+      scenario: Base_2022_LOW_T-$TIME$
 ```
 
 When running the model containing this configuration, the results will be stored in a folder structure that looks like this:
@@ -181,6 +188,16 @@ When running the model containing this configuration, the results will be stored
 
 In this example, the `$TIME$` placeholder was replaced by the current timestamp, which is `2024_09_11_09520837` in this case. You can see the top-level config file (`config.iesopt.yaml`), a data folder (`data/`; see the [files](#files) and [paths](#paths) sections), the results folder (`out/`; see the [results](#results) and [paths](#paths) sections). Inside the results folder, IESopt creates a folder for each "model name". Each executed run creates its result files inside that, using the "scenario name" as base filename.
 
+#### `verbosity`
+
+Controls the verbosity of various parts of a model run.
+
+:Parameters:
+:`core`: Verbosity of the (Julia) core, `IESopt.jl`. Supports: `debug`, `info` (default), `warning`, `error`.
+:`progress`: Whether to show progress bars (`on` or `off`), defaults to `on` unless `core` is set to `error` - in that case it defaults to `off`.
+:`python`: Verbosity of the Python wrapper, `iesopt`. Supports: `debug`, `info`, `warning`, `error`. Defaults to the verbosity set in `core`.
+:`solver`: Whether to silence solver prints/outputs (`on` or `off`), defaults to `on` unless `core` is set to `error` - in that case it defaults to `off`.
+
 ### `optimization`
 
 #### `problem_type`
@@ -255,8 +272,10 @@ This allows defining custom objective expressions for the optimization problem.
 ```{code-block} yaml
 :caption: Example for the `objectives` section.
 
-objectives:
-  emissions: [co2_emissions.exp.value]
+config:
+  optimization:
+    objectives:
+      emissions: [co2_emissions.exp.value]
 ```
 
 The above constructs a new objective expression that only consists of `co2_emissions.exp.value`. It may also be initialized empty (`[]`), in which case you can add terms later on in the model definition.
@@ -275,17 +294,37 @@ This setting is part of advanced functionality. It is not fully documented, or m
 ```{code-block} yaml
 :caption: Example for the `multiobjective` section.
 
-multiobjective:
-  mode: EpsilonConstraint
-  terms: [total_cost, emissions]
-  settings:
-    MOA.SolutionLimit: 5
+config:
+  optimization:
+    multiobjective:
+      mode: EpsilonConstraint
+      terms: [total_cost, emissions]
+      settings:
+        MOA.SolutionLimit: 5
 ```
 
 ```{note}
 Refer to the related example models for more information on how to use multi-objective optimization in IESopt.
 ```
 
+#### `soft_constraints`
+
+This was called `constraint_safety` in all versions previous to `v2.0.0`. It controls if and how certain constraints of the model are relaxed to allow penalized violation of said constraints. This can be helpful, e.g., if your model is infeasible for certain external settings and you might want to ensure getting an approximate solution.
+
+```{code-block} yaml
+:caption: Example for the `soft_constraints` section.
+
+config:
+  optimization:
+    soft_constraints:
+      active: true
+      penalty: 1e6
+```
+
+:Parameters:
+:`active` (`bool`, default = `false`): Activate the feature.
+:`penalty` (`float`): Penalty that is used to penalize constraint violations.
+
 ### `files`
 
 This can be used to define input files that are later referenced in the configuration file. Each filename (possibly including a path) is linked to a name that can be used to reference the file later on.
@@ -308,7 +347,8 @@ Settings for the results output. Details to be added.
 :`enabled` (`bool`, default = `true`): Whether to enable the automatic extraction of results. If this is set to `false`, no results will read from the solver after optimizing the model - you can however still access the solver results directly, see for example {py:func}`iesopt.jump_value`.
 :`memory_only` (`bool`, default = `true`): Whether to store the results in memory only, without writing them to disk. This can be useful if you plan to access and further process the results directly in Python or Julia, or only want to store specific results.
 :`compress` (`bool`, default = `false`): Whether to compress the results when writing them to disk. This can save disk space but might increase the time needed to write and read the results. Refer to [JLD2.jl](https://github.com/JuliaIO/JLD2.jl) for more information about compression.
-:`include` (`str`, default = `none` or `all`): A list of result extraction modes to activate, see below for more details. The default depends on the setting of `memory_only`: If `memory_only` is `true`, the default is `none`, otherwise it is `all`.
+:`include` (`str`, default = `none` or `nput+log`): A list of result extraction modes to activate, see below for more details. The default depends on the setting of `memory_only`: If `memory_only` is `true`, the default is `none`, otherwise it is `input+log`. You can use `all` to activate all possible settings.
+:`backend` (`str`, default = `jld2`): Backend to use in the result extraction. Currently `jld2` is the main/working one, while we are trying to improve the `duckdb` backend.
 
 ```{code-block} yaml
 :caption: Example for the `results` section.

src/iesopt/julia/util.py

@@ -61,11 +61,11 @@ def jl_docs(obj: str, module: str = "IESopt"):
 
 
 def recursive_convert_py2jl(item):
-    juliacall = get_iesopt_module_attr("juliacall")
-
     if isinstance(item, dict):
+        juliacall = get_iesopt_module_attr("juliacall")
         return juliacall.Main.Dict({k: recursive_convert_py2jl(v) for (k, v) in item.items()})
     if isinstance(item, list):
+        juliacall = get_iesopt_module_attr("juliacall")
         return juliacall.convert(juliacall.Main.Vector, [recursive_convert_py2jl(v) for v in item])
 
     return item

src/iesopt/model.py

@@ -182,6 +182,18 @@ def get_component(self, component: str):
         except Exception:
             raise Exception(f"Error while retrieving component `{component}` from model")
 
+    def get_components(self, tagged=None):
+        """
+        Get all components of the model, possibly filtered by (a) tag(s).
+
+        Arguments:
+            tagged (str or list of str): The tag(s) to filter the components by, can be `None` (default) to get all components.
+        """
+        if tagged is None:
+            return self._IESopt.get_components(self.core)
+
+        return self._IESopt.get_components(self.core, recursive_convert_py2jl(tagged))
+
     def get_variable(self, component: str, variable: str):
         """Get a specific variable from a core component."""
         raise DeprecationWarning(