Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds documentation for use cases and configurable #18

Merged
merged 1 commit into from
Apr 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,10 @@ wintermute.py: error: the following arguments are required: {linux_privesc,windo
$ python wintermute.py linux_privesc --enable_explanation true --enable_update_state true
~~~

## Contribution

If you want to contribute additional use-cases, please take a look at [docs/use_case.md](docs/use_case.md) for a start on how to extend this codebase.

# Disclaimers

Please note and accept all of them.
Expand Down
88 changes: 88 additions & 0 deletions docs/configurable.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Configurable

Marking a class as `@configurable` allows for the class to be configured via command line arguments or environment variables.

This is done by analyzing the parameters to the class' `__init__` method and its `__dataclass_fields__` attribute if it is a `@dataclass`.
As having a `@configurable` also be a `@dataclass` makes it easier to extend it, it is usually recommended to define a configurable as a `@dataclass`.
Furthermore, using a dataclass allows more natural use of the `parameter()` definition.

All [use-cases](use_case.md) are automatically configurable.

## Parameter Definition

Parameters can either be defined using type hints and default values, or by using the `parameter()` method.

```python
from dataclasses import dataclass
from utils.configurable import configurable, parameter


@configurable("inner-example", "Inner Example Configurable for documentation")
@dataclass
class InnerConfigurableExample:
text_value: str


@configurable("example", "Example Configurable for documentation")
@dataclass
class ConfigurableExample:
inner_configurable: InnerConfigurableExample
text_value: str
number_value_with_description: int = parameter(desc="This is a number value", default=42)
number_value_without_description: int = 43
```

As can be seen, the `parameter` method allows additionally setting a description for the parameter, while returning a `dataclasses.Field` to allow interoperability with existing tools.

The type of a configurable parameter may only be a primitive type (`int`, `str`, `bool`) or another configurable.

## Usage

When a class is marked as `@configurable`, it can be configured via command line arguments or environment variables.
The name of the parameter is automatically built from the field name (in the case of the example to be `text_value`, `number_with_description` and `number_value_without_description`).

If a configurable has other configurable fields as parameters, they can be recursively configured, the name of the parameter is built from the field name and the field name of the inner configurable (here `inner_configurable.text_value`).

These parameters are looked up in the following order:

1. Command line arguments
2. Environment variables (with `.` being replaced with `_`)
3. .env file
4. Default values

When you have a simple use case as follows:

```python
from dataclasses import dataclass
from usecases import use_case, UseCase

@use_case("example", "Example Use Case")
@dataclass
class ExampleUseCase(UseCase):
conf: ConfigurableExample

def run(self):
print(self.conf)
```

You can configure the `ConfigurableExample` class as follows:

```bash
echo "conf.text_value = 'Hello World'" > .env
export CONF_NUMBER_VALUE_WITH_DESCRIPTION=120
export CONF_INNER_CONFIGURABLE_TEXT_VALUE="Inner Hello World"

python3 wintermute.py example --conf.inner_configurable.text_value "Inner Hello World Overwrite"
```

This results in

```
ConfigurableExample(
inner_configurable=InnerConfigurableExample(text_value='Inner Hello World Overwrite'),
text_value='Hello World',
number_value_with_description=120,
number_value_without_description=43
)
```

28 changes: 28 additions & 0 deletions docs/use_case.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Use Cases

Wintermute consists of different use-cases (classes extending `UseCase`, being annotated with `@use_case` and being imported somewhere from the main `wintermute.py` file), which can be run individually.

The `@use_case` annotation takes a name and description as arguments, which are then used for the sub-commands in the command line interface.

When building a use-case, the `run` method must be implemented, which is called after calling the (optional) `init` method (note that this is not the `__init__` method).
The `run` method should contain the main logic of the use-case, though it is recommended to split the logic into smaller methods that are called from `run` for better readability (see the code for [`RoundBasedUseCase`](#round-based-use-case) for an example).

A use-case is automatically a `configurable`, which means, that all parameters of its `__init__` function (or fields for dataclasses) can be set via command line / environment parameters. For more information read the [configurable](configurable.md) documentation.
It is recommended to define a use case to be a `@dataclass`, so that all parameters are directly visible, and the use-case can be easily extended.

## General Use Cases

Usually a use case follows the pattern, that it has a connections to the log database, a LLM and a system with which it is interacting.

The LLM should be defined as closely as necessary for the use case, as prompt templates are dependent on the LLM in use.
If you don't yet want to specify eg. `GPT4Turbo`, you can use `llm: OpenAIConnection`, and dynamically specify the LLM to be used in the parameters `llm.model` and `llm.context_size`.

In addition to that, arbitrary parameters and flags can be defined, with which to control the use-case. For consistency reasons please take a look if similar parameters are used in other use cases, and try to have them act accordingly.

When interacting with a LLM, the prompt and output should always be logged `add_log_query`, `add_log_analyze_response`, `add_log_update_state` or alike, to record all interactions.

## Round Based Use Case

The `RoundBasedUseCase` is an abstract base class for use-cases that are based on rounds where the LLM is called with a certain input and the result is evaluated using different capabilities.

An implementation needs to implement the `perform_round` method, which is called for each round. It can also optionally implement the `setup` and `teardown` methods, which are called before and after the rounds, respectively.