Skip to content

Commit

Permalink
add logging dir, new generator questions to FAQ (#866)
Browse files Browse the repository at this point in the history
  • Loading branch information
leondz authored Aug 30, 2024
1 parent e58d4be commit 218dffd
Showing 1 changed file with 16 additions and 0 deletions.
16 changes: 16 additions & 0 deletions FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,9 +87,25 @@ Recommended practice: it's really context dependent. The builtin "fast" config w

We update garak by improving existing probes or adding new ones quite frequently, and so scores will go down over time - garak isn't a benchmark, and the more we learn about failures in LLMs, the harder garak gets. But if you're looking at a short period of just a month or two, then the scores will probably stay pretty much the same. We do not recommend relying on scores over six months old.

## How can I create my own generator?

Adding a custom generator is fairly straight forward. One can either add a new class in the existing module, or a new module in the `generators/` directory with a class that extends garak.generators.base.Generator that will be loaded at runtime. The reference documentation has a [full guide to creating garak generators](https://reference.garak.ai/en/latest/contributing.generator.html).

## How can I redirect `garak_runs/` and `garak.log` to another place instead of `~/.local/share/garak/`?

* `garak_runs` is configured via top-level config param `reporting.report_dir` and also CLI argument `--report_prefix` (which currently can include directory separator characters, so an absolute path can be given)
* An example of the location of the config param can be seen in https://github.com/leondz/garak/blob/main/garak/resources/garak.core.yaml
* If `reporting.report_dir` is set to an absolute path, you can move it anywhere
* If it's a relative path, it will be within the garak directory under the "data" directory following the cross-platform [XDG base directory specification](https://specifications.freedesktop.org/basedir-spec/latest/) for local storage
* There's no CLI or config option for moving `garak.log`, which is also stored in the XDG data directory
* We would welcome a PR implementing configurability of logfile path
* The Python implementation of XDG that garak uses allows overriding the data directory using the `XDG_DATA_HOME` environment variable
* An alternative is to symlink the paths to where you want them to be



<!-- ## Why the name?
Congrats, if you're reading this, you found a flag!
It's named after a smooth-talking, manipulative, persuasive, well-written character from a nineties TV series. Because we need tools like that to dissect LLM behavior. -->

0 comments on commit 218dffd

Please sign in to comment.