Skip to content

Commit

Permalink
Improvement: apply use_precomputed_responses for prompt-path requirement
Browse files Browse the repository at this point in the history
Summary:
Currently, when we enable the flag `--use-precomputed-responses`, such as:
```
python3 -m CybersecurityBenchmarks.benchmark.run \
-- --benchmark=instruct \
--use-precomputed-responses \
--llm-under-test="TOGETHER::togethercomputer/llama-2-7b::KEY" \
--response-path=security/genai/CybersecurityBenchmarks/datasets/internal/demo/instruct/instruct_llama2-7b-chat_response_precomputed.json \
--num-test-cases=1
```
We will received the error:
```
error: the following arguments are required: --prompt-path
```

This is not correct since when this flag is enabled, there is no need to use `--prompt-path`.
 ---
arg `--llm-under-test` has the similar issue. Once this one gets approved, I'll try to fix the argument with the similar design.

Reviewed By: YueLi28

Differential Revision: D52707709

fbshipit-source-id: 3d1cb89c8e1026ad072b6c1977b1a01d4b9cdcb2
  • Loading branch information
SimonWan committed Jan 16, 2024
1 parent 27e3cfa commit f2163ca
Showing 1 changed file with 12 additions and 2 deletions.
14 changes: 12 additions & 2 deletions CybersecurityBenchmarks/benchmark/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,17 @@ def __str__(self) -> str:
return self.value


def validate_arguments(parser: argparse.ArgumentParser) -> argparse.Namespace:
args = parser.parse_args()
if not args.use_precomputed_responses:
if args.prompt_path is None:
parser.error(
"The following argument is required: --prompt-path, unless argument --use-precomputed-responses is specified."
)

return args


def main(
*,
default_benchmark: BenchmarkKind | None = None,
Expand Down Expand Up @@ -68,7 +79,6 @@ def main(
"--prompt-path",
help="The under-test LLM(s) would be queried with the prompts in this file. This argument works for benchmarks: autocomplete, instruct, mitre.",
default=default_prompt_path,
required=default_prompt_path is None and not default_use_precomputed_responses,
)

parser.add_argument(
Expand Down Expand Up @@ -140,7 +150,7 @@ def main(
help="Enable debug logging",
)

args = parser.parse_args()
args = validate_arguments(parser)

logging.basicConfig(
format="%(asctime)s [%(levelname)s] %(message)s",
Expand Down

0 comments on commit f2163ca

Please sign in to comment.