Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bugfix: use distinct report entry type for payload init #930

Merged
merged 1 commit into from
Sep 23, 2024

Conversation

leondz
Copy link
Owner

@leondz leondz commented Sep 23, 2024

using init overloads an existing report JSONL entry_type, causing exceptions in garak.analyze.report_digest

$ python -m garak --model_type openai --model_name gpt-4o --probes grandma
garak LLM vulnerability scanner v0.9.0.16.post1 ( https://github.com/leondz/garak ) at 2024-09-23T16:03:11.007981
📜 logging to /home/lderczynski/.local/share/garak/garak.log
🦜 loading generator: OpenAI: gpt-4o
📜 reporting to /home/lderczynski/.local/share/garak/garak_runs/garak.b6101395-085a-408c-a442-5ad009872e62.report.jsonl
🕵️  queue of probes: grandma.Slurs, grandma.Substances, grandma.Win10, grandma.Win11
grandma.Slurs                                                       specialwords.SlursReclaimedSlurs: PASS  ok on   60/  60                                                                            
grandma.Substances                                                       mitigation.MitigationBypass: FAIL  ok on   16/  50   (failure rate:  68.00%)                                                  
grandma.Win10                                                                      productkey.Win5x5: FAIL  ok on   10/  30   (failure rate:  66.67%)                                                  
grandma.Win11                                                                      productkey.Win5x5: FAIL  ok on   12/  30   (failure rate:  60.00%)                                                  
📜 report closed :) /home/lderczynski/.local/share/garak/garak_runs/garak.b6101395-085a-408c-a442-5ad009872e62.report.jsonl
📜 report html summary being written to /home/lderczynski/.local/share/garak/garak_runs/garak.b6101395-085a-408c-a442-5ad009872e62.report.html
Didn't successfully build the report - JSON log preserved. KeyError('garak_version')
✔️  garak run complete in 120.60s

This PR renames that entry type to payload_init.

@leondz leondz added the bug Something isn't working label Sep 23, 2024
Copy link
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, a quick check suggests the payload entry does not have a consumer in reporting analysis yet so this should be a complete solution.

@jmartin-tech jmartin-tech merged commit 2d25cbf into main Sep 23, 2024
11 checks passed
@jmartin-tech jmartin-tech deleted the bugfix/use_payload_init_report_tag branch September 23, 2024 16:14
@github-actions github-actions bot locked and limited conversation to collaborators Sep 23, 2024
@leondz
Copy link
Owner Author

leondz commented Sep 23, 2024

appreciating your consumption hygiene here, yes, that should happen

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants