Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON evaluator bug fixes #3

Conversation

joshuaphelpsms
Copy link
Contributor

Purpose

This fixes a couple bugs:

  1. When the ground truth had an array but the actual didn't have all values in the array, the total scores were calculated incorrectly
  2. When the ground truth contained a key that didn't exist in the actual, the individual output for that field was not represented in the output data.
    Ex:
ground_truth = {"test1": "1", "test2": "2"}
actual = {"test1": "1"}
expected output = {"json_evaluator.CustomStringEvaluator.test1": 1, "json_evaluator.CustomStringEvaluator.test2": 0}
previous_output_before_fix =  {"json_evaluator.CustomStringEvaluator.test1": 1}

This also adds the ability to process a schema that is an array. It previously only handled JSON dicts as the initial input

Does this introduce a breaking change?

[ ] Yes
[x] No

Pull Request Type

What kind of change does this Pull Request introduce?

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

@joshuaphelpsms joshuaphelpsms changed the title bug fix JSON evaluator bug fixes Aug 7, 2024
@kjeffc
Copy link

kjeffc commented Aug 8, 2024

Thanks, was running into the bug where the missing fields in the actual data wasn't being taken into consideration when comparing against the ground truth, but this fixed that discrepancy I was seeing.

@albertaga27 albertaga27 merged commit 5ddaa84 into Azure-Samples:main Aug 14, 2024
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants