Skip to content

Number of configurations for successes #376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Sep 6, 2020

Conversation

jpreese
Copy link
Member

@jpreese jpreese commented Sep 5, 2020

Most of the diff is from formatting, and simplifying a couple of the tests. The most notable changes include:

  • Bump OPA from 0.20.5 to 0.23.2
  • Convert successes property in JSON output to a count of successes.
  • Use the number of configurations as part of the success count calculation.

Further explanation:

Currently, we're counting the number of rules that match the rule regex (e.g. deny), and we use that value to calculate the number of successes after all evaluations have completed:

	for i := len(totalErrors) + len(totalSuccesses) + len(totalExceptions); i < numberOfRules; i++ {
		totalSuccesses = append(totalSuccesses, output.Result{})
	}

(https://github.com/open-policy-agent/conftest/blob/master/internal/runner/test.go#L267)

However, when multiple configurations are involved, such is the case with multi-document yamls, json, etc--the number of tests is actually the number of rules multipled by how many configurations we are testing against:

	// When there are multiple configurations to be tested, we multiply the number of rules
	// by how many configurations will be tested to get the total number of tests.
	totalTests := numberOfRules
	if _, ok := input.([]interface{}); ok {
		totalTests = numberOfRules * len(input.([]interface{}))
	}

Resolves:
#309
#370

@jpreese jpreese changed the title Successes Number of configurations for successes Sep 5, 2020
Signed-off-by: John Reese <john@reese.dev>
Signed-off-by: John Reese <john@reese.dev>
Signed-off-by: John Reese <john@reese.dev>
@jpreese jpreese force-pushed the successes branch 2 times, most recently from 39d2796 to 2ab5489 Compare September 5, 2020 21:43
Signed-off-by: John Reese <john@reese.dev>
Copy link
Collaborator

@Blokje5 Blokje5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Good cleanups as well!

@Blokje5 Blokje5 merged commit 6e79a2f into open-policy-agent:master Sep 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants