Skip to content

Commit

Permalink
feat: Disable policy check on some repositories (runatlantis#3503)
Browse files Browse the repository at this point in the history 
* feat: Allow enabling policy check per repository

* chore: Tests

* chore: Docs

* chore: Improve tests

* chore: E2E tests
  • Loading branch information
@AndreZiviani
AndreZiviani authored Jul 6, 2023
1 parent 163c5e9 commit 974eed3
Show file tree
Hide file tree
Showing 61 changed files with 1,254 additions and 101 deletions.
4 changes: 2 additions & 2 deletions cmd/server.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1086,8 +1086,8 @@ func (s *ServerCmd) deprecationWarnings(userConfig *server.UserConfig) error {
}
if userConfig.AllowRepoConfig {
deprecatedFlags = append(deprecatedFlags, AllowRepoConfigFlag)
yamlCfg += "\n allowed_overrides: [plan_requirements, apply_requirements, import_requirements, workflow]\n allow_custom_workflows: true"
jsonCfg += `, "allowed_overrides":["plan_requirements","apply_requirements","import_requirements","workflow"], "allow_custom_workflows":true`
yamlCfg += "\n allowed_overrides: [plan_requirements, apply_requirements, import_requirements, workflow, policy_check]\n allow_custom_workflows: true"
jsonCfg += `, "allowed_overrides":["plan_requirements","apply_requirements","import_requirements","workflow", "policy_check"], "allow_custom_workflows":true`
}
jsonCfg += "}]}"

Expand Down
34 changes: 34 additions & 0 deletions runatlantis.io/docs/policy-checking.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@ This section will provide a guide on how to get set up with a simple policy that

Enable the workflow using the following server configuration flag `--enable-policy-checks`

::: warning
All repositories will have policy checking enabled.
:::

### Step 2: Define the policy configuration

Policy Configuration is defined in the [server-side repo configuration](https://www.runatlantis.io/docs/server-side-repo-config.html#reference).
Expand Down Expand Up @@ -191,3 +195,33 @@ When the policy check workflow runs, a file is created in the working directory
]
```

## Running policy check only on some repositories

When policy checking is enabled it will be enforced on all repositories, in order to disable policy checking on some repositories first [enable policy checks](https://www.runatlantis.io/docs/policy-checking.html#getting-started) and then disable it explicitly on each repository with the `policy_check` flag.

For server side config:
```yml
# repos.yaml
repos:
- id: /.*/
plan_requirements: [approved]
apply_requirements: [approved]
import_requirements: [approved]
- id: /special-repo/
plan_requirements: [approved]
apply_requirements: [approved]
import_requirements: [approved]
policy_check: false
```

For repo level `atlantis.yaml` config:
```yml
version: 3
projects:
- dir: project1
workspace: staging
- dir: project1
workspace: production
policy_check: false
```
6 changes: 5 additions & 1 deletion runatlantis.io/docs/server-side-repo-config.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,9 @@ repos:
post_workflow_hooks:
- run: my-post-workflow-hook-command arg1

# policy_check defines if policy checking should be enable on this repository.
policy_check: false

# id can also be an exact match.
- id: github.com/myorg/specific-repo

Expand Down Expand Up @@ -483,7 +486,8 @@ If you set a workflow with the key `default`, it will override this.
| allowed_workflows | []string | none | no | A list of workflows that `atlantis.yaml` files can select from. |
| allow_custom_workflows | bool | false | no | Whether or not to allow [Custom Workflows](custom-workflows.html). |
| delete_source_branch_on_merge | bool | false | no | Whether or not to delete the source branch on merge. |
| repo_locking | bool | false | no | Whether or not to get a lock |
| repo_locking | bool | false | no | Whether or not to get a lock. |
| policy_check | bool | false | no | Whether or not to run policy checks on this repository. |


:::tip Notes
Expand Down
154 changes: 129 additions & 25 deletions server/controllers/events/events_controller_e2e_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -828,10 +828,14 @@ func TestGitHubWorkflowWithPolicyCheck(t *testing.T) {
ModifiedFiles []string
// Comments are what our mock user writes to the pull request.
Comments []string
// PolicyCheck is true if we expect Atlantis to run policy checking
PolicyCheck bool
// ExpAutomerge is true if we expect Atlantis to automerge.
ExpAutomerge bool
// ExpAutoplan is true if we expect Atlantis to autoplan.
ExpAutoplan bool
// ExpPolicyChecks is true if we expect Atlantis to execute policy checks
ExpPolicyChecks bool
// ExpQuietPolicyChecks is true if we expect Atlantis to exclude policy check output
// when there's no error
ExpQuietPolicyChecks bool
Expand All @@ -846,10 +850,12 @@ func TestGitHubWorkflowWithPolicyCheck(t *testing.T) {
ExpReplies [][]string
}{
{
Description: "1 failing policy and 1 passing policy ",
RepoDir: "policy-checks-multi-projects",
ModifiedFiles: []string{"dir1/main.tf,", "dir2/main.tf"},
ExpAutoplan: true,
Description: "1 failing policy and 1 passing policy ",
RepoDir: "policy-checks-multi-projects",
ModifiedFiles: []string{"dir1/main.tf,", "dir2/main.tf"},
PolicyCheck: true,
ExpAutoplan: true,
ExpPolicyChecks: true,
Comments: []string{
"atlantis apply",
},
Expand All @@ -861,10 +867,12 @@ func TestGitHubWorkflowWithPolicyCheck(t *testing.T) {
},
},
{
Description: "failing policy without policies passing using extra args",
RepoDir: "policy-checks-extra-args",
ModifiedFiles: []string{"main.tf"},
ExpAutoplan: true,
Description: "failing policy without policies passing using extra args",
RepoDir: "policy-checks-extra-args",
ModifiedFiles: []string{"main.tf"},
PolicyCheck: true,
ExpAutoplan: true,
ExpPolicyChecks: true,
Comments: []string{
"atlantis apply",
},
Expand All @@ -876,10 +884,12 @@ func TestGitHubWorkflowWithPolicyCheck(t *testing.T) {
},
},
{
Description: "failing policy without policies passing",
RepoDir: "policy-checks",
ModifiedFiles: []string{"main.tf"},
ExpAutoplan: true,
Description: "failing policy without policies passing",
RepoDir: "policy-checks",
ModifiedFiles: []string{"main.tf"},
PolicyCheck: true,
ExpAutoplan: true,
ExpPolicyChecks: true,
Comments: []string{
"atlantis apply",
},
Expand All @@ -906,10 +916,12 @@ func TestGitHubWorkflowWithPolicyCheck(t *testing.T) {
},
},
{
Description: "failing policy additional apply requirements specified",
RepoDir: "policy-checks-apply-reqs",
ModifiedFiles: []string{"main.tf"},
ExpAutoplan: true,
Description: "failing policy additional apply requirements specified",
RepoDir: "policy-checks-apply-reqs",
ModifiedFiles: []string{"main.tf"},
PolicyCheck: true,
ExpAutoplan: true,
ExpPolicyChecks: true,
Comments: []string{
"atlantis apply",
},
Expand All @@ -921,10 +933,12 @@ func TestGitHubWorkflowWithPolicyCheck(t *testing.T) {
},
},
{
Description: "failing policy approved by non owner",
RepoDir: "policy-checks-diff-owner",
ModifiedFiles: []string{"main.tf"},
ExpAutoplan: true,
Description: "failing policy approved by non owner",
RepoDir: "policy-checks-diff-owner",
ModifiedFiles: []string{"main.tf"},
PolicyCheck: true,
ExpAutoplan: true,
ExpPolicyChecks: true,
Comments: []string{
"atlantis approve_policies",
"atlantis apply",
Expand All @@ -941,7 +955,9 @@ func TestGitHubWorkflowWithPolicyCheck(t *testing.T) {
Description: "successful policy checks with quiet flag enabled",
RepoDir: "policy-checks-success-silent",
ModifiedFiles: []string{"main.tf"},
PolicyCheck: true,
ExpAutoplan: true,
ExpPolicyChecks: true,
ExpQuietPolicyChecks: true,
Comments: []string{
"atlantis apply",
Expand All @@ -956,7 +972,9 @@ func TestGitHubWorkflowWithPolicyCheck(t *testing.T) {
Description: "failing policy checks with quiet flag enabled",
RepoDir: "policy-checks",
ModifiedFiles: []string{"main.tf"},
PolicyCheck: true,
ExpAutoplan: true,
ExpPolicyChecks: true,
ExpQuietPolicyChecks: true,
ExpQuietPolicyCheckFailure: true,
Comments: []string{
Expand All @@ -970,10 +988,12 @@ func TestGitHubWorkflowWithPolicyCheck(t *testing.T) {
},
},
{
Description: "failing policy with approval and policy approval clear",
RepoDir: "policy-checks-clear-approval",
ModifiedFiles: []string{"main.tf"},
ExpAutoplan: true,
Description: "failing policy with approval and policy approval clear",
RepoDir: "policy-checks-clear-approval",
ModifiedFiles: []string{"main.tf"},
PolicyCheck: true,
ExpAutoplan: true,
ExpPolicyChecks: true,
Comments: []string{
"atlantis approve_policies",
"atlantis approve_policies --clear-policy-approval",
Expand All @@ -988,6 +1008,86 @@ func TestGitHubWorkflowWithPolicyCheck(t *testing.T) {
{"exp-output-merge.txt"},
},
},
{
Description: "policy checking disabled on specific repo",
RepoDir: "policy-checks-disabled-repo",
ModifiedFiles: []string{"main.tf"},
PolicyCheck: true,
ExpAutoplan: true,
ExpPolicyChecks: false,
Comments: []string{
"atlantis apply",
},
ExpReplies: [][]string{
{"exp-output-autoplan.txt"},
{"exp-output-apply.txt"},
{"exp-output-merge.txt"},
},
},
{
Description: "policy checking disabled on specific repo server side",
RepoDir: "policy-checks-disabled-repo-server-side",
ModifiedFiles: []string{"main.tf"},
PolicyCheck: true,
ExpAutoplan: true,
ExpPolicyChecks: false,
Comments: []string{
"atlantis apply",
},
ExpReplies: [][]string{
{"exp-output-autoplan.txt"},
{"exp-output-apply.txt"},
{"exp-output-merge.txt"},
},
},
{
Description: "policy checking enabled on specific repo but disabled globally",
RepoDir: "policy-checks-enabled-repo",
ModifiedFiles: []string{"main.tf"},
PolicyCheck: false,
ExpAutoplan: true,
ExpPolicyChecks: false,
Comments: []string{
"atlantis apply",
},
ExpReplies: [][]string{
{"exp-output-autoplan.txt"},
{"exp-output-apply.txt"},
{"exp-output-merge.txt"},
},
},
{
Description: "policy checking enabled on specific repo server side but disabled globally",
RepoDir: "policy-checks-enabled-repo-server-side",
ModifiedFiles: []string{"main.tf"},
PolicyCheck: false,
ExpAutoplan: true,
ExpPolicyChecks: false,
Comments: []string{
"atlantis apply",
},
ExpReplies: [][]string{
{"exp-output-autoplan.txt"},
{"exp-output-apply.txt"},
{"exp-output-merge.txt"},
},
},
{
Description: "policy checking disabled on previous regex match but not on repo",
RepoDir: "policy-checks-disabled-previous-match",
ModifiedFiles: []string{"main.tf"},
PolicyCheck: true,
ExpAutoplan: true,
ExpPolicyChecks: false,
Comments: []string{
"atlantis apply",
},
ExpReplies: [][]string{
{"exp-output-autoplan.txt"},
{"exp-output-apply.txt"},
{"exp-output-merge.txt"},
},
},
}

for _, c := range cases {
Expand All @@ -996,7 +1096,7 @@ func TestGitHubWorkflowWithPolicyCheck(t *testing.T) {

// reset userConfig
userConfig = server.UserConfig{}
userConfig.EnablePolicyChecksFlag = true
userConfig.EnablePolicyChecksFlag = c.PolicyCheck
userConfig.QuietPolicyChecks = c.ExpQuietPolicyChecks

ctrl, vcsClient, githubGetter, atlantisWorkspace := setupE2E(t, c.RepoDir, setupOption{})
Expand Down Expand Up @@ -1061,6 +1161,10 @@ func TestGitHubWorkflowWithPolicyCheck(t *testing.T) {
}

_, _, actReplies, _ := vcsClient.VerifyWasCalled(Times(expNumReplies)).CreateComment(Any[models.Repo](), Any[int](), Any[string](), Any[string]()).GetAllCapturedArguments()
if !c.ExpPolicyChecks {
expNumReplies--
}

Assert(t, len(c.ExpReplies) == len(actReplies), "missing expected replies, got %d but expected %d", len(actReplies), len(c.ExpReplies))
for i, expReply := range c.ExpReplies {
assertCommentEquals(t, expReply, actReplies[i], c.RepoDir, c.ExpParallel)
Expand Down
4 changes: 4 additions & 0 deletions ...ontrollers/events/testdata/test-repos/policy-checks-disabled-previous-match/atlantis.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
version: 3
projects:
- dir: .
workspace: default
3 changes: 3 additions & 0 deletions ...nts/testdata/test-repos/policy-checks-disabled-previous-match/exp-output-apply-failed.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Ran Apply for dir: `.` workspace: `default`

**Apply Failed**: All policies must pass for project before running apply.
12 changes: 12 additions & 0 deletions ...ers/events/testdata/test-repos/policy-checks-disabled-previous-match/exp-output-apply.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
Ran Apply for dir: `.` workspace: `default`

```diff
null_resource.simple:
null_resource.simple:

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

workspace = "default"
```
5 changes: 5 additions & 0 deletions ...testdata/test-repos/policy-checks-disabled-previous-match/exp-output-approve-policies.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Approved Policies for 1 projects:

1. dir: `.` workspace: `default`


29 changes: 29 additions & 0 deletions ...estdata/test-repos/policy-checks-disabled-previous-match/exp-output-auto-policy-check.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
Ran Policy Check for dir: `.` workspace: `default`

**Policy Check Failed**: Some policy sets did not pass.
#### Policy Set: `test_policy`
```diff
FAIL - <redacted plan file> - main - WARNING: Null Resource creation is prohibited.

1 test, 0 passed, 0 warnings, 1 failure, 0 exceptions

```


#### Policy Approval Status:
```
policy set: test_policy: requires: 1 approval(s), have: 0.
```
* :heavy_check_mark: To **approve** this project, comment:
* `atlantis approve_policies -d .`
* :put_litter_in_its_place: To **delete** this plan click [here](lock-url)
* :repeat: To re-run policies **plan** this project again by commenting:
* `atlantis plan -d .`

---
* :heavy_check_mark: To **approve** all unapplied plans from this pull request, comment:
* `atlantis approve_policies`
* :put_litter_in_its_place: To delete all plans and locks for the PR, comment:
* `atlantis unlock`
* :repeat: To re-run policies **plan** this project again by commenting:
* `atlantis plan`
Loading

0 comments on commit 974eed3

Please sign in to comment.