Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Policy refactoring with automation and testing #1020

Closed
wants to merge 124 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
124 commits
Select commit Hold shift + click to select a range
f15e22c
Add `Alz.Tools` PowerShell module
Jul 19, 2022
cfe9c02
Add script to generate policy resource files
Jul 19, 2022
4327089
Add programatically generated resource files
Jul 19, 2022
f28c2bd
Update defaults to run from repository root directory
Jul 19, 2022
bd49029
Add script and template to auto-generate `policies.json`
Jul 19, 2022
696c8bc
Add auto-generated `policies.json` for testing
Jul 19, 2022
3cd0ae2
Update file naming logic
Jul 20, 2022
b12e2cf
Update file names
Jul 20, 2022
911d863
Stage `policies.json` from Bicep for testing
Jul 21, 2022
ce54b09
Update policy resources in `src` for sov. clouds
Jul 21, 2022
52b3e3d
Replace `policies.json` template with `policies.bicep`
Jul 21, 2022
8ffa084
Update `eslzArm` to use new unified `policies.json`
Jul 21, 2022
692f581
Fix template reference error
Jul 21, 2022
cf6cb3a
Fix `depends_on` for Policy Assignment deployments
Jul 21, 2022
6513335
Update logic for setting deployment location
Jul 21, 2022
96cb56a
Update policy metadata (add source)
Jul 22, 2022
0a8a682
Allow `scope` to be set at deployment
Jul 22, 2022
845fa26
Add linting for `src` directory
Jul 26, 2022
57a99c6
Update `ProviderApiVersions.zip`
Jul 27, 2022
bfa8f26
Add automation to GitHub Actions
Jul 27, 2022
45e9b44
Update actions
Jul 27, 2022
18f0462
Update cron schedule
Jul 27, 2022
59d86a5
Update paths for FileList entries
Jul 27, 2022
f5b4fca
Fix error in module
Jul 27, 2022
3ed4110
Add environment and login steps
Jul 27, 2022
b5533d8
Set permissions on workflow
Jul 27, 2022
d511386
Update permissions for workflow
Jul 27, 2022
3b1ce6f
Udpate RequiredModules
Jul 27, 2022
f3b0935
Update to powershell task
Jul 27, 2022
036c391
Auto-update ALZ.Tools [krowlandson/f3b09359]
invalid-email-address Jul 27, 2022
355d21c
Merge branch 'Azure:main' into policy-refactor
krowlandson Jul 27, 2022
d10f1ab
Test re-generation of policies.json
Jul 27, 2022
849a47d
Merge branch 'policy-refactor' of github.com:krowlandson/Enterprise-S…
Jul 27, 2022
50b606f
Update comments
Jul 27, 2022
50c5178
Checkout against ref
Jul 27, 2022
594ddd2
Udpate git logic
Jul 27, 2022
e9682d9
Udpate grep filter
Jul 27, 2022
95396b0
Add show env
Jul 27, 2022
35716a1
Add github_pr_number to env
Jul 28, 2022
c51c7fd
Fix uses and run error
Jul 28, 2022
342a86f
Update push to head branch
Jul 28, 2022
5cb5c86
Auto-update ALZ.Tools [krowlandson/353b87a5]
invalid-email-address Jul 28, 2022
51f5305
Update GitHub email
Jul 28, 2022
39cb952
Update workflow structure
Jul 28, 2022
5d41df6
Merge branch 'policy-refactor' of github.com:krowlandson/Enterprise-S…
Jul 28, 2022
3e2ca8d
Remove ProviderApiVersions.zip for testing
Jul 28, 2022
d60e701
Update push logic
Jul 28, 2022
62a93cc
Auto-update ALZ.Tools [krowlandson/d60e7017]
github-actions[bot] Jul 28, 2022
21652c7
Test re-generation of policies.json
Jul 28, 2022
3643e00
Auto-update Portal experience [krowlandson/7851d42f]
github-actions[bot] Jul 28, 2022
a85ebf1
Update code review
Jul 28, 2022
009dda3
Logically group and order parameters
Jul 28, 2022
cb0d081
Add param file for testing
Jul 28, 2022
f8ca4fb
Remove incorrect location field
Jul 29, 2022
5a83cb7
Rename and update param file
Jul 29, 2022
8ee4aeb
Add configurable `delayCount` for `preparingToLaunch`
Jul 29, 2022
f1d4a16
Add `Register-AzureSubscriptions` to Alz.Tools
Jul 29, 2022
46065d9
Add initial workflow for Portal deployment testing
Jul 29, 2022
2d3b7ef
Remove schedule
Jul 29, 2022
7d3c06a
Update trigger
Jul 29, 2022
e564b88
Fix Import-Module command
Jul 29, 2022
14204cf
Add step to install Az.Subscription module
Jul 29, 2022
73aa77c
Update logic to create Subscriptions
Jul 29, 2022
4fc9f78
Update logic for parsing subscription data between steps
Jul 29, 2022
21da461
Fix linting errors
Jul 29, 2022
94c52da
Debug script issue
Jul 29, 2022
ebba6cc
Debug function
Jul 29, 2022
e0c0b0c
Debug 400 error "bad request"
Jul 29, 2022
54503f1
Update Subscription Aliases to use REST API
Jul 29, 2022
6ab3eee
Remove Az.Subscription pre-req
Jul 29, 2022
239e2a4
Update subscription logic
Jul 29, 2022
cf8cb90
Debug subscription alias
Jul 29, 2022
16c2fec
Update test workflow
Aug 1, 2022
757e537
Debug test-portal
Aug 1, 2022
a213dfa
Debug again
Aug 1, 2022
b17b7c4
Update to Write-Information
Aug 1, 2022
8e4410e
Fix billing scope mapping
Aug 1, 2022
bbefc22
Update subscriptions_json_path
Aug 1, 2022
ca4aae4
Update subscriptions_json_path
Aug 1, 2022
84183ed
Add show env
Aug 1, 2022
a91540c
Debug subscriptions_json_path
Aug 1, 2022
da83794
Update env
Aug 1, 2022
a11392f
Update test deployment
Aug 1, 2022
efae641
Update test deployment logic
Aug 1, 2022
18e849d
Update logging
Aug 1, 2022
35ad3e6
Add new functions to Alz.Tools
Aug 2, 2022
c6bace9
Update `Invoke-RemoveRsgByPattern` function
Aug 2, 2022
b770d30
Add end-to-end test deployment flow
Aug 2, 2022
2c83892
Fix incorrect variable name
Aug 2, 2022
91f87af
Update message and client auth
Aug 2, 2022
5966f54
Update conditional logic
Aug 2, 2022
68f57a8
Add sleep to reduce likelihood of error
Aug 2, 2022
9360454
Update sleep and supress warnings
Aug 2, 2022
07f1137
Update console messages
Aug 2, 2022
74e4a61
Update rootId generation
Aug 2, 2022
872aaac
Fix paddingChar error
Aug 2, 2022
de1b163
Reduce delayCount for test deployment
Aug 2, 2022
01607fb
Remove prefix from rootId
Aug 2, 2022
10907c7
Cleanup rootId code
Aug 2, 2022
df37544
Fix linting issues
Aug 3, 2022
11c8329
Fix bug on Ubuntu and add WhatIf support
Aug 3, 2022
bdb2a6b
Fix bug on Ubuntu and add WhatIf support
Aug 3, 2022
8ffc424
Wrap output to information messages
Aug 4, 2022
ce0e479
Remove trailing whitespace
Aug 4, 2022
e83c9ee
Reduce default value for delayCount
Aug 4, 2022
cab6f1a
Update Set-AzureSubscriptionAlias function
Aug 6, 2022
4d40601
Update WhatIf messages
Aug 7, 2022
62d28d2
Update what if responses
Aug 8, 2022
238f09b
Convert steps to scripts
Aug 8, 2022
81425a5
Set execution bit
Aug 8, 2022
029f7f7
Revert to uses azure/powershell
Aug 8, 2022
bbd6c65
Point to correct scripts
Aug 8, 2022
0769e3c
Enable deployment steps
Aug 8, 2022
4d354d4
Remove `always()` condition
Aug 8, 2022
f8c96f6
Update information messages and formatting
Aug 9, 2022
0068ab6
Remove PSScriptAnalyzer comments
Aug 9, 2022
a9fc259
Convert destroy steps to scripts
Aug 9, 2022
ae13eb9
Fix linting errors
Aug 9, 2022
9f57f29
Auto-update Portal experience [krowlandson/bd3bc86c]
github-actions[bot] Aug 9, 2022
ab6a2e0
List available pwsh modules
Aug 9, 2022
84d51e1
Merge branch 'policy-refactor' of github.com:krowlandson/Enterprise-S…
Aug 9, 2022
bd7c713
Update required version for Az.Resources
Aug 9, 2022
a3490df
Improve workflow
Aug 9, 2022
9f76aee
Update step label
Aug 9, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions .github/workflows/code-review.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
---
name: Code Review

##########################################
# Start the job on push for all branches #
##########################################

# yamllint disable-line rule:truthy
on:
push: {}
pull_request:
branches:
- main
workflow_dispatch: {}

###############
# Set the Job #
###############
jobs:
super-linter:
name: GitHub Super Linter
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Lint eslzArm directory
uses: docker://github/super-linter:v4.9.5
env:
# Lint all code
VALIDATE_ALL_CODEBASE: true
# Need to define main branch as default
# is set to master in super-linter
DEFAULT_BRANCH: main
# Enable setting the status of each individual linter
# run in the Checks section of a pull request
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Only the following files will be linted:
FILTER_REGEX_INCLUDE: .*eslzArm\/.*
# The following linter types will be enabled:
VALIDATE_ARM: true
VALIDATE_JSON: true
VALIDATE_MARKDOWN: true
# Additional settings:
# Disable errors to only generate a report
DISABLE_ERRORS: true

- name: Lint src directory
uses: docker://github/super-linter:v4.9.5
env:
# Lint all code
VALIDATE_ALL_CODEBASE: true
# Need to define main branch as default
# is set to master in super-linter
DEFAULT_BRANCH: main
# Enable setting the status of each individual linter
# run in the Checks section of a pull request
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Only the following files will be linted:
FILTER_REGEX_INCLUDE: .*src\/.*
FILTER_REGEX_EXCLUDE: .*src\/.*\/(Deploy-Diagnostics-Firewall\.json|Deploy-FirewallPolicy\.json)
# The following linter types will be enabled:
VALIDATE_ARM: true
VALIDATE_BASH: true
VALIDATE_BASH_EXEC: true
VALIDATE_JSON: true
VALIDATE_MARKDOWN: true
VALIDATE_POWERSHELL: true
VALIDATE_YAML: true
# Additional settings:
# If a shell script is not executable, the bash-exec
# linter will report an error when set to true
ERROR_ON_MISSING_EXEC_BIT: true
117 changes: 117 additions & 0 deletions .github/workflows/test-portal.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
---
name: Test Portal Experience

##########################################
# Start the job on push for all branches #
##########################################

# yamllint disable-line rule:truthy
on:
pull_request:
types:
- edited
- opened
- reopened
- synchronize
paths:
- "eslzArm/**.json"
- "src/**.json"
- "src/**.bicep"
workflow_dispatch: {}

env:
GITHUB_COMMIT_ID: ${{ github.event.pull_request.head.sha }}
GITHUB_PR_ID: ${{ github.event.pull_request.id }}
TEMP_SUBSCRIPTIONS_JSON_PATH: "./src/data/subscriptions.json"
TEMP_DEPLOYMENT_OBJECT_PATH: "./src/data/eslzArm.test.deployment.json"

permissions:
contents: read
id-token: write

###############
# Set the Job #
###############
jobs:
test-portal:
name: Test Portal Experience
runs-on: ubuntu-latest
environment: csu-rw

steps:
- name: Check out repository
uses: actions/checkout@v3

- name: Show env
run: env | sort

- name: List available pwsh modules
uses: azure/powershell@v1
with:
inlineScript: Get-Module -ListAvailable
azPSVersion: "latest"

- name: Azure login (OIDC)
uses: azure/login@v1
if: ${{ success() && env.AZURE_CLIENT_SECRET == '' }}
with:
client-id: ${{ secrets.AZURE_CLIENT_ID }}
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
enable-AzPSSession: true
env:
AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }}

- name: Azure login (Client Secret)
uses: azure/login@v1
if: ${{ success() && env.AZURE_CLIENT_SECRET != '' }}
with:
creds: '{"clientId":"${{ secrets.AZURE_CLIENT_ID }}","clientSecret":"${{ secrets.AZURE_CLIENT_SECRET }}","subscriptionId":"${{ secrets.AZURE_SUBSCRIPTION_ID }}","tenantId":"${{ secrets.AZURE_TENANT_ID }}"}'
enable-AzPSSession: true
env:
AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }}

- name: Register subscriptions
uses: azure/powershell@v1
with:
inlineScript: |
./src/scripts/Invoke-ActionRegisterSubscriptions.ps1
Get-Content -Path $env:TEMP_SUBSCRIPTIONS_JSON_PATH | jq
azPSVersion: "latest"
env:
BILLING_SCOPE: ${{ secrets.BILLING_SCOPE }}

- name: Generate eslzArm configuration
id: config
uses: azure/powershell@v1
with:
inlineScript: |
./src/scripts/Invoke-ActionGenerateEslzArmConfig.ps1
Get-Content -Path $env:TEMP_DEPLOYMENT_OBJECT_PATH | jq
azPSVersion: "latest"
env:
DEPLOYMENT_LOCATION: ${{ secrets.DEPLOYMENT_LOCATION }}

- name: Run eslzArm deployment (TEST)
uses: azure/powershell@v1
with:
inlineScript: ./src/scripts/Invoke-ActionRunEslzArmDeployment.ps1 -Test
azPSVersion: "latest"

- name: Run eslzArm deployment (WHAT IF)
uses: azure/powershell@v1
with:
inlineScript: ./src/scripts/Invoke-ActionRunEslzArmDeployment.ps1 -WhatIf
azPSVersion: "latest"

- name: Run eslzArm deployment (DEPLOY)
uses: azure/powershell@v1
with:
inlineScript: ./src/scripts/Invoke-ActionRunEslzArmDeployment.ps1
azPSVersion: "latest"

- name: Run eslzArm deployment (DESTROY)
uses: azure/powershell@v1
with:
inlineScript: ./src/scripts/Invoke-ActionRunEslzArmDeployment.ps1 -Destroy
azPSVersion: "latest"
84 changes: 84 additions & 0 deletions .github/workflows/update-alz-tools.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
---
name: Update ALZ Tools

##########################################
# Start the job on push for all branches #
##########################################

# yamllint disable-line rule:truthy
on:
schedule:
- cron: '0 5 * * 0'
workflow_dispatch: {}

env:
github_user_name: 'github-actions'
github_email: '41898282+github-actions[bot]@users.noreply.github.com'
github_commit_message: 'Auto-update ALZ.Tools'

permissions:
contents: write
id-token: write

###############
# Set the Job #
###############
jobs:
update-alz-tools:
name: Update ALZ Tools
runs-on: ubuntu-latest
environment: csu-ro

steps:
- name: Check out repository
uses: actions/checkout@v3

- name: Configure local git
run: |
git config --global user.name "$github_user_name"
git config --global user.email "$github_email"

- name: Azure login
uses: azure/login@v1
with:
client-id: ${{ secrets.AZURE_CLIENT_ID }}
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
enable-AzPSSession: true

- name: Update ProviderApiVersions
uses: azure/powershell@v1
with:
inlineScript: src/Alz.Tools/scripts/Update-ProviderApiVersionsZip.ps1
azPSVersion: "latest"

- name: Stage changes
run: |
echo "==> Stage changes..."
mapfile -t STATUS_LOG < <(git status --short | grep src/Alz.Tools/)
if [ ${#STATUS_LOG[@]} -gt 0 ]; then
echo "Found changes to the following files:"
printf "%s\n" "${STATUS_LOG[@]}"
git add --all ./src/Alz.Tools/
else
echo "No changes to add."
fi

- name: Push changes
run: |
echo "==> Check git diff..."
mapfile -t GIT_DIFF < <(git diff --cached)
printf "%s\n" "${GIT_DIFF[@]}"

if [ ${#GIT_DIFF[@]} -gt 0 ]; then

echo "==> Commit changes..."
git commit --message "$github_commit_message [$GITHUB_ACTOR/${GITHUB_SHA::8}]"

echo "==> Push changes..."
echo "Pushing changes to: $GITHUB_REPOSITORY"
git push origin

else
echo "No changes found."
fi
80 changes: 80 additions & 0 deletions .github/workflows/update-portal.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
---
name: Update Portal Experience

##########################################
# Start the job on push for all branches #
##########################################

# yamllint disable-line rule:truthy
on:
pull_request:
types:
- edited
- opened
- reopened
- synchronize
paths:
- 'src/**.json'
- 'src/**.bicep'
workflow_dispatch: {}

env:
github_user_name: 'github-actions'
github_email: '41898282+github-actions[bot]@users.noreply.github.com'
github_commit_message: 'Auto-update Portal experience'

###############
# Set the Job #
###############

jobs:
update-portal:
name: Update Portal Experience
runs-on: ubuntu-latest

steps:
- name: Check out repository
uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha }}

- name: Configure local git
run: |
echo "git user name : $github_user_name"
git config --global user.name "$github_user_name"
echo "git user email : $github_email"
git config --global user.email "$github_email"

- name: Update policies
run: bicep build ./src/templates/policies.bicep --outfile ./eslzArm/managementGroupTemplates/policyDefinitions/policies.json

- name: Stage changes
run: |
echo "==> Stage changes..."
mapfile -t STATUS_LOG < <(git status --short | grep eslzArm/)
if [ ${#STATUS_LOG[@]} -gt 0 ]; then
echo "Found changes to the following files:"
printf "%s\n" "${STATUS_LOG[@]}"
git add --all ./eslzArm
else
echo "No changes to add."
fi

- name: Push changes
run: |
echo "==> Check git diff..."
mapfile -t GIT_DIFF < <(git diff --cached)
printf "%s\n" "${GIT_DIFF[@]}"

if [ ${#GIT_DIFF[@]} -gt 0 ]; then

echo "==> Commit changes..."
git commit --message "$github_commit_message [$GITHUB_ACTOR/${GITHUB_SHA::8}]"

echo "==> Push changes..."
echo "Pushing changes to: $GITHUB_REPOSITORY [$GITHUB_HEAD_REF]"
git push origin HEAD:"$GITHUB_HEAD_REF"

else
echo "No changes found."
fi
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,6 @@
!.vscode/extensions.json
*.code-workspace
.vscode/settings.json

# GitHub Super Linter
super-linter.log
Loading