Skip to content

Commit

Permalink
Migrated to Cloud Build
Browse files Browse the repository at this point in the history
  • Loading branch information
ivankorn committed Oct 9, 2019
1 parent 2a7c72f commit 0d41989
Show file tree
Hide file tree
Showing 36 changed files with 576 additions and 1,595 deletions.
16 changes: 15 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,17 @@
# Changelog
All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [2.0.1] - 2019-XX-YY

### Changed

- Migrated to Cloud Build. [#18]

Please add new unreleased features here.

## [2.0.0] - 2019-07-25
Expand All @@ -12,8 +24,10 @@ Please add new unreleased features here.

This is the initial release of the module, with full support for org policy management.

[Unreleased]: https://github.com/terraform-google-modules/terraform-google-org-policy/compare/v2.0.0...HEAD
[Unreleased]: https://github.com/terraform-google-modules/terraform-google-org-policy/compare/v2.0.1...HEAD
[2.0.0]: https://github.com/terraform-google-modules/terraform-google-org-policy/compare/v2.0.0...v2.0.1
[2.0.0]: https://github.com/terraform-google-modules/terraform-google-org-policy/compare/v1.0.0...v2.0.0
[1.0.0]: https://github.com/terraform-google-modules/terraform-google-org-policy/releases/tag/v1.0.0

[#11]: https://github.com/terraform-google-modules/terraform-google-org-policy/pull/11
[#18]: https://github.com/terraform-google-modules/terraform-google-org-policy/pull/18
114 changes: 114 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Contributing

This document provides guidelines for contributing to the module.

## Dependencies

The following dependencies must be installed on the development system:

- [Docker Engine][docker-engine]
- [Google Cloud SDK][google-cloud-sdk]
- [make]
- [bats](https://github.com/sstephenson/bats) 0.4.0
- [jq](https://stedolan.github.io/jq/) 1.5
- [terraform-docs](https://github.com/segmentio/terraform-docs/releases) 0.3.0

## Generating Documentation for Inputs and Outputs

The Inputs and Outputs tables in the READMEs of the root module,
submodules, and example modules are automatically generated based on
the `variables` and `outputs` of the respective modules. These tables
must be refreshed if the module interfaces are changed.

### Execution

Run `make generate_docs` to generate new Inputs and Outputs tables.

## Integration Testing

### File structure
The project has the following folders and files:

- /: root folder
- /examples: examples for using this module
- /test: Folders with files for testing the module (see Testing section on this file)
- /main.tf: main file for this module, contains primary logic for operate the module
- /*_constraints.tf: files for manage the policy resources
- /variables.tf: all the variables for the module
- /output.tf: the outputs of the module
- /readme.MD: this file

Integration tests are used to verify the behaviour of the root module,
submodules, and example modules. Additions, changes, and fixes should
be accompanied with tests.

The integration tests should be using [Kitchen][kitchen](the module is in transition to it and uses bats tests
hooked up to Cloud Build with a make target for local tests),[Kitchen-Terraform][kitchen-terraform], and [InSpec][inspec]. These
tools are packaged within a Docker image for convenience.

The general strategy for these tests is to verify the behaviour of the
[example modules](./examples/), thus ensuring that the root module,
submodules, and example modules are all functionally correct.

### Test Environment
The easiest way to test the module is in an isolated test project. The setup for such a project is defined in [test/setup](./test/setup/) directory.

To use this setup, you need a service account with Project Creator access on a folder. Export the Service Account credentials to your environment like so:

```
export SERVICE_ACCOUNT_JSON=$(< credentials.json)
```

You will also need to set a few environment variables:
```
export TF_VAR_org_id="your_org_id"
export TF_VAR_folder_id="your_folder_id"
export TF_VAR_billing_account="your_billing_account_id"
```

With these settings in place, you can prepare a test project using Docker:
```
make docker_test_prepare
```

### Noninteractive Execution

Run `make docker_test_integration` to test all of the example modules
noninteractively, using the prepared test project.

### Manual Execution
The integration tests for this module are built with bats, basically the test checks the following:
- Perform `terraform init` command
- Perform `terraform get` command
- Perform `terraform plan` command and check that it'll create *n* resources, modify 0 resources and delete 0 resources
- Perform `terraform apply -auto-approve` command and check that it has created the *n* resources, modified 0 resources and deleted 0 resources
- Perform several `gcloud` commands and check the policies are in the desired state
- Perform `terraform destroy -force` command and check that it has destroyed the *n* resources

Please edit the *test/integration/<list|boolean>_constraints/launch.sh* files in order to specify the test beahvior

You can use the following command to run the integration tests in the folder */test/integration/<list|boolean>_constraints/*

`. launch.sh`

## Linting and Formatting

Many of the files in the repository can be linted or formatted to
maintain a standard of quality.

### Execution

Run `make docker_test_lint`.

[docker-engine]: https://www.docker.com/products/docker-engine
[flake8]: http://flake8.pycqa.org/en/latest/
[gofmt]: https://golang.org/cmd/gofmt/
[google-cloud-sdk]: https://cloud.google.com/sdk/install
[hadolint]: https://github.com/hadolint/hadolint
[inspec]: https://inspec.io/
[kitchen-terraform]: https://github.com/newcontext-oss/kitchen-terraform
[kitchen]: https://kitchen.ci/
[make]: https://en.wikipedia.org/wiki/Make_(software)
[shellcheck]: https://www.shellcheck.net/
[terraform-docs]: https://github.com/segmentio/terraform-docs
[terraform]: https://terraform.io/
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -199,4 +199,4 @@
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
limitations under the License.
91 changes: 45 additions & 46 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright 2018 Google LLC
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -12,57 +12,56 @@
# See the License for the specific language governing permissions and
# limitations under the License.

# Please note that this file was generated from [terraform-google-module-template](https://github.com/terraform-google-modules/terraform-google-module-template).
# Please make sure to contribute relevant changes upstream!

# Make will use bash instead of sh
SHELL := /usr/bin/env bash

# All is the first target in the file so it will get picked up when you just run 'make' on its own
all: check_shell check_python check_golang check_terraform check_docker check_base_files test_check_headers check_headers check_trailing_whitespace generate_docs

# The .PHONY directive tells make that this isn't a real target and so
# the presence of a file named 'check_shell' won't cause this target to stop
# working
.PHONY: check_shell
check_shell:
@source test/make.sh && check_shell

.PHONY: check_python
check_python:
@source test/make.sh && check_python

.PHONY: check_golang
check_golang:
@source test/make.sh && golang

.PHONY: check_terraform
check_terraform:
@source test/make.sh && check_terraform

.PHONY: check_docker
check_docker:
@source test/make.sh && docker

.PHONY: check_base_files
check_base_files:
@source test/make.sh && basefiles
DOCKER_TAG_VERSION_DEVELOPER_TOOLS := 0.1.0
DOCKER_IMAGE_DEVELOPER_TOOLS := cft/developer-tools
REGISTRY_URL := gcr.io/cloud-foundation-cicd

.PHONY: check_shebangs
check_shebangs:
@source test/make.sh && check_bash
# Enter docker container for local development
.PHONY: docker_run
docker_run:
docker run --rm -it \
-e SERVICE_ACCOUNT_JSON \
-v $(CURDIR):/workspace \
$(REGISTRY_URL)/${DOCKER_IMAGE_DEVELOPER_TOOLS}:${DOCKER_TAG_VERSION_DEVELOPER_TOOLS} \
/bin/bash

.PHONY: check_trailing_whitespace
check_trailing_whitespace:
@source test/make.sh && check_trailing_whitespace
# Execute lint tests within the docker container
.PHONY: docker_test_lint
docker_test_lint:
docker run --rm -it \
-v $(CURDIR):/workspace \
$(REGISTRY_URL)/${DOCKER_IMAGE_DEVELOPER_TOOLS}:${DOCKER_TAG_VERSION_DEVELOPER_TOOLS} \
/usr/local/bin/test_lint.sh

.PHONY: test_check_headers
test_check_headers:
@echo "Testing the validity of the header check"
@python test/test_verify_boilerplate.py
# Execute integration tests within the docker container
.PHONY: docker_test_integration
docker_test_integration:
docker run --rm -it \
-e SERVICE_ACCOUNT_JSON \
-e FOLDER_ID \
-e FOLDER_2_ID \
-e PROJECT_ID \
-e PROJECT_EXCLUDE \
-e GOOGLE_CREDENTIALS \
-e GOOGLE_APPLICATION_CREDENTIALS \
-v $(CURDIR):/workspace \
$(REGISTRY_URL)/${DOCKER_IMAGE_DEVELOPER_TOOLS}:${DOCKER_TAG_VERSION_DEVELOPER_TOOLS} \
/bin/bash -c 'cd test/integration/boolean_constraints && bash launch.sh; cd ../list_constraints && bash launch.sh'

.PHONY: check_headers
check_headers:
@echo "Checking file headers"
@python test/verify_boilerplate.py
# Generate documentation
.PHONY: docker_generate_docs
docker_generate_docs:
docker run --rm -it \
-v $(CURDIR):/workspace \
$(REGISTRY_URL)/${DOCKER_IMAGE_DEVELOPER_TOOLS}:${DOCKER_TAG_VERSION_DEVELOPER_TOOLS} \
/bin/bash -c 'source /usr/local/bin/task_helper_functions.sh && generate_docs'

# Alias for backwards compatibility
.PHONY: generate_docs
generate_docs:
@source test/make.sh && generate_docs
generate_docs: docker_generate_docs
73 changes: 0 additions & 73 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,76 +88,3 @@ For a fast install, please configure the variables on init_centos.sh or init_de
The script will do:
- Environment variables setting
- Installation of base packages like wget, curl, unzip, gcloud, etc.

## Development

### File structure
The project has the following folders and files:

- /: root folder
- /examples: examples for using this module
- /test: Folders with files for testing the module (see Testing section on this file)
- /main.tf: main file for this module, contains primary logic for operate the module
- /*_constraints.tf: files for manage the policy resources
- /variables.tf: all the variables for the module
- /output.tf: the outputs of the module
- /readme.MD: this file

### Requirements
- [bats](https://github.com/sstephenson/bats) 0.4.0
- [jq](https://stedolan.github.io/jq/) 1.5
- [terraform-docs](https://github.com/segmentio/terraform-docs/releases) 0.3.0

### Integration tests
The integration tests for this module are built with bats, basically the test checks the following:
- Perform `terraform init` command
- Perform `terraform get` command
- Perform `terraform plan` command and check that it'll create *n* resources, modify 0 resources and delete 0 resources
- Perform `terraform apply -auto-approve` command and check that it has created the *n* resources, modified 0 resources and deleted 0 resources
- Perform several `gcloud` commands and check the policies are in the desired state
- Perform `terraform destroy -force` command and check that it has destroyed the *n* resources

Please edit the *test/integration/<list|boolean>_constraints/launch.sh* files in order to specify the test beahvior

You can use the following command to run the integration tests in the folder */test/integration/<list|boolean>_constraints/*

`. launch.sh`

### Autogeneration of documentation from .tf files
Run
```
make generate_docs
```

### Linting
The makefile in this project will lint or sometimes just format any shell,
Python, golang, Terraform, or Dockerfiles. The linters will only be run if
the makefile finds files with the appropriate file extension.

All of the linter checks are in the default make target, so you just have to
run

```
make -s
```

The -s is for 'silent'. Successful output looks like this

```
Running shellcheck
Running flake8
Running gofmt
Running terraform validate
Running hadolint on Dockerfiles
Test passed - Verified all file Apache 2 headers
```

The linters
are as follows:
* Shell - shellcheck. Can be found in homebrew
* Python - flake8. Can be installed with 'pip install flake8'
* Golang - gofmt. gofmt comes with the standard golang installation. golang
is a compiled language so there is no standard linter.
* Terraform - terraform has a built-in linter in the 'terraform validate'
command.
* Dockerfiles - hadolint. Can be found in homebrew
50 changes: 50 additions & 0 deletions build/int.cloudbuild.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

timeout: 3600s
steps:
- id: prepare
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && prepare_environment']
env:
- 'TF_VAR_org_id=$_ORG_ID'
- 'TF_VAR_folder_id=$_FOLDER_ID'
- 'TF_VAR_billing_account=$_BILLING_ACCOUNT'
- id: test_list_constraints
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
args: ['/bin/bash', '-c', 'cd test/integration/list_constraints && bash launch.sh']
env:
- 'TF_VAR_org_id=$_ORG_ID'
- 'TF_VAR_folder_id=$_FOLDER_ID'
- 'TF_VAR_billing_account=$_BILLING_ACCOUNT'
- id: test_boolean_constraints
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
args: ['/bin/bash', '-c', 'cd test/integration/boolean_constraints && bash launch.sh']
env:
- 'TF_VAR_org_id=$_ORG_ID'
- 'TF_VAR_folder_id=$_FOLDER_ID'
- 'TF_VAR_billing_account=$_BILLING_ACCOUNT'
- id: cleanup
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
args: ['/bin/bash', '-c', 'source /usr/local/bin/task_helper_functions.sh && cleanup_environment']
env:
- 'TF_VAR_org_id=$_ORG_ID'
- 'TF_VAR_folder_id=$_FOLDER_ID'
- 'TF_VAR_billing_account=$_BILLING_ACCOUNT'
tags:
- 'ci'
- 'integration'
substitutions:
_DOCKER_IMAGE_DEVELOPER_TOOLS: 'cft/developer-tools'
_DOCKER_TAG_VERSION_DEVELOPER_TOOLS: '0.4.1'
Loading

0 comments on commit 0d41989

Please sign in to comment.