Skip to content

Commit

Permalink
Support filtering by resource type (#94)
Browse files Browse the repository at this point in the history
  • Loading branch information
hbuckle authored Sep 29, 2021
1 parent a904f76 commit 62051ac
Show file tree
Hide file tree
Showing 7 changed files with 207 additions and 35 deletions.
50 changes: 26 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# [<img src="ttlogo.png" width="300" alt="Terratag Logo">](https://terratag.io)
# [<img src="ttlogo.png" width="300" alt="Terratag Logo">](https://terratag.io)
[![ci](https://github.com/env0/terratag/workflows/ci/badge.svg)](https://github.com/env0/terratag/actions?query=workflow%3Aci+branch%3Amaster) [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fenv0%2Fterratag.svg?type=small)](https://app.fossa.com/projects/git%2Bgithub.com%2Fenv0%2Fterratag?ref=badge_small)

> <sub>Terratag is brought to you with&nbsp;❤️&nbsp; by
>[<img src="logo.svg" width="150">](https://env0.com)
> Let your team manage their own environment in AWS, Azure and Google. <br/>
> Governed by your policies and with complete visibility and cost management.
> <sub>Terratag is brought to you with&nbsp;❤️&nbsp; by
>[<img src="logo.svg" width="150">](https://env0.com)
> Let your team manage their own environment in AWS, Azure and Google. <br/>
> Governed by your policies and with complete visibility and cost management.
## What?
Terratag is a CLI tool allowing for tags or labels to be applied across an entire set of Terraform files. Terratag will apply tags or labels to any AWS, GCP and Azure resources.
Terratag is a CLI tool allowing for tags or labels to be applied across an entire set of Terraform files. Terratag will apply tags or labels to any AWS, GCP and Azure resources.

### Terratag in action
![](https://assets.website-files.com/5dc3f52851595b160ba99670/5f62090d2d532ca35e143133_terratag.gif)
Expand All @@ -27,18 +27,19 @@ Maintaining tags across your application is hard, especially when done manually.
Or download the latest [release binary](https://github.com/env0/terratag/releases) .

1. Initialize Terraform modules to get provider schema and pull child modules:
```bash
terraform init
```bash
terraform init
```
1. Run Terratag
```bash
1. Run Terratag
```bash
terratag -dir=foo/bar -tags={\"environment_id\": \"prod\"}
```
Terratag supports the following arguments:
- `-dir` - optional, the directory to recursively search for any `.tf` file and try to terratag it.
```

Terratag supports the following arguments:
- `-dir` - optional, the directory to recursively search for any `.tf` file and try to terratag it.
- `-tags` - tags, as valid JSON (NOT HCL)
- `-skipTerratagFiles` - optional. Default to `true`. Skips any previously tagged - (files with `terratag.tf` suffix)
- `-filter` - optional. Only apply tags to the selected resource types (regex)

### Example Output
#### Before Terratag
Expand Down Expand Up @@ -148,14 +149,15 @@ locals {
* `-skipTerratagFiles=false` - Dont skip processing `*.terratag.tf` files (when running terratag a second time for the same directory)
* `-verbose=true` - Turn on verbose logging
* `-rename=false` - Instead of replacing files named `<basename>.tf` with `<basename>.terratag.tf`, keep the original filename
* `-filter=<regular expression>` - defaults to `.*`. Only apply tags to the resource types matched by the regular expression

##### See more samples [here](https://github.com/env0/terratag/tree/master/test/fixture)

## Notes
- Resources already having the exact same tag as the one being appended will be overridden

## Develop
Issues and Pull Requests are very welcome!
Issues and Pull Requests are very welcome!

### Prerequisites
- Go > 1.13.5
Expand All @@ -170,8 +172,8 @@ go build
### Test

#### Structure
The test suite will look for fixtures under `test/fixtures/terraform_xx`.
Each fixture placed there should have the following directory structure:
The test suite will look for fixtures under `test/fixtures/terraform_xx`.
Each fixture placed there should have the following directory structure:
```
my_fixture
|+ input
Expand All @@ -180,23 +182,23 @@ my_fixture
|- expected
```

- `input` is where you should place the terraform files of your fixture.
All commands will be executed wherever down the hierarchy where `main.tf` is located.
We do that to allow cases where complex nested submodule resolution may take place, and one would like to test how a directory higher up the hierarchy gets resolved.
- `input` is where you should place the terraform files of your fixture.
All commands will be executed wherever down the hierarchy where `main.tf` is located.
We do that to allow cases where complex nested submodule resolution may take place, and one would like to test how a directory higher up the hierarchy gets resolved.
- `expected` is a directory in which all `.terratag.tf` files will be matched with the output directory

#### What's being tested?
Each test will run:
- `terraform init`
- `terratag`
- `terraform validate`
- `terraform validate`

And finally, will compare the results in `out` with the `expected` directory
And finally, will compare the results in `out` with the `expected` directory

#### Running Tests
Tests can only run on a specific Terraform version -
Tests can only run on a specific Terraform version -
```
go test -run TestTerraformXX
```
```

We use [tfenv](https://github.com/tfutils/tfenv) to switch between versions. The exact versions used in the CI tests can be found under `test/tfenvconf`.
2 changes: 2 additions & 0 deletions cli/cli.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ type Args struct {
Tags string
Dir string
SkipTerratagFiles string
Filter string
IsSkipTerratagFiles bool
Verbose bool
Rename bool
Expand All @@ -26,6 +27,7 @@ func InitArgs() (Args, bool) {
args.Tags = setFlag("tags", "")
args.Dir = setFlag("dir", ".")
args.IsSkipTerratagFiles = booleanFlag("skipTerratagFiles", true)
args.Filter = setFlag("filter", ".*")
args.Verbose = booleanFlag("verbose", false)
args.Rename = booleanFlag("rename", true)

Expand Down
19 changes: 15 additions & 4 deletions main.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,12 @@ import (
"encoding/json"
"log"
"os"
"regexp"
"strings"

. "github.com/env0/terratag/cli"
"github.com/env0/terratag/convert"
"github.com/env0/terratag/errors"
. "github.com/env0/terratag/errors"
"github.com/env0/terratag/file"
. "github.com/env0/terratag/providers"
Expand Down Expand Up @@ -55,26 +57,26 @@ func Terratag(args Args) {

matches := GetTerraformFilePaths(args.Dir)

counters := tagDirectoryResources(args.Dir, matches, args.Tags, args.IsSkipTerratagFiles, tfVersion, args.Rename)
counters := tagDirectoryResources(args.Dir, args.Filter, matches, args.Tags, args.IsSkipTerratagFiles, tfVersion, args.Rename)
log.Print("[INFO] Summary:")
log.Print("[INFO] Tagged ", counters.taggedResources, " resource/s (out of ", counters.totalResources, " resource/s processed)")
log.Print("[INFO] In ", counters.taggedFiles, " file/s (out of ", counters.totalFiles, " file/s processed)")
}

func tagDirectoryResources(dir string, matches []string, tags string, isSkipTerratagFiles bool, tfVersion convert.Version, rename bool) counters {
func tagDirectoryResources(dir string, filter string, matches []string, tags string, isSkipTerratagFiles bool, tfVersion convert.Version, rename bool) counters {
var total counters
for _, path := range matches {
if isSkipTerratagFiles && strings.HasSuffix(path, "terratag.tf") {
log.Print("[INFO] Skipping file ", path, " as it's already tagged")
} else {
perFile := tagFileResources(path, dir, tags, tfVersion, rename)
perFile := tagFileResources(path, dir, filter, tags, tfVersion, rename)
total.Add(perFile)
}
}
return total
}

func tagFileResources(path string, dir string, tags string, tfVersion convert.Version, rename bool) counters {
func tagFileResources(path string, dir string, filter string, tags string, tfVersion convert.Version, rename bool) counters {
perFileCounters := counters{
totalFiles: 1,
}
Expand All @@ -93,6 +95,15 @@ func tagFileResources(path string, dir string, tags string, tfVersion convert.Ve
log.Print("[INFO] Processing resource ", resource.Labels())
perFileCounters.totalResources += 1

matched, err := regexp.MatchString(filter, resource.Labels()[0])
if err != nil {
errors.PanicOnError(err, nil)
}
if !matched {
log.Print("[INFO] Resource excluded by filter, skipping.", resource.Labels())
continue
}

if IsTaggable(dir, *resource) {
log.Print("[INFO] Resource taggable, processing...", resource.Labels())
perFileCounters.taggedResources += 1
Expand Down
38 changes: 31 additions & 7 deletions terratag_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,14 +51,32 @@ func TestTerraform1o0(t *testing.T) {
testTerraform(t, "15_1.0")
}

func TestTerraform1o0WithFilter(t *testing.T) {
testTerraformWithFilter(t, "15_1.0_filter", "azurerm_resource_group|aws_s3_bucket")
}

func testTerraform(t *testing.T, version string) {
for _, tt := range getEntries(version) {
tt := tt // NOTE: https://github.com/golang/go/wiki/CommonMistakes#using-goroutines-on-loop-iterator-variables
t.Run(tt.suite, func(t *testing.T) {
t.Parallel() // marks each test case as capable of running in parallel with each other
g := NewGomegaWithT(t)
itShouldTerraformInit(tt.entryDir, g)
itShouldRunTerratag(tt.entryDir, g)
itShouldRunTerratag(tt.entryDir, "", g)
itShouldRunTerraformValidate(tt.entryDir, g)
itShouldGenerateExpectedTerratagFiles(tt.suiteDir, g)
})
}
}

func testTerraformWithFilter(t *testing.T, version string, filter string) {
for _, tt := range getEntries(version) {
tt := tt // NOTE: https://github.com/golang/go/wiki/CommonMistakes#using-goroutines-on-loop-iterator-variables
t.Run(tt.suite, func(t *testing.T) {
t.Parallel() // marks each test case as capable of running in parallel with each other
g := NewGomegaWithT(t)
itShouldTerraformInit(tt.entryDir, g)
itShouldRunTerratag(tt.entryDir, filter, g)
itShouldRunTerraformValidate(tt.entryDir, g)
itShouldGenerateExpectedTerratagFiles(tt.suiteDir, g)
})
Expand Down Expand Up @@ -89,8 +107,8 @@ func itShouldRunTerraformValidate(entryDir string, g *GomegaWithT) {
g.Expect(err).To(BeNil(), "terraform validate failed")
}

func itShouldRunTerratag(entryDir string, g *GomegaWithT) {
err := terratag(entryDir)
func itShouldRunTerratag(entryDir string, filter string, g *GomegaWithT) {
err := terratag(entryDir, filter)
g.Expect(err).To(BeNil(), "terratag failed")
}

Expand All @@ -109,8 +127,10 @@ func getEntries(version string) []TestCase {
entryFiles, _ := doublestar.Glob(rootDir + terraformDir + entryFilesMatcher)
var testEntries []TestCase
for _, entryFile := range entryFiles {
entryDir := strings.TrimSuffix(entryFile, "/main.tf")
terraformPathSplit := strings.Split(entryFile, terraformDir)
// convert windows paths to use forward slashes
slashed := filepath.ToSlash(entryFile)
entryDir := strings.TrimSuffix(slashed, "/main.tf")
terraformPathSplit := strings.Split(slashed, terraformDir)
pathAfterTerraformDir := terraformPathSplit[1]
suite := strings.Split(pathAfterTerraformDir, "/")[1]
pathBeforeTerraformDir := terraformPathSplit[0]
Expand All @@ -134,14 +154,18 @@ func cloneOutput(inputDirs []string) {
}
}

func terratag(entryDir string) (err interface{}) {
func terratag(entryDir string, filter string) (err interface{}) {
defer func() {
if innerErr := recover(); innerErr != nil {
fmt.Println(innerErr)
err = innerErr
}
}()
os.Args = append(args, "-dir="+entryDir)
if filter == "" {
os.Args = append(args, "-dir="+entryDir)
} else {
os.Args = append(args, "-dir="+entryDir, "-filter="+filter)
}
args, isMissingArg := cli.InitArgs()
if isMissingArg {
return errors.New("Missing arg")
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
}
aws = {
source = "hashicorp/aws"
version = "~> 2.0"
}
}
}

provider "azurerm" {
features {}
}

provider "aws" {
region = "us-east-1"
}

resource "azurerm_resource_group" "should_have_tags" {
name = "example-resources"
location = "West Europe"
tags = merge(tomap({
"oh" = "my"
}), local.terratag_added_main)
}

resource "azurerm_virtual_network" "should_not_have_tags" {
name = "example-network"
resource_group_name = azurerm_resource_group.should_have_tags.name
location = azurerm_resource_group.should_have_tags.location
address_space = ["10.0.0.0/16"]
}

resource "aws_s3_bucket" "should_have_tags" {
bucket = "my-tf-test-bucket"
acl = "private"

tags = merge(tomap({
"Name" = "My bucket"
}), local.terratag_added_main)
}

locals {
terratag_added_main = {"env0_environment_id"="40907eff-cf7c-419a-8694-e1c6bf1d1168","env0_project_id"="43fd4ff1-8d37-4d9d-ac97-295bd850bf94"}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
}
aws = {
source = "hashicorp/aws"
version = "~> 2.0"
}
}
}

provider "azurerm" {
features {}
}

provider "aws" {
region = "us-east-1"
}

resource "azurerm_resource_group" "should_have_tags" {
name = "example-resources"
location = "West Europe"
tags = {
"oh" = "my"
}
}

resource "azurerm_virtual_network" "should_not_have_tags" {
name = "example-network"
resource_group_name = azurerm_resource_group.should_have_tags.name
location = azurerm_resource_group.should_have_tags.location
address_space = ["10.0.0.0/16"]
}

resource "aws_s3_bucket" "should_have_tags" {
bucket = "my-tf-test-bucket"
acl = "private"

tags {
Name = "My bucket"
}
}
43 changes: 43 additions & 0 deletions test/fixture/terraform_15_1.0_filter/aws_azure/input/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
}
aws = {
source = "hashicorp/aws"
version = "~> 2.0"
}
}
}

provider "azurerm" {
features {}
}

provider "aws" {
region = "us-east-1"
}

resource "azurerm_resource_group" "should_have_tags" {
name = "example-resources"
location = "West Europe"
tags = {
"oh" = "my"
}
}

resource "azurerm_virtual_network" "should_not_have_tags" {
name = "example-network"
resource_group_name = azurerm_resource_group.should_have_tags.name
location = azurerm_resource_group.should_have_tags.location
address_space = ["10.0.0.0/16"]
}

resource "aws_s3_bucket" "should_have_tags" {
bucket = "my-tf-test-bucket"
acl = "private"

tags {
Name = "My bucket"
}
}

0 comments on commit 62051ac

Please sign in to comment.