Adding tests for vpc, subnets, and route tables

[brandonjbjelland]

I've added 20 tests here that all look to be passing with test-kitchen. Some supporting files were also brought in to ensure test environment consistency.

[brandonjbjelland]

Is there anything I can do to help push this forward, @antonbabenko ? I'd like to get going adding these to the CI/CD pipeline.

Edit: I gather you're at re:Invent this week. Enjoy your time there and let's discuss this when you get home next week 💃

[antonbabenko]

You was absolutely right about re:invent. Now I am back and reviewing bunch of PRs.

I really dislike the way Terraform configurations are mixed with Ruby (which has 200+ dependencies on itself). These dependencies should live in a separate Docker image, managed centrally and be shared between all other Terraform modules projects we want to test automatically or manually. Tests and complete Terraform code can be passed there.

I have started working on this as a subproject a couple weeks ago, and going to continue working on it during this and next week before releasing it publicly in any condition.

[brandonjbjelland]

That's disappointing. What can we do to move this forward? Most maintainers are delighted to see tests added to their project.

Question: Have you used test-kitchen before? If you haven't, fear not. If you're unfamiliar it's been the de-facto standard infrastructure-as-code test harness system for the past many years and rests very neatly in the toolbelt of many DevOps engineers today as the go-to for any sort of config management testing. Likewise, it fits the terraform use case quite well when combined with kitchen-terraform.

Given test-kitchen's ubiquity and the maturity of the tools surrounding it (provisioners nearly all covered, several cloud provider testing libraries supported, winrm/ssh transport libraries to perform local tests, testing through bastions, etc.) test-kitchen easily stands the best chance of becoming a centerpiece of testing in the terraform universe. There's nothing in this realm that comes close. To try and create a competing ecosystem just to have it written in golang and looking more like HCL would be a gargantuan effort and probably not the best use of anyone's time.

I'm not primarily a ruby dev but there's just no substitute for test kitchen in our infra space, and while not ideal, I see mixing languages is just a fact of our profession. You don't need to know ruby well to make use of the handful of best in breed tools it provides. Orient yourself with the infra testing landscape and reconsider.

Supporting links:

• Terraform Testing Framework hashicorp/terraform#5059
• https://www.contino.io/insights/top-3-terraform-testing-strategies-for-ultra-reliable-infrastructure-as-code
• https://devops.stackexchange.com/questions/863/how-to-test-a-terraform-configuration
• https://stackoverflow.com/questions/43875414/how-to-test-terraform-templates-other-than-trial-and-error
• https://www.unixdaemon.net/cloud/testing-terraform-projects/

[antonbabenko]

I think there was some misunderstanding, sorry for that. I have just realized that you may think that I started to work on a project which is similar to existing testing frameworks. I didn't :)

I completely support the need for testing Terraform modules, I have also used kitchen-terraform before (with varied success though), and I have proposed to use it for these modules also. I am completely on-boarded with this, there is no need to convince me.

The only point I was trying to make was that terraform-kitchen tests should be part of the repository, but the whole ruby-world should be living outside of this repository and be treated as a black-box. As a developer I should be able to run something like:

docker run -i -t -v .:/data --workdir=/data ... run-tests

then wait for some time, and get the result back.

Do you agree with this?

[brandonjbjelland]

Heh! Indeed I've misinterpreted and outdone myself here. Sorry from me as well. 😅

Re: ruby being a black box to this repo - If I understand what you're saying, you're trying to package test kitchen and dependent gems into a container, mount a shared volume/directory (e.g. the repo root) and have the container work from those contents? I'm struggling to see how much ruby you can cut from the repo by taking that approach as fixtures, tests, kitchen.yml, ect would all need to still be present. Maybe you've got something else in mind. Lay it out here if you can.

[ncs-alane]

@antonbabenko:

Given that packaging up Ruby and Kitchen-Terraform in a Docker image will only serve to remove the Gemfile from the repository and replace Ruby with Docker in the continuous integration job, can we not merge these tests now and iterate with the new testing approach when it is ready?

[brandonjbjelland]

Anything to share on your project @antonbabenko ?

I have started working on this as a subproject a couple weeks ago, and going to continue working on it during this and next week before releasing it publicly in any condition.

[antonbabenko]

I have made some progress in this direction (packaging most of Terraform/Terragrunt dependencies, integrate with CircleCI, output plan, etc), but it is not as much as I would like. In particular, I am still missing some scripting around packaging Ruby dependencies, for example.

In order to make this repository to run some integration tests, I agree that we can merge this PR as is and improve it BEFORE copy-pasting this solution to other terraform-aws-modules!

Do you agree?

[antonbabenko]

@brandoconnor Could you also upgrade this initial test suite to run on Kitchen Terraform version 3? From what I read it will change some things.

[brandonjbjelland]

I'm just back from holiday but I'll look to upgrade that before end of the weekend.

[brandonjbjelland]

Test suite updated to use latest version of kitchen terraform. Tests are passing:

vpc 'test-example'
  should exist
  should be available
  should have tag "Name"
  should have tag "Owner"
  should have tag "Environment"
  should have route table "test-example-public"
  should have route table "test-example-private-eu-west-1a"
  should have route table "test-example-private-eu-west-1b"

subnet 'test-example-public-eu-west-1a'
  should exist
  should be available
  should belong to vpc "test-example"
  should have tag "Name"
  should have tag "Owner"
  should have tag "Environment"

subnet 'test-example-public-eu-west-1b'
  should exist
  should be available
  should belong to vpc "test-example"
  should have tag "Name"
  should have tag "Owner"
  should have tag "Environment"

Finished in 4.25 seconds (files took 2.75 seconds to load)
20 examples, 0 failures

       Finished verifying <default-aws> (0m9.03s).
-----> Kitchen is finished. (0m9.40s)

[antonbabenko]

Good job with this tests.

[antonbabenko]

Revert these in gitignore.

[brandonjbjelland]

I'm not sure when terraform.tfstate (tests should confine these to an already ignored dir) or terraform.tfvars (all vars are packaged, why would we need override) would exist in the repo. Can you explain the use case for either of those existing within the module?

[antonbabenko]

During work, I run code from examples on my mac and don't want to commit terraform.tfstate file. Sometimes I also have special terraform.tfvars in specific examples but it is rather seldom when I need it.

[brandonjbjelland]

Removed, but consider moving the examples from manually run to kitchen terraform to also run with the automated suite. Even a test fixture without tests is valuable and lowers developer effort. That would make the gitignores here irrelevant.

[antonbabenko]

TF_VAR_region is not required, because there is default value set (eu-west-1), right?

[antonbabenko]

This description does not entirely match to the code in main.tf

[brandonjbjelland]

Looks ready to go.

[antonbabenko]

v1.18.0 has been released. Thank you @brandoconnor for this work!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.