Automating Infrastructure Testing: Using Terraform and Terratest to Validate DevOps Environments

Introduction

Infrastructure as Code (IaC) has revolutionized how we manage cloud environments, allowing teams to treat infrastructure with the same rigor as application code. However, writing Terraform configurations is only half the battle. Without automated validation, a syntax-correct Terraform plan can still result in a misconfigured security group, an unreachable database, or a resource that fails to initialize correctly. This is where testing your infrastructure becomes critical. By integrating automated testing into your CI/CD pipeline, you can catch configuration errors, verify resource state, and ensure that your infrastructure meets your security and operational requirements before it is ever applied to a production environment.

In this guide, we will explore how to use Terratest, a Go-based library, to write automated tests for your Terraform code. We will move beyond simple terraform plan checks and look at how to deploy real resources, verify their state, and ensure they behave as expected.

Why Infrastructure Testing Matters

Traditional infrastructure management often relied on manual verification—deploying a resource and then checking the console to see if it worked. This approach is prone to human error and difficult to scale. Automated testing provides several key benefits:

1. Early Detection of Failures: Catching misconfigurations during the development phase prevents costly downtime and security vulnerabilities in production.
2. Confidence in Refactoring: When you need to update your Terraform modules or upgrade provider versions, tests provide a safety net that confirms your changes haven't introduced regressions.
3. Documentation as Code: Tests serve as living documentation. By reading the test suite, other engineers can understand exactly what a module is supposed to do and how it should behave under specific conditions.
4. Compliance and Security: You can write tests that assert specific security policies, such as ensuring an S3 bucket is not public or that an EC2 instance is running in a private subnet.

Understanding Terratest

Terratest is an open-source library written in Go that makes it easier to write automated tests for your infrastructure. Unlike static analysis tools that only check your code for syntax or style, Terratest actually provisions the infrastructure you define in your Terraform files, verifies that it works, and then cleans it up afterward.

The Workflow

When you run a Terratest suite, the following lifecycle occurs:

1. Setup: Terratest initializes the Terraform directory and runs terraform init and terraform apply.
2. Validation: The test suite executes assertions against the deployed resources. This might involve making HTTP requests to a load balancer, checking the status of a database, or querying the cloud provider's API.
3. Teardown: Terratest runs terraform destroy to clean up the resources, ensuring you don't incur unnecessary costs or leave orphaned infrastructure behind.

Prerequisites

To follow this guide, you will need the following installed on your development machine:

• Go: Terratest is a Go library, so you need the Go toolchain installed.
• Terraform: The version you intend to test.
• Cloud Provider CLI: Access to the cloud provider (e.g., AWS, GCP, or Azure) where you will be testing your infrastructure.

Setting Up Your Testing Environment

Let's assume you have a simple Terraform module that creates an AWS S3 bucket. We want to verify that the bucket is created and that it is private.

1. Project Structure

Organize your project to separate your infrastructure code from your test code:

.
├── main.tf
├── outputs.tf
├── variables.tf
└── test
    ├── go.mod
    ├── go.sum
    └── s3_test.go

2. Initializing the Go Module

Navigate to the test directory and initialize a new Go module:

go mod init example.com/my-infrastructure-tests
go get github.com/gruntwork-io/terratest/modules/terraform

Writing Your First Test

In test/s3_test.go, we will write a test that deploys the infrastructure and checks if the bucket name matches our expectations.

package test

import (
	"testing"
	"github.com/gruntwork-io/terratest/modules/terraform"
	"github.com/stretchr/testify/assert"
)

func TestS3Bucket(t *testing.T) {
	terraformOptions := &terraform.Options{
		TerraformDir: "../",
	}

	// Ensure cleanup happens at the end
	defer terraform.Destroy(t, terraformOptions)

	// Deploy the infrastructure
	terraform.InitAndApply(t, terraformOptions)

	// Get the output from Terraform
	bucketName := terraform.Output(t, terraformOptions, "bucket_name")

	// Assert the result
	assert.Contains(t, bucketName, "my-app-bucket")
}

Explaining the Code

• terraform.Options: This structure tells Terratest where your Terraform code lives. You can also pass variables here using the Vars map.
• defer terraform.Destroy: This is a crucial step. It ensures that even if your test fails, the infrastructure will be destroyed, preventing cost accumulation.
• terraform.InitAndApply: This runs the standard Terraform workflow. If this fails, the test stops immediately.
• terraform.Output: This allows you to retrieve values from your outputs.tf file, which is essential for verifying that the infrastructure was configured as expected.

Advanced Validation Strategies

While checking outputs is a great start, the power of Terratest lies in its ability to interact with the resources you deploy.

Testing HTTP Endpoints

If you are deploying a web server or a load balancer, you should verify that it is actually serving traffic. You can use the http-helper module in Terratest to perform these checks.

import (
	"github.com/gruntwork-io/terratest/modules/http-helper"
	"time"
)

// ... inside your test function
url := terraform.Output(t, terraformOptions, "url")
http_helper.HttpGetWithRetry(t, url, nil, 200, "Hello, World!", 30, 5*time.Second)

This code will attempt to reach the URL every 5 seconds for up to 30 attempts, checking for a 200 OK status and the expected response body. This is much more robust than a single, immediate check, as it accounts for the time it takes for cloud resources to become fully initialized.

Verifying Cloud Provider State

Sometimes you need to check properties that aren't exposed in your Terraform outputs. You can use the official AWS SDK for Go (or the equivalent for your cloud provider) within your test file to perform deep inspections.

import (
	"github.com/aws/aws-sdk-go/service/s3"
	"github.com/gruntwork-io/terratest/modules/aws"
)

// ... inside your test function
region := "us-east-1"
bucketName := terraform.Output(t, terraformOptions, "bucket_name")

// Use Terratest's AWS helper to check if the bucket is private
aws.AssertS3BucketPolicyExists(t, region, bucketName)

Operational Considerations and Best Practices

Testing infrastructure is not without its challenges. Because you are creating real resources, you need to be mindful of cost, speed, and isolation.

1. Cost Management

Always use ephemeral environments for testing. If you are testing a complex database cluster, consider using smaller instance sizes during testing than you would use in production to minimize costs. Ensure that your defer terraform.Destroy logic is robust.

2. Test Isolation

Never run tests against your production infrastructure. Use dedicated AWS accounts or GCP projects for your CI/CD testing pipeline. This prevents accidental deletion of production resources and ensures that your tests don't interfere with real workloads.

3. Speeding Up Tests

Infrastructure tests are inherently slow because they involve API calls to cloud providers. To keep your CI pipeline fast:

• Parallelize Tests: Go's testing framework supports parallel execution. Use t.Parallel() in your test functions to run multiple tests at once.
• Modularize: Break your infrastructure into smaller, testable modules rather than one giant monolithic configuration.
• Use Mocking: Where possible, use mocks for external dependencies to reduce the number of real resources you need to provision.

Troubleshooting Common Failures

When a test fails, the first step is to examine the output of the terraform apply command. Terratest captures this output and prints it to the console. Often, the error is a simple configuration issue, such as a missing IAM permission or a resource conflict.

If the infrastructure deploys successfully but the test fails, check the following:

• Timing Issues: Did your test try to access a resource before it was fully ready? Use retry helpers like http_helper.HttpGetWithRetry to give the resource time to initialize.
• State Mismatch: Is your test expecting a value that isn't actually being output by Terraform? Verify your outputs.tf file.
• Permissions: Does the user or service account running the test have sufficient permissions to create and destroy the resources defined in the test?

Conclusion

Automating your infrastructure testing with Terratest is a significant step toward a more mature DevOps practice. By treating your infrastructure as testable code, you gain the ability to catch errors early, refactor with confidence, and ensure that your production environments remain stable and secure. While it requires an investment in time to set up, the long-term benefits of reduced outages and faster deployment cycles are well worth the effort.