Terratest for testing Terraform-managed Datadog Synthetics

Do you work with infrastructure and yearn for testability like those software devs get to safeguard their code? Terratest can help. Today I am going to focus on what Terratest is, walk you through my first steps at using it by implementing it with Datadog Synthetic Tests in Terraform, my key takeaways from that implementation experience, and also where I would work to improve the code moving forward.

What is Terratest? Developed by Gruntwork, Terratest is meant to test infrastructure and infrastructure as code work. This is exciting for people like me who utilize IaC regularly but crave the opportunity to employ better engineering practices like testing in our use. Written as a Go library, Terratest provides a setup for testing infrastructure from Terraform to Kubernetes and more.

The use case I chose was testing Terraform-managed Synthetic Tests and Monitors in Datadog, as I've been working with Datadog and Synthetics recently, now looking to best bring them into an IaC setup. I heavily based my implementation on the docs example for the Terraform Synthetics Test resource under the Datadog provider and the basic Terratest Terraform example. It spins up the infrastructure, makes a Go-based call to the Datadog Monitor API and confirms the type of the monitor to be synthetic, then tears it down using Terratest.

NOTE: You can access the associated repo for this blog post here!

My main.tf consists of the following, much of it boilerplate and placeholder, to set up a simple Synthetic Test that GETs looking for a 200 status code at a URL:

terraform {
  required_providers {
    datadog = {
      source = "DataDog/datadog"
    }
  }
}

variable "datadog_api_key" {
  type        = string
  description = "Datadog API Key"
  # default = "API key here"
}

variable "datadog_app_key" {
  type        = string
  description = "Datadog Application Key"
  # default = "API key here"
}

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
}

resource "datadog_synthetics_test" "example" {
  type    = "api"
  subtype = "http"
  request_definition {
    method = "GET"
    # url    = "your-url-here"
  }
  request_headers = {
    Content-Type = "application/json"
  }
  assertion {
    type     = "statusCode"
    operator = "is"
    target   = "200"
  }
  locations = ["aws:eu-central-1"]
  options_list {
    tick_every = 900

    retry {
      count    = 2
      interval = 300
    }

    monitor_options {
      renotify_interval = 120
    }
  }
  name    = "An API test on your url"
  message = "Example message"
  tags    = ["foo:bar", "foo", "env:test"]

  status = "live"
}

In my outputs.tf, there's one, crucial output variable:

# Output variables -- monitor ID needed for API call for terratest

output "synthetic-monitor-id" {
  value = datadog_synthetics_test.example.monitor_id
}

This output variable is used for the API call near the end of my Terratest code in Go. Please forgive me, I know not where I Go, this is my first time writing something non-trivial(-ish) in the language. But you'll get the point, here's synthetics_test.go (which needs to be named _test.go to be run with Go test):

package test

import (
	"context"
	"encoding/json"
	"fmt"
	"log"
	"math/big"
	"os"
	"testing"

	"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
	"github.com/DataDog/datadog-api-client-go/v2/api/datadogV1"
	"github.com/gruntwork-io/terratest/modules/terraform"
	"github.com/stretchr/testify/assert"
)

func TestTerraformDataDogSyntheticExample(t *testing.T) {
	// Construct the terraform options with default retryable errors to handle the most common
	// retryable errors in terraform testing.
	terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
		// Set the path to the Terraform code that will be tested.
		TerraformDir: "./",
	})

	// Clean up resources with "terraform destroy" at the end of the test.
	defer terraform.Destroy(t, terraformOptions)

	// Run "terraform init" and "terraform apply". Fail the test if there are any errors.
	terraform.InitAndApply(t, terraformOptions)

	// Run `terraform output` to get the values of output variable: synthetic-monitor-id
	output := terraform.Output(t, terraformOptions, "synthetic-monitor-id")

	// Convert synthetic-monitor-id float to Int64
	flt, _, err := big.ParseFloat(output, 10, 0, big.ToNearestEven)
	if err != nil {
		/* handle any parsing errors here */
	}
	MonitorID, _ := flt.Int64()

	// Spin up DD API client and call Monitors API for info on newly-created monitor by ID
	ctx := datadog.NewDefaultContext(context.Background())
	configuration := datadog.NewConfiguration()
	apiClient := datadog.NewAPIClient(configuration)
	api := datadogV1.NewMonitorsApi(apiClient)
	resp, r, err := api.GetMonitor(ctx, MonitorID, *datadogV1.NewGetMonitorOptionalParameters())

	if err != nil {
		fmt.Fprintf(os.Stderr, "Error when calling `MonitorsApi.GetMonitor`: %v\n", err)
		fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
	}

	// Structure and parse JSON response
	responseContent, _ := json.MarshalIndent(resp, "", "  ")
	m := make(map[string]interface{})
	jsonError := json.Unmarshal(responseContent, &m)
	if jsonError != nil {
		log.Fatal(jsonError)
	}

	// Test that type of alert == synthetics, ideally the test case would be a running monitor
	assert.Equal(t, "synthetics alert", m["type"])
}

How would I improve this work? First, I'd test for something more business-critical than the simple presence of the monitor, perhaps by having Terratest delay, and then look for the monitor to be receiving data and that it's in an OK state. I'd also implement proper handling for API and app keys, which would've been a distraction in getting the MVP off the ground to see how this all works and just write better Go code with more knowledge in the future. Finally, I'd portion out main.tf into a better practices folder and file structure for Terraform.

My key takeaways from writing this bit of code and tweaking it to get it to work are that Terratest with Terraform is useful and reasonably intuitive to get off the ground, presuming you know how to write the tests and what to test for and some Go. Also, Go is pretty fun to work with, coming from a loosely typed background it and Rust have been interesting to pick up recently. Combining HCL and Go to create something more significant than the sum of its parts in providing a testing capability for Terraform, I look forward to exploring Terratest more. Hope this helps you on your journey toward testable IaC! Finally, as usual, leave comments here or contact me on LinkedIn if you have any questions or ways to improve this article/repo. Thanks for reading and take it easy...but take it!