Let’s dispense with the debate over whether make or autoconf are appropriate tools in the 21st century and assume that they aren’t going to die anytime soon. Let’s also assume that maybe, just maybe they can be useful?
Cool. Now that we’ve gotten past that hurdle, we can look at one of the uses I have found for these tools — automating infrastructure builds with Terraform. I’ll assume that anyone reading this article is familiar with Terraform and the general pattern of…
terraform plan ...
terraform apply ...…to create resources, typically in a cloud environment. I’ll also assume that you are at least somewhat familiar with make and its use of targets and dependencies to build the artifacts of a project (binaries, libraries, etc).
Makefile dependencies
When using Terraform our target in the general sense is a set of resources to be created. Terraform’s configuration language (HCL) is the source code used to build those resources. For example, here’s the resource definition for creating an AWS VPN connection.
resource "aws_vpn_connection" "google_vpn" {
count = var.google_vpn_enabled ? 1 : 0
customer_gateway_id = aws_customer_gateway.google_gw[0].id
vpn_gateway_id = aws_vpn_gateway.google_vpn_gw[0].id
type = "ipsec.1"
static_routes_only = true
tags = {
Name = "aws-google-vpn"
}
}The HCL that represents our infrastructure resources then becomes the dependencies in our Makefile.
TF_MODULES = \
modules/aws_vpn/vpn.tf \
modules/gcp_vpn/vpn/tfTF_FILES = \
main.tf \
vpn.tf
$(TF_MODULES)TF_STATE_DIR = \
.terraform
Here I have a Terraform project consisting of two modules, one that provisions the components for an AWS VPN and one that provisions the corresponding resources in the Google Cloud Platform. The root of the project contains a main.tf which specifies the provisioners and a vpn.tf file that invokes the modules.
main.tf
terraform {
backend "s3" {
bucket = "some-bucket-name"
key = "terraform/state/terraform.tfstate"
region = "us-east-1"
}
}provider "aws" {
region = "us-east-1"
profile = "my-profile"}provider "google" {
region = "us-east4"
project = "my-project-id"
}data "aws_vpc" "default_vpc" {
provider = aws default = true
}data "aws_route_table" "default_route_table" {
provider = aws vpc_id = data.aws_vpc.default_vpc.id
}data "aws_network_acls" "default_network_acl" {
provider = aws vpc_id = data.aws_vpc.default_vpc.id
}variable "google_vpn_enabled" { }data "aws_region" "aws_region" {
provider = aws
}data "google_client_config" "google_config" {
provider = google
}data "google_compute_subnetwork" "google_subnetwork" {
provider = googlename = "default"
region = data.google_client_config.google_config.region
}
vpn.tf
module "gcp_vpn" {
source = "./modules/gcp_vpn"providers = {
aws = aws
google = google
} google_vpn_enabled = var.google_vpn_enabled
project_id = data.google_client_config.google_config.project
aws_cidr = data.aws_vpc.default_vpc.cidr_block region = data.google_client_config.google_config.region aws_tunnel1_ip = module.aws_vpn.tunnel1_ip
aws_tunnel2_ip = module.aws_vpn.tunnel2_ip aws_tunnel1_psk = module.aws_vpn.tunnel1_psk
aws_tunnel2_psk = module.aws_vpn.tunnel2_psk
}module "aws_vpn" {
source = "./modules/aws_vpn" providers = {
aws = aws
google = google
} region = data.aws_region.aws_region.name vpc_id = data.aws_vpc.default_vpc.id
route_table_id = data.aws_route_table.default_route_table.id
aws_network_acl_id = join("", data.aws_network_acls.default_network_acl.ids)
google_cidr = data.google_compute_subnetwork.google_subnetwork.ip_cidr_range google_vpn_enabled = var.google_vpn_enabled
google_vpn_ip = module.gcp_vpn.google_vpn_ip
}
The specifics of the HCL to create the VPN between AWS and GCP is not important in this article (perhaps part 3?). What is important is that the HCL is a dependency and any time any of these files are modified, we’ll want to update our infrastructure.
make Targets
make wants a build target, so how do we represent the infrastructure as a target that make can build?
Abstractly we can represent a target with a name associated with a set of dependencies and a build rule that will run whenever one of the dependencies changes.
plan: $(TF_FILES)
terraform plan -no-color -var google_vpn_enabled=trueHere we’re saying that the target plan is dependent on the HCL source files.
In our make rule above however there is no artifact named plan that is ever created, hence anytime we run make plan we will run this rule. Let’s add another rule that will really only be run if our HCL changes.
plan.out: $(TF_FILES)
terraform plan -no-color -var google_vpn_enabled=true -out plan.out > plan.txtThis rules creates a target plan.out. terraform has an option that will store the plan in binary format using the -out option. By passing this option with the our target name ( plan.out) we can prevent make from running the plan unless one of our source files has been modified. Now only when the HCL source file’s timestamp is greater than theplan.out timestamp will the plan be run. The plan: rule is still useful if you just want to run the plan. Just run make plan.
We could also get a little fancy here and use a pattern rule and the automatic variables $@ and $* as shown below to make the rule more generic:
%.out: $(TF_FILES)
terraform plan -no-color -var google_vpn_enabled=true -out $@ > $*.txt$@ is the target output file and $* is the stem of the filename, in this case plan. We may also want to take our variables that are input to the Terraform resources and store them in a .tfvars file. That file too is a dependency that should trigger a new plan.
TF_VARS = \
vpn.tfvars%.out: $(TF_FILES) $(TF_VARS)
terraform plan -no-color -out $@ > $*.txt
Forcing a Rebuild
What about forcing a rebuild? Well, you could touch any of the source files, but a better way might be to remove the target build file plan.out as part of a clean rule.
clean:
rm -f plan.*Destroying the Infrasture
…and destroying the infrastructure? Let’s add a new rule called destroy which depends on running the clean rule.
destroy: clean
terraform apply -var google_vpn_enabled=falseNow to destroy the VPN we can run make destroy. Finally, I’ve added an order-only rule to do a first time terraform init if the local state file directory does not yet exist. If any of make plan, make apply or make init are run and Terraform has not yet been initialized then the Makefile will cause the rule to do a terraform init to be executed.
Finally…
Here’s what my final Makefile looks like:
TERRAFORM = /usr/bin/terraform
MODULES = \
modules/aws_vpn/vpn.tf \
modules/gcp_vpn/vpn.tf# dependencies
TF_FILES = \
main.tf \
vpn.tf \
$(MODULES)TF_VARS = \
vpn.tfvarsTF_STATE_DIR = \
.terraformPLAN = plan.outapply: $(PLAN)
$(TERRAFORM) apply -no-color -var-file $(TF_VARS)# terraform plan only
plan: $(TF_FILES) $(TF_STATE) $(TF_VARS) | $(TF_STATE_DIR)
$(TERRAFORM) plan -no-color -var-file $(TF_VARS)# ...this is a 'make' order-only rule, only run terraform init once
# https://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html
%.out: $(TF_FILES) $(TF_STATE) $(TF_VARS) | $(TF_STATE_DIR)
$(TERRAFORM) plan -no-color -var-file $(TF_VARS) -out $@ > $*.txtall: apply.terraform:
$(TERRAFORM) initinit: | $(TF_STATE_DIR)clean:
rm -f plan.*destroy: clean
test -e $(PLAN) && $(TERRAFORM) apply -var google_vpn_enabled=false || true
Note that the apply rule has a prerequisite of $(PLAN) which is set to plan.out So in order to build plan.out we run the rule that matches the pattern %.out.
%.out: $(TF_FILES) $(TF_STATE) $(TF_VARS) | $(TF_STATE_DIR)
$(TERRAFORM) plan -no-color -var-file $(TF_VARS) -out $@ > $*.txtThere you have it a Makefile for bringing up your Terraform resources! I’ve also used make to create AWS AMIs using packer as well as Docker images. In practice though I use Autotools to automate projects. So, in part 2 of this blog post I’ll dive into using Autotools to configure the project to create an even more generic and flexible solution.
