Deep State: Extract Code and Resources From the Terraform State File

Terraform Tips — #1

From the Deep — DALLE2

HashiCorp Terraform is a great and useful tool that every DevOps Practitioner should know for creating and maintaining Infrastructure as Code. We use Terraform extensively here at BigPanda to fully manage and define our SAAS Infrastructure as Code. I would like to share a series of useful tips about working with Terraform.

What Prompted Me

One day, when testing different ways to create a large resource as code, I accidentally deleted a complex resource that I had created manually. I needed to find an easy way to recreate the deleted resource, even though I did not have any local Terraform code definition file for it yet. What I found led me to deep dive into Terraform’s state file, for a solution that allowed me to create a full Terraform HCL (HashiCorp Configuration Language) file 750 lines of code long, using just a few simple Terraform commands.

It was a valuable learning experience for me to better understand Terraform and how the state works under the hood and I hope this may help you as well.

The Challenge

The resource I deleted was created manually, and I needed to find a way to easily recreate it without having to write it all over again from scratch. Luckily in my case, although I had deleted the resource remotely on the cloud provider, I had imported the resource into my local Terraform state. Using careful Terraform state manipulation, I managed to create a full Terraform definition file from scratch and successfully re-created the deleted resource using a few simple Terraform commands which I’m going to share with you.

But first, let’s cover some basic Terraform concepts.

Provider

Terraform uses a provider architecture so that when working with a configuration, you must define which Provider you are working with. A Terraform Provider is essentially a Terraform code module that can be used to create resources of a Cloud Infrastructure or SAAS service as code. Terraform uses the module definitions and parameters to know how to create and update your resources in the chosen Provider.

For example, when working with the Amazon Web Services Provider, you must specify the AWS Region you are working in and make sure to authenticate using IAM Roles or AWS Access and Secret Keys.

Resource

Terraform uses the providers to create different objects as code, they can be an AWS VPC, an S3 Bucket, or an RDS Instance.

When writing Terraform code, this is typically referred to as follows:

resource "resource_type" "resource_name" {
  <RESOURCE_PARAMETERS_AND_CONFIGURATION>
}

State

The most important file to know when working with Terraform is the Terraform State file. The state file, as its name suggests, is what the terraform binary interacts with to be able to know what the current state of the remote infrastructure is versus the desired state configured in the Terraform configuration language code. The state is persisted between runs of Terraform as a physical file on local or remote storage, and the file is typically called terraform.tfstate. Before each run, Terraform will start by checking the known state, and then compares the Terraform definition files to the remote infrastructure environment.

Let’s dive into an example terraform.tfstate file.

{
  "version": 4,                 # Version of Terraform state file syntax
  "terraform_version": "1.1.2", # Version of Terraform binary being used
  "serial": 31,                 # Serial number of this version of the state
  "lineage": "4978360c-2654-49fc-38d2-c86f4dc0abfe", # Unique state ID
  "outputs": {},                # Outputted resources
  "resources": [
    {
      <ACTUAL RESOURCES LISTED HERE>
    }
  ]
}

What we can see is that there is quite a lot of useful information available to us in the state file. That information can be inspected manually to check what Terraform currently knows about the current state of the infrastructure.

There is even a command to display the full contents of the current state file:

terraform state pull

We can also list the resources managed by the current state file by using the command:

terraform state list

It is also possible to show a specific resource by specifying the resource ID as follows:

terraform state show resource_type.resource_name

Importing Remote State

Once we have resources in the cloud, even if there are no Terraform files locally that describe them as code, we can import them into our local state file using the terraform import command.

terraform import resource_type.resource_name unique_resource_id

Once the resources are imported into the Terraform state, they will appear in the state file and can be referenced from new Terraform code files or the terraform binary.

Solution

Now that we know a bit more about Terraform state, let's try to combine the commands we learned to recreate our deleted resource!

As mentioned before, although the remote resource had been deleted, I still had the resource definition imported into my local state file. So I knew the resource definition was there and I just needed some state manipulation commands to extract the definition from the state.

Steps

1. Created an empty resource block in a Terraform main.tf file for representing the remote resource.
In my case, this was a datadog_synthetics_test resource.

resource "datadog_synthetics_test" "my_test" {

}

2. Imported the remote resource into the local Terraform state

terraform import datadog_synthetics_test.my_test <RESOURCE_ID>

3. Exported the resource from the state into a new file my_test.tf

terraform state show -no-color datadog_synthetics_test.my_test > my_test.tf

NOTE: I used the -no-color flag, to have the target file as clean as possible without any weird shell color characters.

4. Removed IDs from the file created.

In my case with datadog_synthetics_test resource, the resource IDs are the Terraform resource id and the DataDog monitor id.
(Alternatively, you can just comment them out in the output file)

resource "datadog_synthetics_test" "my_test" {
  # id = "***-***-***"
  # monitor_id      = ******
  .
  .
  .
}

5. Applied the changes

terraform apply my_test.tf

6. Voilà!
My datadog_synthetics_test resource was re-created successfully!

Caveats

As mentioned, the steps above worked for me with the datadog_synthetics_test resource.
The versions of Terraform and the DataDog provider I used were:

❯ terraform -version
Terraform v1.1.2
+ provider registry.terraform.io/datadog/datadog v3.12.0

I hope this solution can help you recreate your resources and generate Terraform code.
Although this worked for me here, my hunch is it will probably not easily translate to work for all resources and providers.

Conclusion

It’s useful to understand the Terraform state file, its format, and its contents when using Terraform for managing infrastructure as code.

Although Terraform is usually used to define infrastructure proactively from code to cloud, this solution can help us do the reverse, restore and recreate resources from cloud to code.

Would love to hear if you find this useful and if this helped you as well!