Manage dev and prod using terraform and local yaml files

Manage Dev and Prod Using Terraform and Local yaml Files

Make your terraform spin in multiple projects using yaml

In a corporate production setting, typically cloud projects are split into development (dev) and production (prod) to segregate the two environments. As an engineer, it is important to be able to do the necessary testing and development in (dev) and move them into (prod) once approved, in a quick and efficient manner.

As expected, there are several solutions to do this, terragrunt is an immediate one that comes to mind. However, not everyone is familiar with terraform, much less terragrunt. How can I simplify this template to enable others in a team simply using terraform and yaml files? Here is my take on a simplified implementation.

1. Repository Structure

As a reference for this article, here was how i structured my repository.

.
├── environment            # Folder to contain both dev and prod yaml
│   ├── dev-config.yaml
│   └── prod-config.yaml
├── modules                # Folder to contain terraform code by services
│   ├── module1
│   │   └──main.tf
│   └── module2
│       └── main.tf
├── backend.tf             # backend file to configure terraform backend
├── locals.tf              # locals file to point to environment yaml
└── main.tf                # main file to point to modules

2. Create your terraform locals.tf

The locals.tf file is the default way terraform interacts with locally stored files. Your locals.tf should point to the terraform workspace that you would like to work in, this in turn will call the correct config file.

# Select the workspace
terraform workspace select dev
# This should run terraform plan against 'environment/dev-config.yaml' file
terraform plan

Using a local files that can be committed via git is much preferred over maintaining several .tfvars individually across several developers. Please take care not to commit sensitive information in the yaml file.

Also terraform does allow for other types of config files other than yaml, including json and text [see link].

# filename: locals.tf
locals {
    config = yamldecode(file("environment/${terraform.workspace}-config.yaml"))
}

2. Create configuration files for both dev and prod

In the environment directory, create your config files. Be sure in your naming that the name is also the same one as your terraform workspace as we will be using the variable `${terraform.workspace}` to differentiate between the two files.

# filename: dev-config.yaml
project_id: "cloud-project-dev"  
terraform_state_bucket_name: "terraform_state_bucket-name"
some_dataset_map:
  dataset_1:
    description: "dataset_1"
  dataset_2:
    description: "dataset_2"

# filename: prod-config.yaml
project_id: "cloud-project-prod"  
terraform_state_bucket_name: "terraform_state_bucket-name"

3. Modules

I use module blocks to separate resources used. There is no strict rule as to what goes in a module, but I typically group resources that are spun up for the same service together (e.g. An airflow job that writes to a SQL table, or a VM and its associated load balancers).

The main thing here is that your modules point to a local.config in this case the yaml file that we created. The locals.tf should then take care of the rest.

# filename: main.tf
module "module1" {
  source                      = "./modules/module1"
  terraform_state_bucket_name = local.config.terraform_state_bucket_name
  location                    = local.config.location
}

module "module2" {
  source                      = "./modules/module2"
  project_id                  = local.config.project_id
}

Sources

• Terraform workspaces: link
• Terraform locals: link

Manage dev and prod using terraform and local yaml files was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.