"Where should we start?" : The biggest question!

That's a very tough part to decide when starting with something new. This blog will guide you on where and how to start with Terraform project. We will walk through the method to gracefully structure the project, basic coding rules to follow, and best practices to make the project efficient and scalable for the future.

Key Factors for a Terraform project

The important factors to consider while creating and working on a Terraform project are:

Project Structure, to keep the files and directories in a standard and consistent format across all the other projects.
Use of Modules, to modularize, create abstraction and increase the reusability of the code.
Naming convention, for configuration objects like variables, modules, files, etc., to ensure consistency in the name of all resources.
Code format and style, to keep the code cleaner and easy to read, understand and maintain.
Variables declaration and assignment, to keep the hard-coded values away, making the code reusable and properly organized in case of modifications in the future.
Outputs from modules, to integrate the modules and the resources within them.
Use of Data Sources, to use the resources defined outside the Terraform project.
Remote State Management, to make it easier for a team of people to collaborate on a project and keep the state file safe from deletion.
Blast radius of the project, to keep the size of the project optimal, and the related resources together, maintaining a good balance between coupling and cohesion.
Version control of the code, to maintain, and manage different versions of the code, which makes it easier to collaborate and share code within teams.

Good way to structure a Terraform Project

It is quite subjective to decide the way the Terraform code should be divided. There are a number of ways in which a Terraform project structure can be defined, based on the size and requirements of the project. For a very simple project, everything can be clubbed into one file which can do our job.

But, if you consider a good architected project, then it should be able to scale easily without the need to recreate everything in case the project grows in the future. There are some ground rules which must be kept in mind while deciding the structure of the project.

One of the decent ways of doing so is like this:

    projectname/
    |
    |-- environment/
    |   |
    |   |-- prod/
    |   |   |
    |   |   |-- region1/
    |   |   |   |
    |   |   |   |-- provider.tf
    |   |   |   |-- version.tf
    |   |   |   |-- backend.tf
    |   |   |   |-- main.tf
    |   |   |   |-- variables.tf
    |   |   |   |-- terraform.tfvars
    |   |   |   |-- outputs.tf
    |   |   |
    |   |   |-- region2/
    |   |   |   |
    |   |   |   |-- ...
    |   |
    |   |-- dev/
    |   |   |
    |   |   |-- ...
    |   |
    |   |-- stage/
    |       |
    |       |-- ...
    |
    |-- modules/
        |
        |-- network/
        |   |
        |   |-- provider.tf
        |   |-- network.tf
        |   |-- loadbalancer.tf
        |   |-- variables.tf
        |   |-- variables-local.tf
        |   |-- outputs.tf
        |   |-- README.md
        |   |-- examples/
        |   |-- docs/
        |
        |-- vm/
        |   |
        |   |-- ...
        |
        |-- scripts/
        |-- files/
        |-- templates/
        |-- ...

In principle, the nesting of folders should not go deeper than 3-4 levels, if possible, in order to keep the complexity of the project less.

Major components of a project structure

Environment:

Using Environment, we can develop Terraform code which can deploy multiple instances of the project in different environments. This is very useful when working with multiple environment development which is similar in nature. In the majority of real-world projects, the Terraform code is developed for multiple environments like development, staging, production, etc.
```
     environment/
     |
     |-- prod/
     |-- dev/
     |-- stage/
     |-- ...
```
Also, within each environment, if the project is expected to be deployed in different regions or zones, it is recommended to further split the project based on regions or zone, or subdivisions, if any.

If there's no such division based on region, then this level can be skipped.
```
     prod/
     |
     |-- global/
     |-- region1/
     |-- region2/
     |-- zone1/
     |-- ...
```
In the next level, we can have all the terraform specification files, where the actual Terraform code exists.
```
     region1/
     |
     |-- provider.tf
     |-- version.tf
     |-- backend.tf
     |-- main.tf
     |-- variables.tf
     |-- terraform.tfvars
     |-- outputs.tf
```
The important files include:
- provider.tf, which specifies the cloud or SaaS providers to use.
- version.tf, for Terraform version specification
- backend.tf, for backend specification to store the state
- main.tf, the root Terraform file from where modules are called
- variables.tf, for variables declaration
- terraform.tfvars, for variables definition or value assignment
- outputs.tf, for output values

Modules:

Modules allow us to create reusable code for infrastructure deployment using Terraform. They act like containers for multiple resources that are used together.

In the modules directory, we create multiple independent subprojects of Terraform, which can be reused from the root Terraform directory.

One of the good ways of structuring a module is as follows:
```
     modulename/
     |
     |-- provider.tf
     |-- resourcegroupname1.tf
     |-- resourcegroupname2.tf
     |-- variables.tf
     |-- variables-local.tf
     |-- outputs.tf
     |-- README.md
     |-- examples/
     |-- docs/  
```
The important files/subdirectories within a module directory are:
- provider.tf file, which specifies the provider used for the resources in the module
- resourcegroup.tf file, which is the actual terraform file for the group of resources which are to be created. It is a good practice to have multiple such terraform files based on a similar group of resources clubbed together in one file, for ease of understanding.
- variables.tf file, for variables declaration
- variables-local.tf file, for local variables which are used only within the module
- outputs.tf file, for output values
- README.md file, for mentioning all the details around the module, including the purpose, resource types, dependencies, etc.
- examples/ directory, for keeping example codes in a separate directory
- docs/ directory, for additional related documents required for the module

Other Directories:
- scripts/ directory, for keeping all the custom scripts, if any.
- files/ directory, for all the static files that Terraform references but does not execute.
- templates/ directory, for the files used by Terraform "templateFile" function.

Conclusion

The initial planning of Terraform project is a very crucial step in every successful project. Since the project structure decides the scalability and reusability, it should be a top priority as it might hamper the efficiency of the project in the long run, if not done properly at an early stage in the project.

We will see more such best practices, methods, and rules around designing and developing projects using Terraform and other IaC Tools in the subsequent blogs. So, stay tuned!

Author: Atul Anand

Fellow