Infrastructure as code: state, modules and the mistakes that cost weekends

Why bother with infrastructure as code?

• Reproducibility: a new environment is an apply, not an afternoon of clicking.
• Review: infrastructure changes go through pull requests like everything else.
• History: git blame explains why a security group has that rule.
• Disaster recovery: rebuilding a region is a documented, tested procedure.
• Consistency: staging and production differ only in variables, so 'it worked in staging' means something.

How does Terraform state work?

State is a JSON file mapping resources in your configuration to real infrastructure identifiers, and it is what Terraform diffs against to produce a plan.

terraform {
  backend "s3" {
    bucket         = "acme-tfstate"
    key            = "production/web/terraform.tfstate"
    region         = "ap-south-1"
    dynamodb_table = "tfstate-locks"   # prevents concurrent applies
    encrypt        = true
  }
}

1. 1Store state remotely with locking so two people cannot apply simultaneously.
2. 2Enable versioning on the bucket — recovering a corrupted state file is otherwise impossible.
3. 3Never commit state: it contains resource identifiers and sometimes secrets in plain text.
4. 4Separate state per environment and per major component to limit blast radius.
5. 5Do not hand-edit state. Use terraform import, state mv and state rm deliberately.

How should code be structured?

infra/
  modules/
    network/          # reusable, no environment-specific values
    web-service/
    database/
  environments/
    staging/
      main.tf         # composes modules with staging variables
      terraform.tfvars
    production/
      main.tf
      terraform.tfvars

• Modules take variables and return outputs; they contain no hardcoded environment values.
• Environments are thin compositions, which makes the diff between staging and production readable.
• Pin provider and module versions — an unpinned provider upgrade can rewrite resources.
• Keep modules small enough to understand; a module that manages an entire platform is not reusable.

What is drift and how do you handle it?

Drift is the gap between your configuration and reality — someone changed a setting in the console during an incident and never told the code. The next apply either reverts their fix or fails confusingly.

1. 1Run terraform plan on a schedule and alert on unexpected diffs.
2. 2Restrict console write access in production to break-glass roles that are audited.
3. 3When an emergency console change is necessary, log it and reconcile the code the same week.
4. 4Use import to bring existing resources under management rather than recreating them.

How should it run in CI?

1. 1terraform fmt and validate on every pull request.
2. 2terraform plan posted as a comment on the pull request so reviewers see the actual diff.
3. 3A policy or security scan (tfsec, Checkov) failing on public buckets, open security groups and unencrypted storage.
4. 4Apply only from the main branch, using short-lived federated credentials rather than static keys.
5. 5Production applies behind a manual approval with the reviewed plan attached.