🎯 Terraform Configuration Drift: Detection, Remediation and Best Practices

Configuration Drift occurs when the actual state of resources deployed in the cloud diverges from the desired state defined in your Terraform HCL code files or state file (terraform.tfstate).

⚠️ The hidden danger of Drift: When undetected drift exists, running a simple terraform apply in production can cause accidental destruction of resources, unexpected downtime or security flaws due to altered configurations outside of version control.

1. What is Configuration Drift?

Terraform operates under a declarative model. The ideal Terraform lifecycle assumes a match between 3 key entities:

• HCL Code (Desired State): What you declare in your Git repository.

• Terraform State (Known State): The metadata map in .tfstate that relates your HCL resources to the provider's APIs.

• Real Infrastructure (Current State): What is actually running on AWS, Azure, GCP or Kubernetes.

Drift manifests itself when the Real Infrastructure is directly modified or when external changes break the balance between these 3 layers.

2. Main Causes of Configuration Drift

1. Manual Interventions (Hotfixes): Engineers clicking in the AWS/Azure web console to solve a midnight emergency and forgetting to translate it into HCL code.

1. Automated Services / Auto-scaling: External tools (Karpenter, AWS Autoscaling, patch policies) modifying parameters such as instance count or volume sizes without Terraform's knowledge.

1. Cloud Provider Updates: Automatic changes applied by the cloud provider in default configurations, database engine versions or certificate rotation.

1. Multiple Backends / Team Conflicts: Changes made from local machines with old versions of code or unsynchronized states.

3. How to Detect Configuration Drift in Terraform

A. Detection with terraform plan (Standard Mode)

By default, when you run terraform plan, Terraform refreshes the state by querying the cloud provider's API and calculates the differences from your local HCL code.

terraform plan

B. Detection without modifying code (-refresh-only)

Introduced in Terraform 0.15+, the -refresh-only flag allows you to inspect the real infrastructure and update the tfstate state file with external changes without forcing modification or destruction of the real infrastructure immediately.

terraform plan -refresh-only

# Aplica la actualización del archivo state si las desviaciones externas son aceptadas
terraform apply -refresh-only -auto-approve

C. Automatic Drift Detection in CI/CD (Scheduled Pipelines)

Set up a GitHub Actions, GitLab CI, or Cron Job in Terraform Cloud that runs terraform plan -detailed-exitcode periodically (example: every night at 02:00 AM).

# Exit code 1 = Error de ejecución/API
# Exit code 2 = Drift detectado (Hay diferencias)
terraform plan -detailed-exitcode -no-color

if [ $? -eq 2 ]; then
  echo "🚨 CRITICAL: Configuration Drift detectado en la infraestructura!"
  # Enviar alerta a Slack / PagerDuty
fi

4. Remediation Strategies

Once a configuration deviation has been detected, there are two paths depending on the legitimacy of the change:

Option A: Reverse the Drift (Restore the desired HCL state)

If the manual modification made in the console was an error or unauthorized change, simply run a normal plan and apply. Terraform will override the manual change to align the cloud with the HCL code.

Option B: Accept and Import the Change to the Code (Align HCL)

If the manual change made in the cloud during the emergency was necessary and must be preserved:

1. Run terraform plan -refresh-only to update tfstate.

1. Modify your .tf code in HCL to incorporate the new parameters.

1. Run terraform plan to confirm that there are no accumulated differences ("No changes. Infrastructure is up-to-date.").

Option C: Ignore Legitimate Dynamic Changes (ignore_changes)

For fields managed externally by Auto Scaling, audit tags, or automated patches, use the lifecycle meta-argument:

  name                 = "app-asg"
  min_size             = 2
  max_size             = 10
  desired_capacity     = 4 # Este valor cambia dinámicamente según la carga

  lifecycle {
    # Evita que Terraform sobrescriba la capacidad deseada calculada por AWS Autoscaler
    ignore_changes = [
      desired_capacity,
      tags["LastScanned"],
    ]
  }
}

5. How to Prevent Drift (Best Practices)

Zero Console Access Principle: The most effective preventative measure is to revoke write permissions (Write/Modify) to web consoles for human users, delegating deployments exclusively to CI/CD identities (Service Accounts / IAM Roles).

• 1. Restrict IAM Permissions (Read-Only Console): Applies the principle of least privilege. Engineers should have read-only access to the web console and channel any changes through Pull Requests in Git.

• 2. Deploy GitOps & CI/CD Strict: No team member should run terraform apply from their local terminal. All modifications must go through code review (Peer Code Review) and be executed through automated runners.

• 3. Remote State Locking: Always use remote backends with lock support (AWS S3 + DynamoDB, Azure Blob Storage, HCP Terraform) to prevent concurrent executions that corrupt the state.

• 4. Scheduled Drift Scans: Automate nightly drift scanning jobs that generate alerts in Slack, Teams, or PagerDuty when a discrepancy is detected.

• 5. Use of SCPs (Service Control Policies): In AWS Organizations or Azure Management Groups, blocks the ability to modify key resources outside of the roles assigned to CI/CD pipelines.

6. Specialized Tools for Drift Management