How to Fix Terraform Permission Denied on AWS EC2


Troubleshooting “Terraform Permission Denied” on AWS EC2

As a Senior DevOps Engineer, encountering “Terraform Permission Denied” errors is a rite of passage. When running Terraform on an AWS EC2 instance, this specific error almost always points to an issue with how your EC2 instance is authorized to interact with AWS services. Let’s break it down.


1. The Root Cause: IAM Role Misconfiguration

AWS EC2 instances do not have inherent permissions to perform actions against AWS APIs. Instead, they assume an IAM Role, which is attached to an Instance Profile. This IAM Role’s policies dictate precisely what permissions the EC2 instance – and any process running on it, like Terraform – possesses within your AWS account.

A “Permission Denied” error arises when:

  • No IAM Role is attached: The EC2 instance has no identity to assume, resulting in a default denial.
  • Insufficient Permissions: The attached IAM Role’s policies do not grant the necessary Action on the specific Resource that Terraform is attempting to manage (e.g., trying to create an S3 bucket with only EC2-related permissions).
  • Incorrect Resource Scope: The policy might grant an action, but only on a subset of resources, not the one Terraform is targeting (e.g., permission to create S3 buckets, but only in a specific region or with specific tags).
  • Explicit Deny: A policy explicitly denies the action, overriding any allowing policies.

2. Quick Fix (CLI): Attaching/Updating an IAM Role

This section assumes you have aws-cli configured locally or on another EC2 instance with sufficient permissions to modify IAM roles and EC2 instances.

Goal: Ensure your EC2 instance has an IAM Role attached with policies broad enough to perform the required Terraform actions (e.g., AdministratorAccess for testing, then pare down).

  1. Identify your EC2 instance ID:

    aws ec2 describe-instances --filters "Name=tag:Name,Values=YourTerraformRunnerInstance" --query "Reservations[*].Instances[*].InstanceId" --output text
    # Replace 'YourTerraformRunnerInstance' with your instance's Name tag or use other filters.
    # e.g., --filters "Name=instance-state-name,Values=running"

    Let’s assume your instance ID is i-xxxxxxxxxxxxxxxxx.

  2. Check the current IAM Instance Profile (and Role):

    aws ec2 describe-instances --instance-ids i-xxxxxxxxxxxxxxxxx --query "Reservations[*].Instances[*].IamInstanceProfile.Arn" --output text
    • If this returns None or nothing, no role is attached.
    • If it returns an ARN (e.g., arn:aws:iam::123456789012:instance-profile/MyInstanceProfile), then a profile is attached. You’ll need to disassociate it first if you’re replacing it.
  3. Create a new IAM Role (if needed): This role will allow EC2 instances to assume it.

    aws iam create-role --role-name TerraformEC2Role --assume-role-policy-document '{
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": { "Service": "ec2.amazonaws.com" },
          "Action": "sts:AssumeRole"
        }
      ]
    }'
  4. Attach a policy to the new IAM Role: CAUTION: AdministratorAccess is highly permissive and should only be used for initial troubleshooting. For production, apply the principle of least privilege.

    aws iam attach-role-policy --role-name TerraformEC2Role --policy-arn arn:aws:iam::aws:policy/AdministratorAccess
    # For more granular control, use a custom policy ARN or a different managed policy
    # e.g., arn:aws:iam::aws:policy/AmazonS3FullAccess if only managing S3.
  5. Create an Instance Profile for the role (if needed): Instance Profiles are containers for IAM Roles that EC2 instances use.

    aws iam create-instance-profile --instance-profile-name TerraformEC2InstanceProfile
    aws iam add-role-to-instance-profile --instance-profile-name TerraformEC2InstanceProfile --role-name TerraformEC2Role
  6. Attach the Instance Profile to your EC2 instance:

    • If an Instance Profile is already attached and you need to replace it: First, get the association ID:
      aws ec2 describe-instances --instance-ids i-xxxxxxxxxxxxxxxxx --query "Reservations[*].Instances[*].IamInstanceProfile.AssociationId" --output text
      # Let's say it's iap-xxxxxxxxxxxxxxxxx
      aws ec2 disassociate-iam-instance-profile --association-id iap-xxxxxxxxxxxxxxxxx
      # Wait a few seconds for the dissociation to complete.
    • Then, associate the new/updated Instance Profile:
      aws ec2 associate-iam-instance-profile --instance-id i-xxxxxxxxxxxxxxxxx --iam-instance-profile Name=TerraformEC2InstanceProfile

3. Configuration Check: Precedence and Overrides

Even with an Instance Profile attached, other configurations can override or interfere with how Terraform authenticates to AWS. It’s crucial to check the order of precedence.

  1. Verify the IAM Role and Policies on the EC2 Instance (AWS Console / CLI):

    • AWS Console: Navigate to EC2 -> Instances, select your instance, go to “Security” tab, and click on the “IAM role” link. This will take you to the IAM console where you can inspect “Permissions.”
    • CLI:
      # Verify the role name
      aws ec2 describe-instances --instance-ids i-xxxxxxxxxxxxxxxxx --query "Reservations[*].Instances[*].IamInstanceProfile.Arn" --output text
      # Extract role name, e.g., 'TerraformEC2Role' from arn:aws:iam::123456789012:instance-profile/TerraformEC2InstanceProfile
      # Then list attached policies
      aws iam list-attached-role-policies --role-name TerraformEC2Role
      # And inspect specific policy details
      aws iam get-policy-version --policy-arn <policy-arn-from-above> --version-id <latest-version>
    • What to look for:
      • Does the role exist?
      • Is it attached to your instance?
      • Are the specific Action values (e.g., s3:CreateBucket, ec2:RunInstances) that Terraform needs present in the policies?
      • Are the Resource values broad enough (*) or too restrictive (e.g., only specific ARNs that don’t match what Terraform is creating)?
  2. Terraform Provider Configuration (main.tf or similar): Terraform’s AWS provider configuration can explicitly define credentials, overriding the EC2 Instance Profile.

    provider "aws" {
      region = "us-east-1"
      # IMPORTANT: If you want to use the EC2 Instance Profile,
      # DO NOT define access_key, secret_key, or session_token here.
      #
      # Example of overriding (which might be causing your issue):
      # access_key = "AKIA..."
      # secret_key = "YOUR_SECRET_KEY"
      #
      # Example of using a named profile from ~/.aws/credentials (less common on EC2):
      # profile = "my-dev-profile"
    }

    Ensure that your provider "aws" block is clean and relies on the default credential chain, which prioritizes the EC2 Instance Profile when available.

  3. Environment Variables on the EC2 instance: Environment variables like AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN take precedence over instance profiles. Someone might have inadvertently set these. Log into your EC2 instance and check:

    printenv | grep AWS_

    If you see any AWS credential variables defined, they will override the Instance Profile. Consider unsetting them (unset AWS_ACCESS_KEY_ID, etc.) or ensuring they point to valid credentials with sufficient permissions.

  4. AWS Credentials Files on the EC2 instance: Files like ~/.aws/credentials and ~/.aws/config on the EC2 instance can also override or provide alternative credentials. Check the contents of these files:

    cat ~/.aws/credentials
    cat ~/.aws/config

    If these files contain credentials or profiles that are incorrect or lack permissions, they could be the source of the “Permission Denied” error if referenced by Terraform (e.g., via the profile attribute in the provider config).


4. Verification: Test Your Fix

Once you’ve made changes, it’s essential to verify that the EC2 instance can now authenticate correctly and that Terraform can perform its actions.

  1. Test AWS CLI from the EC2 instance: Log into your EC2 instance via SSH and run a simple AWS CLI command that shows the assumed identity:

    aws sts get-caller-identity

    Expected Output: This should return details about the IAM Role associated with your EC2 instance, similar to:

    {
        "UserId": "AROAXXXXXXXXXXXXXXXXX:i-xxxxxxxxxxxxxxxxx",
        "Account": "123456789012",
        "Arn": "arn:aws:iam::123456789012:role/TerraformEC2Role"
    }

    If this command fails with a “Permission Denied” error, or if it shows a different identity than expected, your underlying IAM configuration for the instance is still incorrect.

  2. Rerun Terraform: Navigate to your Terraform project directory on the EC2 instance and re-run your Terraform commands:

    terraform init
    terraform plan
    # If plan succeeds, proceed to apply
    terraform apply

    If all configurations are now correct, Terraform should execute without encountering “Permission Denied” errors, and your desired AWS resources will be managed as expected.

By systematically going through these steps, you can pinpoint and resolve “Terraform Permission Denied” errors on AWS EC2 instances efficiently. Remember to always prioritize the principle of least privilege for any IAM Role in a production environment.