IAM User Access Key

Automatically rotate AWS access keys using Keeper Secrets Manager rotations

Overview

This documentation explains how to rotate AWS IAM user access keys using KeeperPAM's rotation option called "Run PAM scripts only". This is a setting in the PAM User rotation settings which tells the Gateway to skip the primary rotation method and directly execute the post-rotation script attached to the PAM User record in the vault.

This guide includes prerequisites, step-by-step instructions, and a Python script example. This provided script supports both provided admin credentials (AWS Access key for an admin account) and EC2 instance role authentication. The script ensures secure access key rotation, including deletion of previous user keys. The new key is stored in the Keeper record after rotation is complete.

Prerequisites

  1. KSM role: Ensure that the Keeper user has a role providing access to Keeper Secrets Manager, and Keeper Secrets Manager rotations.

  2. A Linux instance to run the Keeper Gateway: The Gateway can be deployed in an EC2 instance, any private Cloud or on-prem. This script is capable of leveraging EC2 Instance Roles Authentication to perform the Access Key rotation if the Gateway runs in an EC2 instance. If the gateway runs somewhere else, then an Access Key with role privilege needs to be provided in the Keeper vault to perform the rotation of the end user Access Key.

Rotation Script Logic Flow

1. Admin Credentials Retrieval

The script retrieve admin credentials in three ways:

  1. Record directly attached to the post rotation script.

  2. The access key provided to the AWS PAM config selected for the rotation. This will be used if no access key is found in the record attached (method 1 above) to the post rotation script.

  3. Uses AWS instance role authentication if no credentials are provided from either methods above. The gateway needs to be running on an EC2 Instance with an EC2 Instance Role in place.

2. Key Rotation Logics

The script provides two modes of operation based on the delete_all_keys_before_rotating custom field:

  • If False (default), a new access key is created first, then the old ones are deleted, keeping only the newly created key. This will fail if the user account has already two access keys: AWS will not allow the script to create a third one.

  • If this custom field is set to True, the script deletes all existing access keys for the IAM user before creating a new one. This helps in the scenario described above where the end user account has already two access keys.

3. Updating the Keeper PAM User Record in the Vault

After key rotation, the script updates the rotated PAM User Keeper record with the new AWS access key ID, secret access key, creation date, and any deleted access keys IDs.

PAM User Record - Fields Requirements

You need to create a PAM User record where the rotation will be configured later on. The fields below need to be created.

Fields required:

Field Name
Description

Login

Name of the user account in AWS where the access key needs to be rotated.

Password

It will be a dummy value in this case. The password field gets automatically rotated, but it is not used anywhere. This is still required field.

Custom fields required:

Custom Field Label
Field Type
Description
aws_access_key_id

Text

This field will receive the new access key id after the rotation.

aws_secret_access_key

Hidden Field

This field will receive the new secret access key after the rotation.

CreateDate

Text

This field will contain the timestamp of when the new key has been generated by the script.

Access_Key_ID_Removed_during_previous_rotation

Text

This field will contain the old access key id(s) removed from the user account in AWS during the rotation.

delete_all_keys_before_rotating

Text

This custom field is optional. It could be set to “False” or “True”, the default value is “False”.

If set to “True” then the rotation script will start by deleting all existing access keys on the user account before creating a new one. This is needed when the user has already 2 access keys setup. AWS will not allow the script to create a third one, hence the need to delete the existing keys before adding a new one.

NOOP

Text

This rotation requires the gateway to only execute the rotation script, and not try to rotate something using the built-in rotation features.

The value has to be:

True
Private Key Type

Text

Second field to enable NOOP.

The value has to be:

rsa-ssh

Instead of creating the PAM User record manually using the details above, you could also import the csv file below. It will create a template record you can amend and duplicate as needed. Importing the file will generate a Login record type: make sure to convert it to PAM User.

213B
PAM User template.csv
CSV file to easily create a PAM User template record via an import in the vault
PAM User record example

Setting Up the Rotation in the Keeper Vault

When the gateway runs in an EC2 instance, you don’t need to provide an admin access key to the script. The gateway will leverage the AWS Instance Role permissions assigned to the VM.

The steps below explains how to set up an EC2 Instance role to the gateway EC2 instance with minimal permissions:

Steps to Add/Configure Instance Role With Minimum Permissions to Rotate Access Keys:

Step 1: Create a Policy in AWS

  1. Go to the IAM Management Console.

  2. Select Policies and click Create policy.

  3. Select JSON and paste the following, make sure to replace your AWS Account ID:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iam:CreateAccessKey",
        "iam:ListAccessKeys",
        "iam:DeleteAccessKey"
      ],
      "Resource": "arn:aws:iam::YOUR_AWS_ACCOUNT_ID_HERE:user/*"
    }
  ]
}
Create a policy in AWS

  1. Name the policy and save it.

Step 2: Create an IAM Role in AWS

  1. Go to the IAM Management Console.

  2. Select Roles and click create role.

  3. Choose AWS service and select EC2.

Create an IAM role in AWS

  1. Attach the necessary IAM policies (e.g., the policy we created above with the minimum permissions).

Attach the policy to the role in AWS

Step 3: Assign the Role to Your EC2 Instance

  1. In the EC2 Management Console, find your instance.

  2. Click Actions > Security > Modify IAM Role.

EC2 Instance contextual menu

  1. Select the role you created and click Update

Assign IAM role to EC2 instance

  1. Verify Instance Role Permissions

You can ensure that the instance role has appropriate permissions to interact with IAM by running the command below on the Gateway EC2 Instance:

aws sts get-caller-identity

After configuring this role, your EC2 instance, in this case your Keeper Gateway, will automatically use the attached role credentials for your script, allowing it to perform actions like creating and deleting IAM access keys without needing explicit access key credentials.

Rotation Configuration From the Vault:

  1. Create a shared folder in the vault

  2. Create a PAM User record in the shared folder with the fields and custom fields described above.

  3. In the Secret Manager tab of the Keeper vault, create a new application for the gateway if there is no gateway yet.

  4. Make sure the Application has edit permissions on the shared folder created above.

  5. Provision the gateway (gateway tab after selecting the application) on an EC2 instance. On the EC2 Instance run the install command provided by the Keeper vault and make sure boto3 and keeper_secrets_manager_core are installed by running the following commands in the EC2 instance:

pip3 install boto3
pip3 install keeper_secrets_manager_core

  1. In the Secret Manager tab of the Keeper vault, go to the PAM Configurations tab. Create a new PAM configuration if needed.

  2. Under Environment you can select “Local Network” or “AWS”. If you select “AWS”, please make sure to leave the “Access Key” and “Secret Access Key” field empty. If you provide one, it will be automatically used by the script instead of using the Instance Role authentication. You will still need to provide the AWS Account ID to the AWS PAM configuration.

  3. Select the gateway, select the shared folder and save the PAM configuration.

PAM Configuration without providing an AWS access key

  1. Edit the PAM User record previously described in this documentation:

    • Password Rotation Settings: select your desired schedule and the PAM configuration created above.

    • Add PAM Script to the record: select the provided file below and make sure to specify the script command:

python3
Attach PAM script with a script command

Python Script

10KB
RotateAWSAccessKeys.py
Download script
import boto3
import sys
import base64
import json
from keeper_secrets_manager_core import SecretsManager
from keeper_secrets_manager_core.storage import FileKeyValueStorage
from botocore.exceptions import ClientError, NoCredentialsError

# sys.stdin is not an array, it cannot be subscripted (i.e., sys.stdin[0])
for base64_params in sys.stdin:
    # Retrieve params from the gateway
    params = json.loads(base64.b64decode(base64_params).decode('utf-8'))
    break

# Retrieve records attached to the post-rotation script in Keeper
records = json.loads(base64.b64decode(params.get('records')).decode('utf-8'))

# Initiate Keeper Secrets Manager SDK using the gateway configuration
secrets_manager = SecretsManager(config=FileKeyValueStorage('/etc/keeper-gateway/gateway-config.json'))

# Function to create a new access key for a user
def create_new_access_key(iam_client, user_name: str) -> tuple:
    new_key = iam_client.create_access_key(UserName=user_name)
    return new_key['AccessKey']['AccessKeyId'], new_key['AccessKey']['SecretAccessKey'], new_key['AccessKey']['CreateDate']

# Function to rotate user's access key
def rotate_user_access_key(iam_client, user_name: str) -> dict:
    if not user_name:
        raise ValueError("User name is required for rotation")

    try:
        # Step 1: Create a new access key for the user
        new_access_key_id, new_secret_access_key, create_date = create_new_access_key(iam_client, user_name)

        print(f"New Access Key created successfully for user {user_name}")

        # Step 2: List all existing access keys for the user
        existing_keys = iam_client.list_access_keys(UserName=user_name)
        deleted_keys = [access_key['AccessKeyId'] for access_key in existing_keys['AccessKeyMetadata']
                        if access_key['AccessKeyId'] != new_access_key_id]

        # Step 3: Delete all other access keys, except the newly created one
        for access_key_id in deleted_keys:
            iam_client.delete_access_key(UserName=user_name, AccessKeyId=access_key_id)

        # Return the results
        return {
            'NewAccessKeyId': new_access_key_id,
            'NewSecretAccessKey': new_secret_access_key,
            'CreateDate': create_date,
            'DeletedAccessKeys': deleted_keys
        }

    except ClientError as e:
        print(f"An error occurred: {e}")
        raise e

# Function to delete all keys and then rotate user's access key
def delete_then_rotate_user_access_key(iam_client, user_name: str) -> dict:
    try:
        # Step 1: List all existing access keys for the user
        existing_keys = iam_client.list_access_keys(UserName=user_name)
        deleted_keys = [access_key['AccessKeyId'] for access_key in existing_keys['AccessKeyMetadata']]

        # Step 2: Delete all keys
        for access_key_id in deleted_keys:
            iam_client.delete_access_key(UserName=user_name, AccessKeyId=access_key_id)

        # Step 3: Create a new access key for the user
        new_access_key_id, new_secret_access_key, create_date = create_new_access_key(iam_client, user_name)

        print(f"New Access Key created successfully for user {user_name}")

        # Return the results
        return {
            'NewAccessKeyId': new_access_key_id,
            'NewSecretAccessKey': new_secret_access_key,
            'CreateDate': create_date,
            'DeletedAccessKeys': deleted_keys
        }

    except ClientError as e:
        print(f"An error occurred: {e}")
        raise e

# Function to assume the role in the target AWS account
def assume_role_in_target_account(role_arn, session_name='CrossAccountSession'):
    sts_client = boto3.client('sts')
    try:
        assumed_role = sts_client.assume_role(
            RoleArn=role_arn,
            RoleSessionName=session_name
        )
        credentials = assumed_role['Credentials']
        return {
            'aws_access_key_id': credentials['AccessKeyId'],
            'aws_secret_access_key': credentials['SecretAccessKey'],
            'aws_session_token': credentials['SessionToken']
        }
    except ClientError as e:
        print(f"Error assuming role: {e}")
        raise e

# Inputs from the gateway and full PAM user query using KSM
pam_user_to_update = secrets_manager.get_secrets([params.get('userRecordUid')])[0]
user_name = params.get('user')

# Retrieve admin user account access key
aws_admin_cred_provided = False
for record in records:
    if "role_arn" in record: # AssumeRole auth detected from the Keeper record
        aws_admin_cred_provided = True
        role_arn = record["role_arn"]
        aws_access_key_id = record["aws_access_key_id"]
        aws_secret_access_key = record["aws_secret_access_key"]
        print(f"Admin access key provided along with role arn for AWS AssumeRole. Using Access Key ID {aws_access_key_id}, from record UID {record['uid']}.")
        # Assume the role and get temporary credentials
        assumed_role_credentials = assume_role_in_target_account(role_arn)
        # Initialize IAM client using the assumed role's credentials
        iam_client = boto3.client(
            'iam',
            aws_access_key_id=assumed_role_credentials['aws_access_key_id'],
            aws_secret_access_key=assumed_role_credentials['aws_secret_access_key'],
            aws_session_token=assumed_role_credentials['aws_session_token'],
            region_name='us-east-1'
        )
        break
    if "aws_access_key_id" in record:
        aws_admin_cred_provided = True
        aws_access_key_id = record["aws_access_key_id"]
        aws_secret_access_key = record["aws_secret_access_key"]
        print(f"Admin access key provided. Using Access Key ID {aws_access_key_id}, from record UID {record['uid']}.")
        # Initialize IAM client with admin credentials
        iam_client = boto3.client('iam',
                                  aws_access_key_id=aws_access_key_id,
                                  aws_secret_access_key=aws_secret_access_key,
                                  region_name='us-east-1')
        break
    if "accessKeyId" in record:
        aws_admin_cred_provided = True
        aws_access_key_id = record["accessKeyId"]
        aws_secret_access_key = record["accessSecretKey"]
        print(f"Admin access key provided. Using Access Key ID {aws_access_key_id}, from record UID {record['uid']}.")
        # Initialize IAM client with admin credentials
        iam_client = boto3.client('iam',
                                  aws_access_key_id=aws_access_key_id,
                                  aws_secret_access_key=aws_secret_access_key,
                                  region_name='us-east-1')
        break
else:
    try:
        # Try to retrieve admin access key from the PAM config using Keeper Secrets Manager
        pam_config = secrets_manager.get_secrets([params.get('providerRecordUid')])[0]
        aws_access_key_id = pam_config.field('accessKeyId', single=True)
        aws_secret_access_key = pam_config.field('accessSecretKey', single=True)

        # Check if keys are retrieved successfully
        if aws_access_key_id and aws_secret_access_key:
            aws_admin_cred_provided = True
            print(f"Admin access key retrieved from the PAM Config. Using Access Key ID {aws_access_key_id}, from the PAM Config.")
            # Initialize IAM client with admin credentials
            iam_client = boto3.client('iam',
                                      aws_access_key_id=aws_access_key_id,
                                      aws_secret_access_key=aws_secret_access_key,
                                      region_name='us-east-1')
        else:
            print("Access key ID or secret key not found in the retrieved Keeper Secrets Manager record.")
    except Exception as e:
        # Handle any error that occurs during retrieval
        print(f"Error retrieving admin access key: {e}")

# Check for instance role credentials if no admin access key is provided
if not aws_admin_cred_provided:
    print("No admin access key provided. Checking for instance role authentication...")
    try:
        # Test if instance role credentials are available by creating an IAM client without credentials
        iam_client = boto3.client('iam')
        iam_client.list_access_keys(UserName=user_name)  # Test if we can access the API
        aws_admin_cred_provided = True
        print("Instance role credentials found. Using instance role authentication.")
    except NoCredentialsError:
        print("No instance role credentials available.")
    except ClientError as e:
        print(f"Error using instance role credentials: {e}")

# Exit if no credentials were provided
if not aws_admin_cred_provided:
    raise RuntimeError("No admin access key or instance role credentials provided. The script will exit.")

# Check if the user specified a delete_all_keys_before_rotating custom field
try:
    delete_all_keys_before_rotating = pam_user_to_update.custom_field('delete_all_keys_before_rotating', single=True)
    delete_all_keys_before_rotating = str(delete_all_keys_before_rotating).lower() in ['true', 'yes', '1']
except Exception as e:
    print(f"OPTIONAL field 'delete_all_keys_before_rotating' not found: {e}")
    delete_all_keys_before_rotating = False  # Set to false if an error occurs

# Call the right rotate function based on user parameters
if delete_all_keys_before_rotating:
    result = delete_then_rotate_user_access_key(iam_client, user_name=user_name)
else:
    result = rotate_user_access_key(iam_client, user_name=user_name)

if result:
    print("\nRotation Result:")
    print(f"New Access Key ID: {result['NewAccessKeyId']}")
    print(f"Create Date: {result['CreateDate']}")
    print(f"Deleted Access Keys: {result['DeletedAccessKeys']}")

    # Update the PAM User record in Keeper
    pam_user_to_update.custom_field('aws_access_key_id', result['NewAccessKeyId'])
    pam_user_to_update.custom_field('aws_secret_access_key', result['NewSecretAccessKey'])
    create_date_str = result['CreateDate'].isoformat() if hasattr(result['CreateDate'], 'isoformat') else str(result['CreateDate'])  # Convert the create_date to string format
    pam_user_to_update.custom_field('CreateDate', create_date_str)
    deleted_keys_str = ', '.join(result['DeletedAccessKeys'])  # Convert deleted access keys to a comma-separated string
    pam_user_to_update.custom_field('Access_Key_ID_Removed_during_previous_rotation', deleted_keys_str)
    secrets_manager.save(pam_user_to_update)
PreviousEC2 Virtual Machine UserNextManaged Database

Last updated

Was this helpful?