Leveraging AWS Secret Manager Agent to access Secrets

AWS Secret Manager is a well-known service for storing, managing, and accessing sensitive information in a secure way. From database credentials to API keys to application environment variables, they can all be securely stored and managed using Secret Manager. Additional features such as secret rotation, secret encryption, programmatic access, and cross-region replication make this service very reliable for developers and cloud architects.

In July 2024, AWS announced the open-source release of the Secrets Manager Agent. This update introduces a primary feature that significantly enhances the ease of retrieving secrets by reading them from AWS Secrets Manager and caching them in memory. The focus of this release is to simplify the process of accessing secrets, eliminating the need for constant API calls to AWS and reducing the overhead of maintaining separate code for secret management.

Let's test out this new feature and try to retrieve a secret value.

Table of Contents

Secret Manager Agent Features

Let's quickly go over the features of the Secret Manager Agent:

• Can support environments like: EKS, EC2, ECS, and AWS Lambda.
• Ability to cache secrets in memory.
• Can only make read requests (can't modify secrets)
• Offers protection against Server Side Request Forgery (SSRF) to help improve secret security.
• Periodically refreshes the cached secret value.
• Has the ability to change agent configuration.
• Returns secret in the same format as the response of API GetSecretValue.

While the AWS Secrets Manager Agent is a relatively new service, its setup can be somewhat complex and may present a few challenges. However, with the right approach, you can configure it effectively and resolve any errors that arise along the way. Let's walk through the configuration process and address potential issues to ensure a smooth setup.

Create a Secret

Who would've guessed right? The first step is to create a secret. Log in to your AWS account and create a simple secret in Secret Manager.

Secret that I created

Create an EC2 Instance

We'll be testing the AWS Secrets Manager Agent on an EC2 instance, as it offers easier configuration and eliminates the need for additional credential setup. Ensure that the EC2 instance is configured with the necessary permissions for connection. You can choose your preferred method of access: AWS Systems Manager Session Manager, EC2 Instance Connect, or SSH with a key pair. Additionally, make sure the instance has secure internet access.

Once connection to your instance has been established, we can move on to configuring and installing Secret Manager Agent.

Add Secret Manager Access Policy

Attach an IAM policy to the Instance Role of the previously created EC2 Instance. The policy should allow access to the previously created Secret

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "secretmanager",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:DescribeSecret",
                "secretsmanager:GetSecretValue"
            ],
            "Resource": [
                "arn:aws:secretsmanager:<Region>:<Account_ID>:<secret_name>"
            ]
        }
    ]
}

Cloning Secret Manager Agent Git Repository

We will start by installing git and then cloning the git repository containing Secret Manager Agent source code.

(1) Install Git:

$ sudo dnf install git -y

(2) Clone Repository:

$ sudo git clone https://github.com/aws/aws-secretsmanager-agent.git && cd aws-secretsmanager-agent

Build Secret Manager Agent Binary

We need to install standard development tools and Rust to build the agent binary.

Now perform the following operations:

(1) Install development tools:

$ sudo yum -y groupinstall "Development Tools"

(2) Install Rust:

$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
. "$HOME/.cargo/env"

(3) Build the agent binary

$ cargo build --release

A possible error during this step is a Permission denied error. If you lack permissions to compile rust dependencies, you can provide permission using chmod or chown commands. Alternatively, you can also just switch to root user with sudo su.

This should have created a binary with the name aws_secretmanager_agent in target/release.

Install Secret Manager Agent

At this point, if we directly try to execute the agent binary, we face issues since SSRF token variables are not set up for the environment.

$ target/release/aws_secretmanager_agent
Could not read SSRF token variable(s) ["AWS_TOKEN", "AWS_SESSION_TOKEN"]: environment variable not found

To resolve this issue, AWS have conveniently provided an install script in the repository.

The install script is located in aws-secretsmanager-agent/aws_secretsmanager_agent/configuration/install.

The install script creates a SSRF token and stores it in the file /var/run/awssmatoken. Additonally, it creates a user group awssmatokenreader. The token is readable for users in this group.

If you directly try to run the script, you will most likely face this error:

$ sudo bash aws_secretsmanager_agent/configuration/install
Can not read awssmaseedtoken

Let's open the script and have a look at a few parameters defined in the script:

AGENTDIR=/opt/aws/secretsmanageragent
AGENTBIN=aws_secretsmanager_agent
TOKENGROUP=awssmatokenreader
AGENTUSER=awssmauser
TOKENSCRIPT=awssmaseedtoken
AGENTSCRIPT=awssmastartup

These are the default configuration for the install script. The AGENTBIN value should be the path of the binary we created, but by default it is set to current directory (path of the install script).

We can simply update the path, but instead let's copy the binary into the location of install script.

$ cp target/release/aws_secretsmanager_agent aws_secretsmanager_agent/configuration/

These should be the content of the configuration directory before running the install script

$ ls aws_secretsmanager_agent/configuration/
aws_secretsmanager_agent  awssmaseedtoken  awssmaseedtoken.service  awssmastartup.service  install  uninstall

Let's run the installation script now. The installation script will install and run the secret manager agent. On startup, the script generates a random SSRF token and saves it in /var/run/awssmatoken. The token is accessible to members of the awssmatokenreader group, which is created by the installation script.

$ cd aws_secretsmanager_agent/configuration/
$ bash install

if configuration is not correct, you might get the following error:

Can not read awssmaseedtoken

We face this error if the AGENTBIN path is not properly set to the path of the binary.

If you lack permissions to execute the installation script, you can provide permission using chmod or chown commands. Alternatively, you can also just switch to root user with sudo su.

If all goes well, your output should look like this:

Created symlink /etc/systemd/system/multi-user.target.wants/awssmaseedtoken.service → /etc/systemd/system/awssmaseedtoken.service.
Created symlink /etc/systemd/system/multi-user.target.wants/awssmastartup.service → /etc/systemd/system/awssmastartup.service.

Let's confirm that the SSRF token has been created in /var/run/

$ ls /var/run/ | grep awssmatoken
awssmatoken

Testing the Secret Manager Agent

Testing the secret manager agent can be done simply using a curl command to endpoint of the secret.

I would rather prefer accessing through a Python script, since most of my backend development work involves Python frameworks.

This is a script for retrieving secrets:

import requests
import json

SECRET_NAME = '<YOUR_SECRET_NAME>'

# Function that fetches the secret from Secrets Manager Agent for the provided secret id.
def get_secret():
    # Construct the URL for the GET request
    url = f"http://localhost:2773/secretsmanager/get?secretId={SECRET_NAME}"

    # Get the SSRF token from the token file
    with open('/var/run/awssmatoken') as fp:
        token = fp.read()

    headers = {
        "X-Aws-Parameters-Secrets-Token": token.strip()
    }

    try:
        # Send the GET request with headers
        response = requests.get(url, headers=headers)

        # Check if the request was successful
        if response.status_code == 200:
            # Return the secret value
            return json.loads(response.text)
        else:
            # Handle error cases
            raise Exception(f"Status code {response.status_code} - {response.text}")

    except Exception as e:
        # Handle network errors
        raise Exception(f"Error: {e}")

if __name__ == "__main__":
    my_secrets = get_secret()
    print(my_secrets['SecretString'])

Let's run it to retrieve the secret values:

python secret_manager_test.py
{"secret_key_0":"secret_value_0","secret_key_1":"secret_value_0"}

And there it is! The secret values we set initially can be safely retrieved. The Secrets retrieved will be cached in memory eliminating the need for constant API calls.

I hope this feature enhances your development experience and makes working with AWS more efficient and seamless.

Author:

Rahul Raje

JTP Co., Ltd.