Velero is an open-source tool that allows you to backup and restore your Kubernetes cluster resources and persistent volumes. Velero backups support a number of different storage providers including Amazon S3. The process of setting up Velero backup with S3 using AWS credentials has been documented by Velero. However, at the time of this post, there is no official documentation on how to set up Velero using IRSA or IAM Roles for Service Accounts.
IRSA provides access to Kubernetes pods through service accounts. This offers a more secure alternative to access your AWS services as it eliminates the need to generate and store access key and secret key credentials. Instead, it allows you to assign an AWS Identity and Access Management (IAM) role to a Kubernetes service account. This provides a secure and scalable way to grant permissions to pods in a Kubernetes cluster.
The benefits of using IRSA in AWS include:
- Enhanced security: IRSA allows you to assign granular permissions to pods in a Kubernetes cluster, which helps to reduce the risk of security breaches. By using IAM roles, you can apply the principle of least privilege, ensuring that each pod only has access to the resources it needs to function.
- Simplified access management: By using IRSA, you can manage access to AWS resources at the Kubernetes namespace level, rather than having to manage access on a per-pod basis. This simplifies access management and reduces the risk of errors.
- Improved scalability: IRSA allows you to automate the process of granting permissions to pods in a Kubernetes cluster. This means that you can easily scale your cluster up or down without having to manually manage access permissions.
- Better auditability: IRSA provides better auditability by allowing you to track access to AWS resources at the Kubernetes namespace level. This can help you to identify potential security risks and ensure that your cluster is properly secured.
- Integration with other AWS services: IRSA is fully integrated with other AWS services, such as AWS Security Hub and AWS CloudTrail, which provides additional security and compliance benefits.
The IAM role ARN (Amazon Resource Name) is set as an annotation in the Kubernetes service account which can then be assigned to a pod. IRSA utilizes IAM OIDC, or OpenID Connect, which allows the cluster to authenticate AWS API requests and obtain new JSON Web Tokens (JWT) for each pod that is using the service account.
In this blog post, we go through setting up Velero on an Amazon Elastic Kubernetes Service (EKS) cluster using IRSA. We start by creating the IAM OIDC provider, followed by creating the S3 bucket and the associated IAM role. Finally, we install Velero and configuring the backup storage location to access our S3 bucket without using any credentials.
Prerequisites
- An existing AWS EKS cluster.
- The AWS CLI installed and configured.
AWS CLI install and update instructions – AWS Command Line Interface - The kubectl command installed and configured to access your EKS cluster.
Install Tools – Kubernetes Documentation - The eksctl CLI installed.
Installing or updating eksctl – Amazon EKS - The Velero CLI installed.
Velero Backup Docs – Basic Install - Set the following environment variables:
# The name of the EKS cluster
EKS_CLUSTER_NAME=<name_of_eks_cluster>
# The service account and namespace where Velero will be installed
NAMESPACE=velero
SERVICE_ACCOUNT=velero
# The ID of the AWS account
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
# The region of the EKS cluster
AWS_REGION="us-east-1"
# The IAM OIDC provider of the EKS cluster
OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --region $AWS_REGION --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")
# The name of the S3 bucket that will be created to store Velero backups
BUCKET=<name_of_s3_bucket>
# The name of the IAM role that will be created for Velero
VELERO_ROLE=<name_of_velero_iam_role
Creating an IAM OIDC providerÂ
Although by default every EKS cluster has an OIDC issuer URL associated with it, the IAM OIDC provider for that issuer is not created automatically. Run the following commands to create the IAM OIDC provider if it does not exist:Â
- Get the OIDC provider ID for your EKS cluster:
$ OIDC_ID=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
- Check if an IAM OIDC provider already exists for the cluster:
$ aws iam list-open-id-connect-providers | grep $OIDC_ID | cut -d "/" -f4
- If the IAM OIDC provider does not exist (the above command returned no output), create one using the following command:
$ eksctl utils associate-iam-oidc-provider --cluster $EKS_CLUSTER_NAME --approve
2023-02-28 14:16:27 [ℹ] will create IAM Open ID Connect provider for cluster "test-velero-irsa-cluster" in "us-east-1"
2023-02-28 14:16:27 [✔] created IAM Open ID Connect provider for cluster "test-velero-irsa-cluster" in "us-east-1"
Creating an S3 bucket
- Create an S3 bucket to store Velero backups:
$ aws s3api create-bucket --bucket $BUCKET --region $AWS_REGION
{
"Location": "test-velero-irsa-bucket"
}
2. Create an IAM role to give access to Velero backups:Â
- Create a policy with permissions to take Velero backups and access the S3 bucket that we just created:
$ cat > velero-policy.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeVolumes", "ec2:DescribeSnapshots", "ec2:CreateTags", "ec2:CreateVolume", "ec2:CreateSnapshot", "ec2:DeleteSnapshot" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:DeleteObject", "s3:PutObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts" ], "Resource": [ "arn:aws:s3:::${BUCKET}/*" ] }, { "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::${BUCKET}" ] } ] } EOF
- Create a trust relationship policy document to allow Velero pods to assume the role that we will create in the next step:Â
$ cat > velero-trust-policy.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/$OIDC_PROVIDER" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "$OIDC_PROVIDER:aud": "sts.amazonaws.com", "$OIDC_PROVIDER:sub": "system:serviceaccount:$NAMESPACE:$SERVICE_ACCOUNT" } } } ] } EOF
- Run the following commands to create the role and attach the trust relationship policy document:Â
$ aws iam create-role --role-name $VELERO_ROLE --assume-role-policy-document file://./velero-trust-policy.json { "Role": { "Path": "/", "RoleName": "velero-setup-test-role", "RoleId": "AROARXPYALXLKYXXXXXX", "Arn": "arn:aws:iam::XXXXXXXXXXXX:role/velero-setup-test-role", "CreateDate": "2023-02-28T14:25:30+00:00", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXX" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.us-east-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXX:aud": "sts.amazonaws.com", "oidc.eks.us-east-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXX:sub": "system:serviceaccount:velero:velero" } } } ] } } }
$ aws iam put-role-policy --role-name $VELERO_ROLE --policy-name "velero-trust-policy" --policy-document file://./velero-policy.json
Installing Velero
- Run the following command to install Velero in our EKS cluster:
$ velero install \ --provider aws \ --plugins velero/velero-plugin-for-aws:v1.0.0 \ --sa-annotations eks.amazonaws.com/role-arn=arn:aws:iam::$AWS_ACCOUNT_ID:role/$VELERO_ROLE \ --no-default-backup-location \ --use-node-agent \ --use-volume-snapshots \ --uploader-type "restic" \ --no-secret
- The –sa-annotations parameter contains the ARN of the IAM role that we’ve just created.Â
- No credentials will be used by Velero so the –no-secret flag must be present.Â
- The BSL will be created in the next step, so no bucket information is provided and the –no-default-backup-location flag is set.Â
- To backup persistent volume data, Restic or Kopia may be used by simply setting the –uploader-type and –plugins parameters.
- Confirm that the Velero pod is running before continuing:Â
$ kubectl -n velero get pods NAME READY STATUS RESTARTS AGE velero-5f96c58b96-6krvn 1/1 Running 0 92s
During the installation, a service account called Velero is created and annotated with the ARN of the IAM role. This service account is used by the Velero pod. EKS then injects environment variables AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE into the pod.Â
$ kubectl -n velero get pod velero-5f96c58b96-vsh42 -o jsonpath='{range .spec.containers[0].env[*]}{.name}{"\n"}{end}'
VELERO_SCRATCH_DIR
VELERO_NAMESPACE
LD_LIBRARY_PATH
AWS_STS_REGIONAL_ENDPOINTS
AWS_DEFAULT_REGION
AWS_REGION
AWS_ROLE_ARN
AWS_WEB_IDENTITY_TOKEN_FILE
Velero is set up to access AWS services using IRSA, but we will still need to define the BSL to run backups.Â
2. Create the BSL, or Backup Storage Location, resource:Â
$ velero backup-location create $BUCKET \ --provider aws \ --bucket $BUCKET \ --config region=$AWS_REGION,s3ForcePathStyle="true" \ --default Backup storage location "test-velero-irsa-bucket" configured successfully.
Backup storage location “test-velero-irsa-bucket” configured successfully.Â
- Notice the BSL does not contain any information about AWS credentials. Instead, we only need to reference the S3 bucket that we have created earlier.Â
- The –default flag will make this BSL our default storage location during Velero backups and restores.Â
3. Check if the BSL is in “Available” state:Â
$ velero backup-location get
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT
test-velero-irsa-bucket aws test-velero-irsa-bucket Available 2023-02-28 14:48:54 +0000 UTC ReadWrite true
Run a Backup Using Velero (Optional)Â
- Run the following command to run a Velero backup of the cluster:Â
$ velero backup create test-backup Backup request "test-backup" submitted successfully. Run `velero backup describe test-backup` or `velero backup logs test-backup` for more details.
2. Wait for the Velero backup to be in completed state:Â
$ velero backup get test-backup NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR test-backup Completed 0 0 2023-02-28 14:52:02 +0000 UTC 29d test-velero-irsa-bucket <none>
3. Confirm that the Velero backup exists in the S3 bucket:
$ aws s3api list-objects --bucket $BUCKET
Conclusion
As can be seen, it is possible to configure Amazon S3 backup storage locations in Velero in a secure way using IRSA. CloudCasa, our Velero-compatible Kubernetes backup service, utilizes IRSA natively to integrate with your AWS environments to perform EKS backup and recovery. CloudCasa removes the setup and maintenance burden by providing a predefined CloudFormation template (CFT) when you add an AWS account. CloudCasa has been utilizing this method for almost two years and it is field tested on hundreds of EKS clusters. In addition to the powerful backup and recovery features built within Velero, CloudCasa utilizes IRSA to allow you to compose an entire EKS cluster from your backups – you can recover your EKS clusters across regions, recover EKS clusters cross-accounts and recover EKS clusters cross-clouds providers from AKS and GKE as well.
Sign up for the free service plan and give it a try!