Open source Velero is a popular choice amongst Kubernetes community for backup of their application. Along with scheduling regular backups for your Kubernetes clusters, monitoring Velero and getting automated alerts for failed Velero backups is also important. When your most important backup fails, you want to know about it immediately and make sure your application notifies you. Configuring alerts on your application may sometimes seem too cumbersome and understanding the cause of failure is time consuming.
CloudCasa for Velero solves this major challenge around observability when using open source Velero for backup and restore of Kubernetes clusters. When monitoring Velero job execution manually, one needs to keep a close look at the log file, which is a tedious task. Velero being an open-source tool to back up and restore your Kubernetes cluster resources and persistent volumes does not provide a backup notification mechanism. There are multiple reasons for backup failures like configuration issues, pod failures, destination storage inaccessible or invalid cloud credentials etc. Alerting and monitoring Velero backups helps you detect errors early, before they affect your users. It makes troubleshooting easier by providing comprehensive visibility over cluster activity
In this blog, we will show how CloudCasa for Velero helps configure email notifications for backups and restores job failures. You will also see an example of how easy it is to monitor the job updates and alerts for Velero backups in the CloudCasa UI interface.
Setting Up the Environment for Monitoring Velero Backups
Assumption here is you have a Kubernetes cluster with Velero deployment and CloudCasa agent installed on the cluster. If you don’t have these things running, then follow the sections below to get a minimal setup running on top of minikube with minio as S3-like Object Storage.
1. Creation of Kubernetes Cluster on Minikube
- Install minikube following the instructions in the link https://minikube.sigs.k8s.io/docs/start/
- Start the minikube cluster with the command “minikube start”
$ minikube start -p testemail --driver=docker --cpus='4' --nodes=3 --memory='4096' --kubernetes-version=v1.26
2. Velero Deployment on Minikube Clusters
- Download the latest release’s tarball for your client platform.
- Extract the tarball: tar -xvf <RELEASE-TARBALL-NAME>.tar.gz
- Move the extracted velero binary to $PATH (ex: /usr/local/bin)
$ wget https://github.com/vmware-tanzu/velero/releases/download/v1.11.0/velero-v1.11.0-linux-amd64.tar.gz
$ tar -xvf https://github.com/vmware-tanzu/velero/releases/download/v1.11.0/velero-v1.11.0-linux-amd64.tar.gz
$ sudo mv velero-v1.11.0-linux-amd64/velero /usr/bin/velero
- Create a credentials-velero file that we will use during installation to authenticate against the Minio service. Velero will use these credentials to access Minio to store volume backups:
- To install Velero in the Kubernetes Cluster using Minio as S3-like Object Storage use the following command.
$ velero install --provider aws --plugins velero/velero-plugin-for-aws --bucket <bucketname> --secret-file ./credentials-velero --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://<IP>:9000 --use-volume-snapshots=true --uploader-type restic --use-node-agent true --snapshot-location-config region="default"
Velero is installed! Use “kubectl logs deployment/velero -n velero” to view the status
Note: For any queries regarding Velero installation look at the https://github.com/vmware- tanzu/helm-charts/blob/main/charts/velero/README.md
3. Quick Installation of CloudCasa on the Velero Deployed Kubernetes Cluster
Before we configure email notification and monitor the Velero Clusters, we need to install the CloudCasa agent on the cluster.
- From the menu bar, go to “Configuration” > “Clusters” in the General section.
- Click “Add Cluster +” to open the Add cluster pane.
- Fill in the required fields
- Enable to option “Manage an existing Velero instance”
- Choose Advanced options and enter the namespace in “Velero namespace to manage” if Velero is deployed in any other namespace other than the default namespace ‘velero’.
6. After installing the agent, verify that the cluster is shown as Active in the CloudCasa UI before proceeding.
4. Configure the Email Notifications.
In the CloudCasa UI –
- From any page, Click on the “User Menu” (≡), to access additional option
- Click on “User Settings”
- Enable “Send alerts via email” and save the setting.
The setup is all done and ready to test.
Note: The alerts email notification will be sent for all the registered clusters. In case one chooses to discontinue the service, the user must disable the “Send alerts via email” setting.
Testing the Alerts Raised during Velero Backup Failure
When activated, CloudCasa alerts backup/restore failures, cluster reachability warnings, product updates, skipped or partially successful jobs.
In this example, to simulate the alert situation, we have triggered a Velero backup job of a namespace to to a storage location which is “unavailable”.
#Velero command to create backup using a particular namespace and specific backup location is as shown below
$ velero create backup testbkpemail --include-namespaces=test1 --storage-location test
$ velero get backup-location.
The backup should fail immediately.
1. Monitoring Velero Job Status in CloudCasa UI
Since CloudCasa UI now helps monitor Velero jobs, we can see the job status in the CloudCasa à Dashboard à “Activity Tab” and accordingly the alert for the job failure is seen in the “Alerts” tab.
2. Email Alerts for Velero Backup and Restore Failures
To validate the email notification of the failure, the inbox of the signed-up user will have an email sent out immediately as seen below.
Improve Visibility into your Velero Operations
In this blog, we discussed how easy CloudCasa has made alerting on job failures and monitoring of Velero backups and restores for your Kubernetes clusters. Monitoring and alerting facilitate better workload management and more optimal use of your resources.
To improve visibility and have a centralized view of your Velero operations, you can simply Signup and install the CloudCasa agent on your Kubernetes clusters running Velero. CloudCasa will start monitoring Velero jobs without altering any existing configurations and send email alerts for Velero backup failures. Hope you found this blog on monitoring Velero backups and alerting backup failures using CloudCasa useful. If you have any questions, feel free to reach out to us. For regular updates from CloudCasa, subscribe to our newsletter.
Martin Phan