Velero is the most popular tool for backing up and restoring Kubernetes cluster resources and persistent volumes. However, there may be situations where you need to restore Velero backup data without using Velero itself. For example, if Velero is not installed and configured correctly, or if more fine-grained restore control is required. In this post, we will explore how to do this when either Restic or Kopia was used by Velero to store the persistent volume (PV) data. We will be focusing on File System Backup using Velero version 1.11.0.
Restore Velero Backup Data with Restic
Restic is a backup program that can be used to back up and restore data. Velero can use Restic as an underlying tool to back up and restore persistent volume data. Let’s take a look at how we can use Restic to restore Velero backup data.
First, we need to identify the PVs that were backed up. In our test environment, we have 3 PVs, one is in test1 namespace and two in test2 namespace. We can use “kubectl get pvc –A” command to list PVCs in those namespaces:
$ kubectl get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE test1 mysql-data-disk1 Bound pvc-a0a58f73-4d7f-47d4-b92d-aa2a5642f7bd 1Gi RWO csi-hostpath-sc 6m25s test2 mysql-data-disk2 Bound pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47 1Gi RWO csi-hostpath-sc 6m42s test2 mysql-data-disk3 Bound pvc-0dd5e766-f358-419f-a214-7e2b85368709 1Gi RWO csi-hostpath-sc 6m42s
Next, we will run a couple of backups using the Restic uploader and see which objects are created on the target. We can do this by running “velero backup create” command with “–default-volumes-to-fs-backup” flag and specifying the namespaces to include in the backup:
$ velero backup create backup1 --default-volumes-to-fs-backup --include-namespaces test1,test2 $ velero backup create backup2 --default-volumes-to-fs-backup --include-namespaces test1,test2
On the target, we can see that a Restic repository was created for each namespace in “restic/<NAMESPACE>” and that there are snapshots present at “restic/<NAMESPACE>/snapshots” prefix.
$ aws s3 ls --recursive s3://mybucket | grep restic | awk '{print $4}' restic/test1/config ... restic/test1/snapshots/9c930f4d53ddd8c8ed4e119eeba8cb74fa90b3b49a1efb243f01d50d42aee174 restic/test1/snapshots/c51476a9678767089871616a2c98ed6a13c318ca4c0c4738e4df5329a0be1c7e ... restic/test2/config ... restic/test2/snapshots/13ab195139bbb634178d8acf6235fdb3026ef7749425f601ec01a0793da4972f restic/test2/snapshots/6aedee93c1fa2f7c7600292101353df7197ff9008bd552fb62429a0bd48a3a3b restic/test2/snapshots/bf4b58e1b60e02ea17550323447dd98e1ec60d12f173372481934b29bf86d610 restic/test2/snapshots/ce0c3c68c060c1e07c38d58d368804bf2389d7711048a5da288f1f3ab25d38ee
To get the list of snapshots for a specific namespace, we can use the Restic CLI. First, we need to set the AWS credentials to the ones that provide read-only access so we don’t inadvertently corrupt the repository. The repository password is by default set to “static-passw0rd” but can be changed. If “velero-repo-credentials” secret exists in the Velero namespace, get the repository password from “repository-password” key. We can then use the “restic snapshots” command to list the snapshots for the test2 namespace:
export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXX export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX $ cat restic-password static-passw0rd $ restic -p restic-password -r s3:s3.amazonaws.com/mybucket/restic/test2 snapshots repository f36a56ea opened (version 2, compression level auto) created new cache in /home/myuser/.cache/restic ID Time Host Tags Paths ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ce0c3c68 2023-07-13 16:08:29 velero backup=backup1,backup-uid=17d0cb28-c6fb-4849-bc34-614be8b02eab,ns=test2,pod=mysql-deployment2-776cc9bb8c-cbtt8,pod-uid=f21fb5fa-7f0a-41c3-b34b-7dace80555b1,pvc-uid=ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47,volume=mysql-data2 /host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount 13ab1951 2023-07-13 16:08:39 velero backup=backup1,backup-uid=17d0cb28-c6fb-4849-bc34-614be8b02eab,ns=test2,pod=mysql-deployment2-776cc9bb8c-cbtt8,pod-uid=f21fb5fa-7f0a-41c3-b34b-7dace80555b1,pvc-uid=0dd5e766-f358-419f-a214-7e2b85368709,volume=mysql-data3 /host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-0dd5e766-f358-419f-a214-7e2b85368709/mount bf4b58e1 2023-07-13 16:56:35 velero backup=backup2,backup-uid=cecaf1af-8cd0-4a86-9cbc-e6c6812bc668,ns=test2,pod=mysql-deployment2 -776cc9bb8c-cbtt8,pod-uid=f21fb5fa-7f0a-41c3-b34b-7dace80555b1,pvc-uid=ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47,volume=mysql-data2 /host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount 6aedee93 2023-07-13 16:56:39 velero pod-uid=f21fb5fa-7f0a-41c3-b34b-7dace80555b1,pvc-uid=0dd5e766-f358-419f-a214-7e2b85368709,volume=mysql-data3,backup=backup2,backup-uid=cecaf1af-8cd0-4a86-9cbc-e6c6812bc668,ns=test2,pod=mysql-deployment2-776cc9bb8c-cbtt8 /host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-0dd5e766-f358-419f-a214-7e2b85368709/mount
For each entry in the output, we can see the Velero backup name, PVC, and pod information. To restore one of these backups, we can use the “restic restore“ command, specifying the snapshot ID and target directory:
$ restic -p restic-password -r s3:s3.amazonaws.com/mybucket/restic/test2 restore ce0c3c68 --target restic_restore
repository f36a56ea opened (version 2, compression level auto)
restoring <Snapshot ce0c3c68 of [/host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount] at 2023-07-13 20:08:29.888414931 +0000 UTC by root@velero> to restic_restoreAfter running the restore command, we can verify that the files were correctly restored by listing the contents of the restore target directory:
$ cd host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount $ find . . ./mysql ./mysql/client-cert.pem ./mysql/ib_logfile1 ./mysql/ibdata1 ./mysql/mysql.sock ./mysql/server-cert.pem ./mysql/auto.cnf ./mysql/mysql ./mysql/mysql/time_zone_transition_type.frm ...
For more restore options see the Restic reference guide.
Restore Velero Backup Data with Kopia
Kopia is another tool that can be used to back up and restore data. Like Restic, Velero can use Kopia as an underlying tool to backup and restore persistent volume data. Let’s take a look at how we can use Kopia to restore Velero backup data.
First, we need to identify the objects created on the backup target when using Kopia as an uploader. Repeating the same test with Kopia produces files on the backup target with a similar structure as for Restic, with one repository per namespace:
2023-07-13 17:32:00 30 kopia/test1/kopia.blobcfg 2023-07-13 17:32:00 1075 kopia/test1/kopia.repository 2023-07-13 17:32:38 1638603 kopia/test1/p3cd8f09c4a983ccdcc46ee897a0df00f-sf6d58fcead1c49d511e 2023-07-13 17:32:22 21533338 kopia/test1/pad73ca1e8c11f292cabd9b485cecd2cc-sf6d58fcead1c49d511e ... 2023-07-13 17:32:40 30 kopia/test2/kopia.blobcfg 2023-07-13 17:32:40 1075 kopia/test2/kopia.repository 2023-07-13 17:32:44 24232396 kopia/test2/p0ddb521ca63b16cf927da6ec827e0bf1-sfcda7a530c4d9a0e11e 2023-07-13 17:33:01 21533338 kopia/test2/p39abaad7b2d64041a206a145339b6c7f-sfcda7a530c4d9a0e11e …
We can connect to the repository by using the Kopia CLI “repository connect“ command and providing our repository password. Make sure to use the read-only access-key so the repository does not accidentally get corrupted. Once connected, we can use the “kopia snapshot list” command to list the snapshots. PVC ID can be found as part of the source path:
$ kopia repository connect s3 --bucket=mybucket --access-key=XXXXXXXXXXXXX --secret-access-key=XXXXXXXXXXXXX --prefix=kopia/test1/ Enter password to open repository: static-passw0rd $ kopia snapshot list --all -l default@default:/host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-0dd5e766-f358-419f-a214-7e2b85368709/mount 2023-07-13 17:33:19 EDT k89d638fbdb3ac7ef7612f07a9da91e2e 0 B drwxr-xr-x files:0 dirs:2 (latest-2) 2023-07-13 17:48:00 EDT k89d638fbdb3ac7ef7612f07a9da91e2e 0 B drwxr-xr-x files:0 dirs:2 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1) default@default:/host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount 2023-07-13 17:32:43 EDT k2dc092f82bba74df45503df9d67ae208 219.5 MB drwxr-xr-x files:284 dirs:5 (latest-2) 2023-07-13 17:47:58 EDT k2dc092f82bba74df45503df9d67ae208 219.5 MB drwxr-xr-x files:284 dirs:5 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1)
To restore PV data from a snapshot, we can use the “kopia snapshot restore” command, specifying the snapshot ID and path we are restoring:
$ kopia snapshot restore k2dc092f82bba74df45503df9d67ae208 host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount Restoring to local filesystem (/home/myuser/kopia_restore/host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount) with parallelism=8... Processed 83 (193.5 MB) of 289 (219.5 MB) 191.5 MB/s (88.2%) remaining 0s. Processed 184 (219 MB) of 289 (219.5 MB) 108.5 MB/s (99.8%) remaining 0s. Processed 290 (219.5 MB) of 289 (219.5 MB) 63.4 MB/s (100.0%) remaining 0s. Processed 290 (219.5 MB) of 289 (219.5 MB) 63.2 MB/s (100.0%) remaining 0s. Restored 284 files, 5 directories and 1 symbolic links (219.5 MB).
After running the restore command, we can see that our files were successfully restored:
$ cd host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount $ find . . ./mysql ./mysql/client-cert.pem ./mysql/ib_logfile1 ./mysql/ibdata1 ./mysql/mysql.sock ./mysql/server-cert.pem ./mysql/auto.cnf ./mysql/mysql ./mysql/mysql/time_zone_transition_type.frm …
For more restore options see the Kopia reference guide.
Conclusion
By running these tests, we demonstrated that it is possible to restore PVC data from a Velero backup when either Restic or Kopia was used to store the PVC data. This workflow provides a data recovery alternative in situations where Velero may not be available or more fine-grained restore control is required. For safety, we always want to use read-only credentials when accessing data directly.
CloudCasa offers both Velero management as a service as well as affordable open source support for customers adopting Velero. We provide value to existing Velero users by offering services that enhance the user experience and accessibility of it, such as an easy-to-use UI, a guided recovery path, monitoring and alerting, and support for multi-cluster and multi-cloud environments.
If you use Velero for backups today, we’d love to demonstrate how we can make your life easier. Sign up for the CloudCasa for Velero plan to get both CloudCasa features and commercial support for Velero.
David Zaninovic
Nitish Sharma