How to Restore Velero Backup Data Without Velero

Home BlogHow to Restore Velero Backup Data Without Velero
Visit Our Youtube Playlist for Velero

Velero is the most popular tool for backing up and restoring Kubernetes cluster resources and persistent volumes. However, there may be situations where you need to restore Velero backup data without using Velero itself. For example, if Velero is not installed and configured correctly, or if more fine-grained restore control is required. In this post, we will explore how to do this when either Restic or Kopia was used by Velero to store the persistent volume (PV) data. We will be focusing on File System Backup using Velero version 1.11.0.

Restore Velero Backup Data with Restic

Restic is a backup program that can be used to back up and restore data. Velero can use Restic as an underlying tool to back up and restore persistent volume data. Let’s take a look at how we can use Restic to restore Velero backup data.

First, we need to identify the PVs that were backed up. In our test environment, we have 3 PVs, one is in test1 namespace and two in test2 namespace. We can use “kubectl get pvc –A” command to list PVCs in those namespaces:

$ kubectl get pvc -A
NAMESPACE   NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE 
test1       mysql-data-disk1   Bound    pvc-a0a58f73-4d7f-47d4-b92d-aa2a5642f7bd   1Gi        RWO            csi-hostpath-sc   6m25s 
test2       mysql-data-disk2   Bound    pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47   1Gi        RWO            csi-hostpath-sc   6m42s 
test2       mysql-data-disk3   Bound    pvc-0dd5e766-f358-419f-a214-7e2b85368709   1Gi        RWO            csi-hostpath-sc   6m42s

Next, we will run a couple of backups using the Restic uploader and see which objects are created on the target. We can do this by running “velero backup create” command with “–default-volumes-to-fs-backup” flag and specifying the namespaces to include in the backup:

$ velero backup create backup1 --default-volumes-to-fs-backup --include-namespaces test1,test2
$ velero backup create backup2 --default-volumes-to-fs-backup --include-namespaces test1,test2

On the target, we can see that a Restic repository was created for each namespace in “restic/<NAMESPACE>” and that there are snapshots present at “restic/<NAMESPACE>/snapshots” prefix.

$ aws s3 ls --recursive s3://mybucket | grep restic | awk '{print $4}' 
restic/test1/config 
... 
restic/test1/snapshots/9c930f4d53ddd8c8ed4e119eeba8cb74fa90b3b49a1efb243f01d50d42aee174 
restic/test1/snapshots/c51476a9678767089871616a2c98ed6a13c318ca4c0c4738e4df5329a0be1c7e 
... 
restic/test2/config 
... 
restic/test2/snapshots/13ab195139bbb634178d8acf6235fdb3026ef7749425f601ec01a0793da4972f 
restic/test2/snapshots/6aedee93c1fa2f7c7600292101353df7197ff9008bd552fb62429a0bd48a3a3b 
restic/test2/snapshots/bf4b58e1b60e02ea17550323447dd98e1ec60d12f173372481934b29bf86d610 
restic/test2/snapshots/ce0c3c68c060c1e07c38d58d368804bf2389d7711048a5da288f1f3ab25d38ee 

To get the list of snapshots for a specific namespace, we can use the Restic CLI. First, we need to set the AWS credentials to the ones that provide read-only access so we don’t inadvertently corrupt the repository. The repository password is by default set to “static-passw0rd” but can be changed. If “velero-repo-credentials” secret exists in the Velero namespace, get the repository password from “repository-password” key. We can then use the “restic snapshots” command to list the snapshots for the test2 namespace:

export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXX 
export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

$ cat restic-password  
static-passw0rd 

$ restic -p restic-password -r s3:s3.amazonaws.com/mybucket/restic/test2 snapshots 
repository f36a56ea opened (version 2, compression level auto) 
created new cache in /home/myuser/.cache/restic 
ID        Time                 Host        Tags                                                                                                                                                                                                                         Paths 
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 
ce0c3c68  2023-07-13 16:08:29  velero      backup=backup1,backup-uid=17d0cb28-c6fb-4849-bc34-614be8b02eab,ns=test2,pod=mysql-deployment2-776cc9bb8c-cbtt8,pod-uid=f21fb5fa-7f0a-41c3-b34b-7dace80555b1,pvc-uid=ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47,volume=mysql-data2  /host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount 
13ab1951  2023-07-13 16:08:39  velero      backup=backup1,backup-uid=17d0cb28-c6fb-4849-bc34-614be8b02eab,ns=test2,pod=mysql-deployment2-776cc9bb8c-cbtt8,pod-uid=f21fb5fa-7f0a-41c3-b34b-7dace80555b1,pvc-uid=0dd5e766-f358-419f-a214-7e2b85368709,volume=mysql-data3  /host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-0dd5e766-f358-419f-a214-7e2b85368709/mount 
bf4b58e1  2023-07-13 16:56:35  velero      backup=backup2,backup-uid=cecaf1af-8cd0-4a86-9cbc-e6c6812bc668,ns=test2,pod=mysql-deployment2 
-776cc9bb8c-cbtt8,pod-uid=f21fb5fa-7f0a-41c3-b34b-7dace80555b1,pvc-uid=ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47,volume=mysql-data2  /host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount 
6aedee93  2023-07-13 16:56:39  velero      pod-uid=f21fb5fa-7f0a-41c3-b34b-7dace80555b1,pvc-uid=0dd5e766-f358-419f-a214-7e2b85368709,volume=mysql-data3,backup=backup2,backup-uid=cecaf1af-8cd0-4a86-9cbc-e6c6812bc668,ns=test2,pod=mysql-deployment2-776cc9bb8c-cbtt8  /host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-0dd5e766-f358-419f-a214-7e2b85368709/mount 

For each entry in the output, we can see the Velero backup name, PVC, and pod information. To restore one of these backups, we can use the “restic restore“ command, specifying the snapshot ID and target directory:

$ restic -p restic-password -r s3:s3.amazonaws.com/mybucket/restic/test2 restore ce0c3c68 --target restic_restore
repository f36a56ea opened (version 2, compression level auto)
restoring <Snapshot ce0c3c68 of [/host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount] at 2023-07-13 20:08:29.888414931 +0000 UTC by root@velero> to restic_restore

After running the restore command, we can verify that the files were correctly restored by listing the contents of the restore target directory:

$ cd host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount
$ find .
.
./mysql
./mysql/client-cert.pem
./mysql/ib_logfile1
./mysql/ibdata1
./mysql/mysql.sock
./mysql/server-cert.pem
./mysql/auto.cnf
./mysql/mysql
./mysql/mysql/time_zone_transition_type.frm
...

For more restore options see the Restic reference guide.

Restore Velero Backup Data with Kopia

Kopia is another tool that can be used to back up and restore data. Like Restic, Velero can use Kopia as an underlying tool to backup and restore persistent volume data. Let’s take a look at how we can use Kopia to restore Velero backup data.

First, we need to identify the objects created on the backup target when using Kopia as an uploader. Repeating the same test with Kopia produces files on the backup target with a similar structure as for Restic, with one repository per namespace:

2023-07-13 17:32:00         30 kopia/test1/kopia.blobcfg 
2023-07-13 17:32:00       1075 kopia/test1/kopia.repository 
2023-07-13 17:32:38    1638603 kopia/test1/p3cd8f09c4a983ccdcc46ee897a0df00f-sf6d58fcead1c49d511e 
2023-07-13 17:32:22   21533338 kopia/test1/pad73ca1e8c11f292cabd9b485cecd2cc-sf6d58fcead1c49d511e 
... 
2023-07-13 17:32:40         30 kopia/test2/kopia.blobcfg 
2023-07-13 17:32:40       1075 kopia/test2/kopia.repository 
2023-07-13 17:32:44   24232396 kopia/test2/p0ddb521ca63b16cf927da6ec827e0bf1-sfcda7a530c4d9a0e11e 
2023-07-13 17:33:01   21533338 kopia/test2/p39abaad7b2d64041a206a145339b6c7f-sfcda7a530c4d9a0e11e 
… 
 

We can connect to the repository by using the Kopia CLI “repository connect“ command and providing our repository password. Make sure to use the read-only access-key so the repository does not accidentally get corrupted. Once connected, we can use the “kopia snapshot list” command to list the snapshots. PVC ID can be found as part of the source path:

$ kopia repository connect s3 --bucket=mybucket --access-key=XXXXXXXXXXXXX --secret-access-key=XXXXXXXXXXXXX --prefix=kopia/test1/ 
Enter password to open repository: static-passw0rd 
  $ kopia snapshot list --all -l 
default@default:/host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-0dd5e766-f358-419f-a214-7e2b85368709/mount 
  2023-07-13 17:33:19 EDT k89d638fbdb3ac7ef7612f07a9da91e2e 0 B drwxr-xr-x files:0 dirs:2 (latest-2) 
  2023-07-13 17:48:00 EDT k89d638fbdb3ac7ef7612f07a9da91e2e 0 B drwxr-xr-x files:0 dirs:2 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1)
default@default:/host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount 
  2023-07-13 17:32:43 EDT k2dc092f82bba74df45503df9d67ae208 219.5 MB drwxr-xr-x files:284 dirs:5 (latest-2) 
  2023-07-13 17:47:58 EDT k2dc092f82bba74df45503df9d67ae208 219.5 MB drwxr-xr-x files:284 dirs:5 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1)  

To restore PV data from a snapshot, we can use the “kopia snapshot restore” command, specifying the snapshot ID and path we are restoring:

$ kopia snapshot restore k2dc092f82bba74df45503df9d67ae208 host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount 
Restoring to local filesystem (/home/myuser/kopia_restore/host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount) with parallelism=8... 
Processed 83 (193.5 MB) of 289 (219.5 MB) 191.5 MB/s (88.2%) remaining 0s. 
Processed 184 (219 MB) of 289 (219.5 MB) 108.5 MB/s (99.8%) remaining 0s. 
Processed 290 (219.5 MB) of 289 (219.5 MB) 63.4 MB/s (100.0%) remaining 0s. 
Processed 290 (219.5 MB) of 289 (219.5 MB) 63.2 MB/s (100.0%) remaining 0s. 
Restored 284 files, 5 directories and 1 symbolic links (219.5 MB). 

After running the restore command, we can see that our files were successfully restored:

$ cd host_pods/f21fb5fa-7f0a-41c3-b34b-7dace80555b1/volumes/kubernetes.io~csi/pvc-ae0b9b84-f9dd-4c57-8732-ab8ecb6aaa47/mount
$ find .
.
./mysql
./mysql/client-cert.pem
./mysql/ib_logfile1
./mysql/ibdata1
./mysql/mysql.sock
./mysql/server-cert.pem
./mysql/auto.cnf
./mysql/mysql
./mysql/mysql/time_zone_transition_type.frm
…

For more restore options see the Kopia reference guide.

Conclusion

By running these tests, we demonstrated that it is possible to restore PVC data from a Velero backup when either Restic or Kopia was used to store the PVC data. This workflow provides a data recovery alternative in situations where Velero may not be available or more fine-grained restore control is required. For safety, we always want to use read-only credentials when accessing data directly.

CloudCasa offers both Velero management as a service as well as affordable open source support for customers adopting Velero. We provide value to existing Velero users by offering services that enhance the user experience and accessibility of it, such as an easy-to-use UI, a guided recovery path, monitoring and alerting, and support for multi-cluster and multi-cloud environments.

If you use Velero for backups today, we’d love to demonstrate how we can make your life easier. Sign up for the CloudCasa for Velero plan to get both CloudCasa features and commercial support for Velero.