Kubernetes backup, recovery, and disaster recovery for RAG and inference workloads
A production AI service accumulates data that your team needs to recover: documents, embeddings, vector indexes, application settings, and conversation history. Once a retrieval-augmented generation (RAG) pipeline connects a model to your organization’s information, those dependencies become part of the service you operate.
SUSE and CloudCasa by Catalogic are working together to protect that state in SUSE AI Factory. The partnership combines SUSE’s Kubernetes-based AI platform with CloudCasa’s backup, restore, workload migration, and disaster recovery capabilities. The goal is a recovery process that your platform team can configure, test, and run using the same tools it uses for other stateful Kubernetes applications.
The SUSE and CloudCasa partnership
Our collaboration with SUSE now includes work at the Certified for SUSE AI Factory certification level, focused on protecting AI applications running on the platform. It builds on CloudCasa’s integration with SUSE Rancher Prime and the work the two teams have already done on Kubernetes backup and recovery.
CloudCasa integrates at the Kubernetes platform layer. When you deploy an application from an existing SUSE AI Factory blueprint, CloudCasa can protect its Kubernetes resources and persistent data, including the state behind RAG-enabled chatbots. You choose which resources and data to protect and where to store the recovery copies.
For administrators, the practical questions are straightforward: which resources and volumes belong to the application, how to capture their state consistently, and what needs to be ready when you restore them. Those questions define the protection workflow below.
Identify the state behind the AI service
SUSE AI Factory’s preconfigured RAG chatbot blueprint includes vLLM, Open WebUI, and Milvus. The application has several recovery dependencies:
- Model serving: serving configuration and any locally stored model weights, adapters, or checkpoints that you need to retain.
- Vector data: embeddings, indexes, and the metadata the vector database uses to find them.
- Object data: documents and the underlying objects used by the vector store. In our reference setup, MinIO holds the object data and etcd holds Milvus metadata.
- User and application state: accounts, settings, chats, and notes, including any separate database or identity service that the application depends on.
- Kubernetes configuration: workloads, Services, ConfigMaps, Secrets, persistent volume claims, and required custom resources.
Map these dependencies before defining the backup. Your deployment may place them in several namespaces or use services outside the cluster. If an object store is external, include its protection and recovery in the plan explicitly. A backup of the Kubernetes application does not by itself establish that the external bucket has been protected.
This inventory also helps you decide what is worth preserving. Restoring saved embeddings and their associated data can avoid repeating document ingestion and embedding work. The value depends on the size of the corpus, the cost of rebuilding it, and whether the original model and processing configuration remain available.
How CloudCasa protects applications deployed from blueprints
Register the cluster and define the backup scope
The CloudCasa extension for SUSE Rancher lets administrators view protection status and manage backups from the Rancher dashboard. Configure the extension for your CloudCasa service or self-hosted deployment, then verify that the agent is connected on each cluster you want to protect.
Create a backup definition that includes the AI application’s namespaces, required Kubernetes resources, and persistent volumes. CloudCasa supports namespace, resource, and label-based selection. Check the selected volumes against your dependency inventory so the vector store, its metadata, and its object data are all covered. The Kubernetes backup guide explains the available selections and scheduling options.
Capture a recoverable application state
In our reference setup, CloudCasa uses Container Storage Interface (CSI) snapshots with SUSE Storage, based on Longhorn, and copies the protected data to backup storage under a retention policy.
Consistency needs attention when an application writes to several stores. Configure and test the database-specific preparation required by your deployment, including any coordinated pause, flush, or recovery steps. CloudCasa’s application hooks can run commands before and after backup and after restore. Their configuration needs to match the application and its versions.
After the first backup, review the job results and confirm that all expected resources and volumes have a recovery point. A completed snapshot is only one part of that check; the restore test establishes whether the captured application state is usable.
Keep recovery copies available after an outage
A snapshot on production storage shares that storage’s failure risks. Copy backups to a target that remains available if the production cluster or storage system is lost. Choose retention, location, and access controls around the incidents you need to recover from.
For supported S3-compatible targets, configure and verify Object Lock retention on the storage. CloudCasa SafeLock separately protects recovery-point metadata in the catalog. Check that both layers preserve your recovery copies for the intended period.
Keep the recovery credentials and runbook accessible during an outage, and test access to the backup target from the recovery environment. These details become important when the cluster that normally hosts your tools is unavailable.
Restore and check the service
CloudCasa can restore selected namespaces, resources, and volumes to the original cluster or an alternate cluster. Restore options include namespace changes and storage-class mapping, which are useful for recovery rehearsals, migrations, and infrastructure refreshes.
Prepare the destination first. It needs compatible Kubernetes components, storage, network access, and enough compute capacity for the workload. An inference service that requires a GPU also needs suitable GPU capacity and its runtime on the destination.
Then test the application itself. Sign in to the chat interface, check restored conversation data, and run retrieval against known documents. Verify that the vector store can read its metadata and object data and that inference completes successfully. Record the recovery point used and the elapsed time to a usable service.
Choose a recovery method around your requirements
Backup-based recovery gives you retained recovery points and the ability to restore to another cluster. Recovery time depends on data volume, storage and network throughput, available infrastructure, and application startup.
For workloads with tighter recovery objectives, CloudCasa DR can coordinate recovery using supported storage replication, including SUSE Storage’s Longhorn disaster recovery volumes. This requires configured source and target clusters, replicated application volumes, a replicated resource-inventory volume, and a tested disaster recovery plan.
Keep retained backups alongside the replication workflow so you can recover an earlier application state when needed. Set your recovery point objective, the acceptable data-loss window, and recovery time objective, the acceptable downtime. Then test both approaches against those requirements.
Keep recovery data under your control
Sovereign AI planning includes the location and operation of recovery copies. CloudCasa’s self-hosted deployment lets you run the control plane and backup catalog on infrastructure you manage. Agents write backup data directly to your selected storage, and the self-hosted service does not require a connection to CloudCasa SaaS.
That gives you control over the placement of the service, backup storage, and recovery clusters. Align each location with your organization’s residency and security requirements, and apply the same care to access permissions and retention. Air-gapped deployments also need a plan for local images, updates, and access to backup targets.
Retain backup job records and the results of restore exercises. Together with a documented runbook, they help your operations and compliance teams review what was protected, what was recovered, and how the recovery was tested.
Start with one AI workload and a restore test
Choose the RAG application that would be most disruptive to rebuild. Map its dependencies, configure a backup that includes its persistent data, and restore it into an isolated recovery environment. Use what you learn to set schedules, retention, and recovery procedures before extending protection to more workloads.
The SUSE and CloudCasa partnership gives platform teams a way to include AI data protection in their existing Kubernetes operations. The useful outcome is a repeatable recovery process with clear coverage and tested results.
Learn more about the solution in our datasheet or talk to the CloudCasa team about protecting your SUSE AI Factory deployment or evaluating CloudCasa self-hosted.