Disaster Recovery
You can use SolidFire native replication (see Awesome SolidFire) to replicate Datastores and Volumes used by Rancher to a remote site. Both local and remote volumes can take scheduled and on-demand snapshots.
Rancher clusters can be scheduled to take scheduled and on-demand snapshots to local disk or S3 bucket. These snapshots protect data from etcd and can be used to restore etcd, Kubernetes version, and cluster configuration. Cluster must exist and be able to run a snapshot restore procedure.
Business Continuity for Storage
Assuming we stretch a Kubernetes cluster across two independent sites, each with its own SolidFire or NetApp HCI storage cluster, we don’t need to deal with Kubernetes, VMware or NetApp HCI storage (SolidFire) replication and failover (BC through Kubernetes, below)
If Rancher Kubernetes runs on one site and we want to fail it over to another site with same network configuration, we can take advantage of SolidFire SRA and VMware SRM to replicate VMs and Trident PVs to the other site, and fail over when required.
VMware SRM can fail-over to another site and bring up Rancher VMs, but we still need to expose Trident volumes to VMs (Rancher worker nodes). To do that we’d stop SolidFire replication for PVs and make it possible to access those volumes in a read-write fashion.
Trident CSI Volume Import Feature
Since Trident started to support Volume Import feature, it is possible - and in fact recommended - to import SolidFire replica volumes, rather than use search-and-replace of volume IQN strings.
This video shows how Kubernetes can failover to another SolidFire cluster in seconds. There’s a very detailed post about it here.
Pre-CSI Trident: Search-and-Replace for Trident Volumes
Storage-dependent procedure:
- Run a Search-and-Replace operation for Trident targets (paths) on PROD vs. DR site (this replaces Cluster UUID string from PROD with Cluster UUID from DR site and loads updated iSCSI target information)
- Stop-Reverse-Start (replication) sequence for storage replication (this stops PROD->DR replication, makes DR volumes Read-Write, and PROD volumes (if still reachable) Read-Only, and kicks off DR->PROD replication)
These aren’t currently documented specifically for the Rancher on HCI solution or even in the NetApp Trident documentation, but use SolidFire, Kuberetes and Trident features and therefore do not directly depend on Rancher itself. In theory we could use vSphere SRM and SolidFire SRA to failover VMs running Rancher, but that would be a complicated approach and I doubt we’ll see it officially recommended.
Awesome SolidFire has examples of how Target IQNs differ between clusters. YouTube has a demo of how PowerShell can be used to pair clusters, create replication relationship between two SolidFire storage clusters. It takes 30 seconds. YouTube also has a (longish, 15 min) video with a demonstration of SolidFire storage cluster failover and failback in a K8s/Trident environment.
BC through Kubernetes
As mentioned earlier, if services running on Kubernetes are designed and protected in a storage- and VMware-independent fashion, BC can be obtained independently of SolidFire and VMware.
Example: a Web site backed by a NOSQL cluster with Web and Database containers spread across multiple sites running indepdently NetApp HCI or SolidFire clusters.
This is a recommended approach because it features separation of concerns between infrastructure and services.
Backup and Restore of Kubernetes clusters, settings, applications and data (PVs)
Backup of clusters and cluster settings
Rancher provides its own “snapshot” feature that can backup cluster configuration to a local disk or S3 storage such as NetApp StorageGRID or Public Cloud Object Storage. For cluster configuration I would prefer that over VMware or SolidFire snapshots or backups.
Backup of Persistent Volumes (PVs) and applications
For cluster data, you could use application backups or a 3rd party backup software that can backup K8s containers (see Awesome SolidFire). Some - such as Kasten - integrate with CSI and also Kuberntes API to provide data and application recovery (see this post to get started).
Applications (Namespaces, Projects, Labels, etc.) should presumably be backed up with scheduled Rancher cluster snapshots, but if you want to take those in sync with PVs, then you either have two places to manage (one in Rancher snapshot settings, another in either SolidFire snapshot schedules, or backup application (Commvault BR, Kasten K10). For complex environments it makes sense to unify these workflows and manage them from a commercially supported application.