Skip to content
Hands-on Kubernetes & DevOpsPart 6 of 6 — open for all chapters

7 min readPart 6 of 6

Kubernetes Disaster Recovery with Velero: Backup, Restore, and Measuring RTO

A deleted namespace, a broken deployment, a full cluster wipe — here's how to recover from all three in under 15 minutes, and how to prove it before disaster strikes.

On this page

A deleted namespace, a broken deployment, a full cluster wipe — here's how to recover from all three in under 15 minutes, and how to prove it before disaster strikes.


Why DR Practice Matters More Than DR Planning#

Most teams have a disaster recovery plan. Few have actually tested it.

The difference becomes clear at 3am when someone runs kubectl delete namespace prod thinking they're on the staging cluster. A plan sitting in Confluence doesn't help. Muscle memory from having done the restore three times in a lab does.

Velero is the industry standard tool for Kubernetes backup and restore. It snapshots your cluster resources — Deployments, Services, ConfigMaps, Secrets, PersistentVolumes — to object storage, and lets you restore them to any cluster in minutes.

This post covers the full DR cycle: backup configuration, scheduled backups, and three realistic disaster scenarios with measured recovery times.


RTO and RPO — Define These Before You Build#

Two metrics define your DR posture:

RTO (Recovery Time Objective) — how long can your system be down before it causes unacceptable business impact? "We must be back online within 15 minutes."

RPO (Recovery Point Objective) — how much data loss is acceptable? "We can lose at most 1 hour of changes."

Your backup frequency determines RPO. Hourly backups mean worst-case 59 minutes of data loss. Five-minute backups mean worst-case 4 minutes. The cost of more frequent backups (storage, network, compute) has to be weighed against the cost of losing that data.

Define both before building your backup system. They drive every configuration decision — backup frequency, retention period, restore target, and alerting thresholds.

For this project: RTO < 15 minutes, RPO < 1 hour.


Architecture#

code
Kubernetes cluster
    ↓
Velero (runs as a pod in the cluster)
    ↓ backs up resources every hour
AWS S3 bucket (encrypted, versioned)
    ↓
Disaster happens
    ↓
Velero restore ← points at same S3 bucket
    ↓
Cluster rebuilt from object storage

Velero has two components:

Velero server — runs in the cluster, watches for backup/restore requests, talks to the cloud provider API.

Node agent (formerly Restic) — runs on each node as a DaemonSet, handles PersistentVolume backup by reading data directly from the node's filesystem.


Setup: S3 and IAM#

Velero needs an S3 bucket and an IAM user with permissions to read/write it.

bash
AWS_REGION=ap-southeast-1
BUCKET_NAME=velero-backup-$(whoami)-$(date +%s)

aws s3api create-bucket \
  --bucket $BUCKET_NAME \
  --region $AWS_REGION \
  --create-bucket-configuration LocationConstraint=$AWS_REGION

# Versioning protects against accidental bucket-level deletion
aws s3api put-bucket-versioning \
  --bucket $BUCKET_NAME \
  --versioning-configuration Status=Enabled

Create a dedicated IAM user with least-privilege access — only the S3 and EC2 snapshot permissions Velero needs, scoped to the specific bucket:

bash
aws iam create-user --user-name velero

cat > velero-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeVolumes",
        "ec2:DescribeSnapshots",
        "ec2:CreateTags",
        "ec2:CreateVolume",
        "ec2:CreateSnapshot",
        "ec2:DeleteSnapshot"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:PutObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::${BUCKET_NAME}/*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::${BUCKET_NAME}"
    }
  ]
}
EOF

aws iam put-user-policy \
  --user-name velero \
  --policy-name velero-policy \
  --policy-document file://velero-policy.json

aws iam create-access-key --user-name velero

Install Velero on the cluster:

bash
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.8.0 \
  --bucket $BUCKET_NAME \
  --backup-location-config region=$AWS_REGION \
  --snapshot-location-config region=$AWS_REGION \
  --secret-file ./velero-credentials \
  --use-node-agent

Verify the backup location is reachable:

bash
velero backup-location get
# NAME      PROVIDER   BUCKET/PREFIX          PHASE
# default   aws        velero-backup-xxx      Available

Available means Velero can authenticate to S3 and read/write the bucket. Unavailable means check IAM credentials and bucket permissions.


Scheduled Backups#

One-off backups are for testing. Production needs schedules:

bash
# Prod namespace — hourly, keep 3 days
velero schedule create prod-hourly \
  --schedule="0 * * * *" \
  --include-namespaces prod \
  --ttl 72h

# Full cluster — daily at midnight, keep 7 days
velero schedule create full-daily \
  --schedule="0 0 * * *" \
  --include-namespaces dev,prod \
  --ttl 168h

The ttl (time to live) is critical. Without it, backups accumulate forever and your S3 costs grow unbounded. With it, Velero automatically deletes backups older than the TTL.

Verify schedules are running:

bash
velero schedule get
velero backup get   # shows backups created by schedules

Disaster Scenario 1: Accidental Namespace Deletion#

This is the most common production disaster. Someone runs a kubectl delete command on the wrong cluster context.

Create the disaster:

bash
kubectl delete namespace prod
kubectl get namespace prod
# Error: namespaces "prod" not found

Everything in prod — Deployments, Services, ConfigMaps, running pods — is gone. Users are getting connection errors.

Measure time from this point.

Restore:

bash
velero restore create prod-restore-v1 \
  --from-backup prod-backup-v1 \
  --wait

kubectl get all -n prod --watch

Watch resources reappear: ConfigMap → Service → Deployment → ReplicaSet → Pods. Within 3-5 minutes, pods are Running and serving traffic.

Measure time to restoration.

In this test: RTO = 4 minutes 23 seconds. Well within the 15-minute target.


Disaster Scenario 2: Broken Deployment#

A bad image tag gets pushed to production. Pods are in ImagePullBackOff. Rolling back via kubectl rollout undo isn't an option because the issue is in the registry, not the previous rollout.

Create the disaster:

bash
kubectl set image deployment/my-api-prod \
  my-api-prod=nginx:this-tag-does-not-exist \
  -n prod

kubectl get pods -n prod
# ImagePullBackOff on new pods, old pods terminated

Restore just the Deployment — surgical restore:

bash
velero restore create deployment-restore \
  --from-backup prod-backup-v1 \
  --include-resources deployments \
  --include-namespaces prod \
  --wait

kubectl get pods -n prod --watch
# Pods recovering with correct image from backup

This is the power of resource-scoped restores. You don't have to restore the entire namespace — just the specific resource that's broken. Services and ConfigMaps remain untouched.


Disaster Scenario 3: Full Cluster Loss#

The worst case. The cluster itself is gone — corrupted etcd, accidental deletion, cloud provider incident.

Simulate:

bash
kubectl delete namespace dev prod
kubectl get pods --all-namespaces
# Only kube-system and velero remain

Restore everything:

bash
velero restore create full-restore \
  --from-backup full-cluster-backup \
  --wait

kubectl get pods -n dev
kubectl get pods -n prod

Both namespaces restored from S3. On a real cluster loss, you'd provision a new EKS cluster first (~10 minutes), install Velero pointing at the same S3 bucket, then run this restore. The S3 bucket is the source of truth — the cluster is disposable.

Total RTO for full cluster rebuild: ~15 minutes (10 min EKS provisioning + 5 min restore).


The DR Runbook#

The restore commands above need to be written down and accessible before the cluster is gone. A DR runbook in Git (not in the cluster) is the right place:

markdown
# Disaster Recovery Runbook

## RTO/RPO Targets
- RTO: < 15 minutes
- RPO: < 1 hour

## Scenario 1: Namespace Deleted
velero restore create <name>-restore \
  --from-backup <latest-backup> --wait

## Scenario 2: Broken Deployment
velero restore create fix \
  --from-backup <latest-backup> \
  --include-resources deployments \
  --include-namespaces prod --wait

## Scenario 3: Full Cluster Loss
1. Provision new EKS cluster
2. Install Velero pointing at same S3 bucket
3. velero restore create full --from-backup <backup> --wait

The runbook is only useful if it's accessible when the cluster isn't. Store it in GitHub, not in a ConfigMap.


What Velero Doesn't Back Up By Default#

Cluster-scoped resources — Namespaces, ClusterRoles, PersistentVolumes (the definition, not the data) are included. But some custom cluster-scoped resources may need explicit inclusion.

Secrets — Velero backs up Kubernetes Secret objects, but they're only base64 encoded in etcd. If you're using Vault, the secrets aren't in Kubernetes at all — Velero won't back them up because they don't exist there. Your Vault backup strategy is separate.

External state — Velero backs up Kubernetes resources, not your database. RDS, ElastiCache, DynamoDB — these have their own backup mechanisms (automated snapshots, point-in-time recovery). Velero + RDS automated backups together give you a complete DR story.

The cluster itself — Velero can't restore a cluster that doesn't exist. The control plane, node groups, and VPC need to be provisioned before Velero can restore workloads into them. Terraform handles this — your infrastructure code is what lets you rebuild the cluster itself.


Velero Quick Reference#

bash
# List all backups
velero backup get

# Create a manual backup
velero backup create manual-$(date +%Y%m%d-%H%M) \
  --include-namespaces prod --wait

# Describe a backup (shows what was included)
velero backup describe <backup-name>

# List all restores
velero restore get

# Restore from backup
velero restore create <restore-name> \
  --from-backup <backup-name> --wait

# Restore specific resources only
velero restore create <name> \
  --from-backup <backup> \
  --include-resources deployments,services \
  --include-namespaces prod --wait

# Check backup location health
velero backup-location get

# Delete a backup manually
velero backup delete <backup-name>

Key Takeaways#

  • Define RTO and RPO before building — they drive every configuration decision
  • Test your restore process before you need it — untested DR is no DR
  • Scheduled backups with TTL prevent unbounded S3 cost growth
  • Surgical restores (specific resources only) are faster and safer than full namespace restores
  • The DR runbook must be stored outside the cluster — GitHub, not a ConfigMap
  • Velero backs up Kubernetes resources, not external state — RDS and databases need separate backup strategies
  • On full cluster loss, Terraform rebuilds the cluster, Velero restores the workloads — they complement each other

Part of a hands-on DevOps learning series. Code at github.com/kaungmyathan18/golang-k8s-portfolio.