Kubernetes Cluster Backup & Restore on S3 Bucket using Velero

Introduction

Someone deletes a namespace by accident. A migration goes sideways and you need last week’s cluster state back, not just today’s. A node group gets torn down before anyone thought to check what was running on it. Kubernetes does not protect you from any of this by default etcd holds your cluster’s state, but a raw etcd snapshot backs up everything or nothing, and restoring one is not something you want to be doing for the first time during an actual outage.

This is where Velero comes in. It is an open-source backup, recovery and migration tool built for Kubernetes and it does the one thing native tools like etcdctl and etcdutl cannot: back up and restore at the namespace or label level and store that backup somewhere durable away from the cluster itself.

This walkthrough covers what Velero does differently how to set it up against an AWS S3 bucket and how to run a backup and restore end to end.

What Velero Does That Native Tools Do Not

etcdctl and etcdutl snapshot the entire cluster state. That is useful, but blunt — there is no way to back up just one namespace, or just the resources carrying a specific label, and no built-in way to ship that snapshot off to cloud storage. Velero fixes both problems. It backs up Kubernetes resources and persistent volumes at whatever scope you choose, and pushes the result to an object store like S3, so a cluster failure does not also take out your only copy of the backup.

Velero runs as two separate pieces – first, there’s the client (a CLI installed on the user’s workstation), second there’s the server running on the control plane in the cluster that manages the communication with your cloud provider.

Requirements:

• A running Kubernetes cluster with kubectl.
• Access to a cloud account. This walkthrough uses AWS.
• An AWS CLI configured on the system of the user performing this walkthrough, with appropriate permissions to create S3 Buckets and IAM users.

Setting Up the S3 Backend

Start with the bucket:

aws s3api create-bucket \
  --bucket your-cluster-backups \
  --region ap-south-1 \
  --create-bucket-configuration LocationConstraint=ap-south-1

Enable versioning for the bucket, so that we can restore old versions of the data –

aws s3api put-bucket-versioning \
  --bucket your-cluster-backups \
  --versioning-configuration Status=Enabled

Leave the bucket’s public access block turned on as this bucket holds full cluster state, and there is no reason for any of that to be reachable outside your account.

You should not assign the administrative rights of your cloud provider to Velero, since you should just give it restricted permissions. Instead, creating a new IAM user will be prudent. Assign GetObject, PutObject, DeleteObject, ListObject privileges to this IAM user and also the necessary access for EC2 snapshots if it’s your intended purpose to backup PVs. After a user has been created, make an access key and pair, you can store these details within a file called credentials-velero, like what’s shown below:

[default]
aws_access_key_id=<ACCESS_KEY>
aws_secret_access_key=<SECRET_KEY>

Installing Velero

Install the Velero client by downloading the correct zip file for your OS from the latest release of Velero,uncompress and move it to your $PATH, use below commands for doing so:

Link for the release – https://github.com/velero-io/velero/releases/

tar -xvf velero-*-linux-amd64.tar.gz
sudo mv velero-*-linux-amd64/velero /usr/local/bin/

The server install happens from the client, against your cluster:

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:latest \
  --bucket your-cluster-backups \
  --backup-location-config region=ap-south-1 \
  --secret-file ./credentials-velero

The flag worth calling out is --plugins. Skip it, and the install fails outright, since Velero has no idea how to talk to S3 without the AWS provider plugin loaded. Once the install runs, check that it actually landed:

kubectl get namespaces
kubectl get pods -n velero

Give the pod a minute to reach Running. Once it does, Velero is live on the cluster.

Running a Backup and Restore

This is the part that matters, and it takes four commands:

1. Back up a namespace = velero backup create dev-backup --include-namespaces dev pushes everything in the dev namespace to S3.
2. Confirm it landed = velero backup describe dev-backup or a quick look in the S3 console shows the backup object sitting in the bucket.
3. Simulate the disaster = kubectl delete namespace dev will actually destroy your entire ‘dev’ namespace and everything running in it, on purpose, to prove the recovery path actually works.
4. Restore it = velero restore create --from-backup dev-backup pulls the backup back from S3 and rebuilds the namespaceand all its contents, as it was when backed up.

A minute or two later, kubectl get ns shows the namespace back, and with all its previously running workloads!

Conclusion

Backups which are never tested are useless. That’s why the most critical step above-actually deleting a namespace, and then recovering it-is not something many users skip. With that step accomplished, any team running Kubernetes in multiple locations can avoid major headaches, and turn what could be several hours of outage into merely an annoying inconvenience.