Kubernetes Disaster Recovery with Velero


main

Velero is an open source tool to safety back up, recover, and migrate Kubernetes clusters and persistent volumes. It works both on premises and in public clouds.

Velero components are:

  • Command line interface
  • Kubernetes server application

Velero is a convenient backup tool for Kubernetes clusters that compresses and backs up Kubernetes objects to object storage. It also takes snapshots of your cluster’s Persistent Volumes using your cloud provider’s block storage snapshot features, and can then restore your cluster’s objects and Persistent Volumes to a previous state.

In this tutorial we’ll set up and configure the velero command line tool on a local machine, and deploy the server component into our Kubernetes cluster. We’ll then deploy a sample Nginx app and a Redis cluster that uses Persistent Volume for data and then simulate a disaster recovery scenario

At the time of writing this article. The last published version is: v1.5.0-beta.1

However, to avoid stability problems, we are going to install the latest stable version: v1.4.2

Velero Uses Cases

velero-use-cases

source: www.cncf.io

• Backup and restore of kubernetes objects
    ◦ Uses Kubernetes Discovery API
    ◦ Does not need to talk directly to etcd
    ◦ Backups stored in Cloud Object Storage

• Backup and restore persisten volumes
    ◦ Uses cloud provider snapshots APIs
    ◦ Restic support for file system backups

Backup/restore applications and Persitent Volumes

In order to demonstrate the capabilities of Velero for disaster recovery, I have a Nginx pod running with a Persistent Volume on AWS using the GP2 storageclass

Checking our app and the PV inside kubernetes:

Below we can see the Persistent volume on AWS:

persistent-volume

Let’s gather the Nginx logs stored on the Persistent Volume (noted the timestamp):

We can see the timestamp for the logs generated are:

172.70.32.120 - - [22/Aug/2020:18:47:25 +0000] "GET / HTTP/1.0" 200 612 "-" "Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)" "-"
172.70.112.241 - - [22/Aug/2020:18:47:46 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0" "-"
172.70.96.193 - - [22/Aug/2020:18:47:47 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0" "-"
100.96.3.1 - - [22/Aug/2020:18:48:42 +0000] "GET / HTTP/1.1" 200 612 "-" "ELinks/0.13.1 (textmode; Linux 5.4.0-42-generic x86_64; 238x58-2)" "-"
172.70.96.193 - - [22/Aug/2020:18:49:21 +0000] "GET / HTTP/1.1" 200 612 "-" "ELinks/0.13.1 (textmode; Linux 5.4.0-42-generic x86_64; 238x58-2)" "-"

Velero Install and Prerequisites

The prerequisites for having velero working and be able to store backup on AWS S3 are:

  • AWS S3 bucket: kubernetes.test.velero
  • Set IAM permissions for velero
  • Install and configure velero

let’s go over the step by step.

Create AWS S3 bucket (if you don’t have one already):

Velero requires an object storage bucket to store backups in, preferably unique to a single Kubernetes cluster (see the FAQ for more details). Create an S3 bucket, replacing placeholders appropriately:
BUCKET=kubernetes.test.velero
REGION=us-east-1
aws s3api create-bucket \
    --bucket $BUCKET \
    --region $REGION \
    --create-bucket-configuration LocationConstraint=$REGION

Set Permissions for Velero:

aws iam create-user --user-name velero

Attach policies to give velero the necessary permissions:

cat > velero-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVolumes",
                "ec2:DescribeSnapshots",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateSnapshot",
                "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}"
            ]
        }
    ]
}
EOF
aws iam put-user-policy \
  --user-name velero \
  --policy-name velero \
  --policy-document file://velero-policy.jso

Create an access key for the user:

aws iam create-access-key --user-name velero

Create a Velero-specific credentials file (credentials-velero) in your local directory:

Add below content to a new file. Replace with the proper values of the AWS CLI Credentials. Name it as credentials-velero

[default]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>

Install Velero Client on your local computer

I am running a Linux Debian based OS. Therefore I am going to show the steps to run on this OS.

cd /tmp
wget https://github.com/vmware-tanzu/velero/releases/download/v1.4.2/velero-v1.4.2-linux-amd64.tar.gz
tar -xzvf velero-v1.4.2-linux-amd64.tar.gz
sudo mv velero-v1.4.2-linux-amd64/velero  /usr/local/bin/velero

velero install \
    --provider aws \
    --use-restic \
    --plugins velero/velero-plugin-for-aws:v1.1.0 \
    --bucket $BUCKET \
    --backup-location-config region=$REGION \
    --snapshot-location-config region=$REGION \
    --secret-file ./credentials-velero

Validate Velero installation

kubectl get all -n velero

velero-install

BACKUP & RESTORE

Let’s see now how velero actually works. We are goign to:

  • Backup an application based on label.
  • Delete a namespace where we have this application running.
  • Restore all components (namespace, deployment, service, Persistent Volume etc..) of the application using velero.
velero backup create nginx-backup --selector app=nginx
velero backup describe nginx-backup --details

Below we can see all components are backed up an stored in our AWS s3 bucket:

velero-install

Now I am going to delete the entire namespace simulating a disaster and tne recovery everything using velero:

kubectl delete namespace nginx-example

kubectl get deployments --namespace=nginx-example

velero restore create --from-backup nginx-backup

kubectl get deployments --namespace=nginx-example

kubectl get pvc --namespace=nginx-example

kubectl get services --namespace nginx-example

As we could see during the whole process, we were able to make a backup of the various stateless and statefull Kubernetes components and save them in an Amazon S3 bucket.

For those who have clusters running on premises and AWS S3 is not an option. The Object Storage Minio Open Source service can be used as an option.

Below I wanted to share gew URLs for different cloud provider plugins that can be usefull:

As usual, if you have any question, send me a message at contact@wecloudpro.com

Back to blog