Pre-Requisites

The following configurations should be applied to the platform to use Arthur's Backup and Restore capabilities:

1. Arthur must be configured to use external object storage, specifically, AWS S3
2. The access to external storage must be configured using IRSA annotations
3. In order to use IRSA annotations, the cluster must be deployed using Amazon EKS

If the above are not true/possible for your deployment, please reach out to Arthur Support.

Configuring Velero

The only component that needs to be installed separately from Arthur to perform backup and restores is Velero. Instructions are provided below for setting up Velero to store backups in S3 using IRSA.
The general overview of the installation is as follows:

1. Setup permissions for Velero
2. Install Velero
3. Confirm Velero is installed and configured correctly
4. Configure the Backup Storage Destination to point to S3

Velero permissions

1. Generate the below policy which will grant Velero the necessary permissions:

   TextShell

   $ cat > velero-policy.json <<EOF
   {
     "Version": "2012-10-17",
     "Statement": [
         {
             "Effect": "Allow",
             "Action": [
                 "ec2:DescribeVolumes",
                 "ec2:DescribeSnapshots",
                 "ec2:CreateTags",
                 "ec2:CreateVolume",
                 "ec2:CreateSnapshot",
                 "ec2:DeleteSnapshot"
             ],
             "Resource": "*"
         },
         {
             "Effect": "Allow",
             "Action": [
                 "s3:GetObject",
                 "s3:DeleteObject",
                 "s3:PutObject",
                 "s3:AbortMultipartUpload",
                 "s3:ListMultipartUploadParts"
             ],
             "Resource": [
                 "arn:aws:s3:::${BUCKET}/*"
             ]
         },
         {
             "Effect": "Allow",
             "Action": [
                 "s3:ListBucket"
             ],
             "Resource": [
                 "arn:aws:s3:::${BUCKET}"
             ]
         },
         {
             "Effect": "Allow",
             "Action": [
                 "kms:CreateGrant*",
                 "kms:ReEncrypt*",
                 "kms:GenerateDataKey*",
                 "kms:Encrypt*",
                 "kms:DescribeKey*",
                 "kms:Decrypt*"
             ],
             "Resource": "*"
         }
     ]
   }
   EOF
   

   $ aws iam create-policy \
     --policy-name velero-perms \
     --policy-document file://velero-policy.json
   

2. Attach this IAM policy to the IAM role that the Arthur service account (IRSA) assumes

   Shell

   $ aws iam attach-role-policy \
     --role-name <IRSA-Role-name> \
     --policy-arn <policy-ARN>
   

📘

Encrypting data at rest

Arthur highly recommends that your EBS volumes are encrypted with KMS.
In addition to giving Velero the permission for KMS, please make sure that the IAM roles assumed by Arthur service account also have access to KMS so the restored encrypted volumes can be re-attached.
If you're using separate KMS keys on the cluster you backed-up and the cluster you're restoring to, the EBS volume snapshots must be copied with the new KMS key so the new cluster can work with the snapshots.

Install Velero

Velero can be installed on the Kubernetes cluster using helm.

1. Generate a velero-values file as follows (taken from the official source with defaults removed for brevity):

   YAML

   $ cat > velero-values.yaml <<EOF
   resources:
     requests:
       cpu: 500m
       memory: 128Mi
     limits:
       cpu: 1000m
       memory: 512Mi
   
   initContainers:
     - name: velero-plugin-for-aws
       image: velero/velero-plugin-for-aws:v1.6.1
       imagePullPolicy: IfNotPresent
       volumeMounts:
         - mountPath: /target
           name: plugins
   
   podSecurityContext:
     runAsNonRoot: true
     runAsUser: 1000
     runAsGroup: 1000
   
   upgradeCRDs: true
   
   cleanUpCRDs: false
   
   configuration:
     # Cloud provider being used (e.g. aws, azure, gcp).
     provider: aws
   
     backupStorageLocation:
       # name is the name of the backup storage location where backups should be stored.
       name: <insert-bsl-name-here>
       provider: aws
       # bucket is the name of the bucket to store backups in. Required.
       bucket: <insert-s3-bucket-name-here>
       config:
         region: us-east-2
   
     volumeSnapshotLocation:
       # name is the name of the volume snapshot location where snapshots are being taken. Required.
       name: <insert-vsl-name-here>
       config:
         region: us-east-2
   
     # These are server-level settings passed as CLI flags to the `velero server` command.
     logLevel: debug
     namespace: <insert-velero-namespace-here>
   
   rbac:
     # Whether to create the Velero role and role binding to give all permissions to the namespace to Velero.
     create: true
     # Whether to create the cluster role binding to give administrator permissions to Velero
     clusterAdministrator: true
     # Name of the ClusterRole.
     clusterAdministratorName: cluster-admin
   
   # Information about the Kubernetes service account Velero uses.
   serviceAccount:
     server:
       create: true
       name: velero
       annotations:
         eks.amazonaws.com/sts-regional-endpoints: "true"
         eks.amazonaws.com/role-arn: <insert-IAM-Role-ARN-here>
       labels:
   
   credentials:
     useSecret: false
   
   backupsEnabled: true
   
   snapshotsEnabled: true
   
   deployNodeAgent: false
   EOF
   

2. Install the Velero helm chart with the above values file:

   Shell{note}

   $ velero_namespace=<insert-velero-namespace-here>
   $ helm install velero vmware-tanzu/velero \
     --create-namespace \
     --namespace $velero_namespace \
     --version 3.2.0 \
     -f velero-values.yaml
   

   Arthur recommends installing Velero in a different namespace from Arthur, so Velero can be managed separately from Arthur.
   

📘

Where to install velero

Arthur recommends installing Velero in a different namespace from Arthur, so Velero can be managed separately from Arthur.

Verify Velero Installation

To confirm that Velero is installed and configured correctly:

1. Open the Kots Admin Interface and navigate to the "Snapshots" tab
2. Click the "Check for Velero" button (see the screenshot below)

Validate the Backup Storage Location

The Backup Storage Location is a Velero resource that points to the S3 Bucket where backups will be stored. Use kubectl to validate connectivity/access to AWS S3, which should say "Available".

$ velero_namespace=<insert-velero-namespace-here>
$ kubectl get backupstoragelocation -n $velero_namespace

Please do not proceed until the Backup Storage Location is Available.

Configuring clickhouse-backup

Configuring clickhouse-backup to store backups in remote storage (e.g., S3) can be done in the Admin Console.
Once your cluster is set up for Backup and Restore, you should see the "Enable OLAP Database Backup Capabilities" option in the "OLAP Database" section.

Ensure that:

1. The configuration that points to the bucket is correct
   • The Bucket Name
   • The Bucket Region
2. The ServiceAccount is the same ServiceAccount that you've configured with the IRSA annotation (if you are not sure, enter the default value)
3. The IAM Role that you are using for the IRSA annotation has the appropriate permissions to read/write/list from the S3 bucket
4. The S3 Path is where you want to be storing backups