Product DocumentationAPI and Python SDK ReferenceRelease Notes
Schedule a Demo
Product Documentation
Schedule a Demo

Pre-Requisites

The following configurations should be applied to the platform to use Arthur's Backup and Restore capabilities:

  1. Arthur must be configured to use external object storage, specifically, AWS S3
  2. The access to external storage must be configured using IRSA annotations
  3. In order to use IRSA annotations, the cluster must be deployed using Amazon EKS

If the above are not true/possible for your deployment, please reach out to Arthur Support.

Configuring Velero

The only component that needs to be installed separately from Arthur to perform backup and restores is Velero. Instructions are provided below for setting up Velero to store backups in S3 using IRSA.
The general overview of the installation is as follows:

  1. Setup permissions for Velero
  2. Install Velero
  3. Confirm Velero is installed and configured correctly
  4. Configure the Backup Storage Destination to point to S3

Velero permissions

  1. Generate the below policy which will grant Velero the necessary permissions:

    $ cat > velero-policy.json <<EOF
    {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": [
                  "ec2:DescribeVolumes",
                  "ec2:DescribeSnapshots",
                  "ec2:CreateTags",
                  "ec2:CreateVolume",
                  "ec2:CreateSnapshot",
                  "ec2:DeleteSnapshot"
              ],
              "Resource": "*"
          },
          {
              "Effect": "Allow",
              "Action": [
                  "s3:GetObject",
                  "s3:DeleteObject",
                  "s3:PutObject",
                  "s3:AbortMultipartUpload",
                  "s3:ListMultipartUploadParts"
              ],
              "Resource": [
                  "arn:aws:s3:::${BUCKET}/*"
              ]
          },
          {
              "Effect": "Allow",
              "Action": [
                  "s3:ListBucket"
              ],
              "Resource": [
                  "arn:aws:s3:::${BUCKET}"
              ]
          },
          {
              "Effect": "Allow",
              "Action": [
                  "kms:CreateGrant*",
                  "kms:ReEncrypt*",
                  "kms:GenerateDataKey*",
                  "kms:Encrypt*",
                  "kms:DescribeKey*",
                  "kms:Decrypt*"
              ],
              "Resource": "*"
          }
      ]
    }
    EOF
    
    $ aws iam create-policy \
      --policy-name velero-perms \
      --policy-document file://velero-policy.json
    
  2. Attach this IAM policy to the IAM role that the Arthur service account (IRSA) assumes

    $ aws iam attach-role-policy \
      --role-name <IRSA-Role-name> \
      --policy-arn <policy-ARN>
    

📘

Encrypting data at rest

Arthur highly recommends that your EBS volumes are encrypted with KMS.
In addition to giving Velero the permission for KMS, please make sure that the IAM roles assumed by Arthur service account also have access to KMS so the restored encrypted volumes can be re-attached.
If you're using separate KMS keys on the cluster you backed-up and the cluster you're restoring to, the EBS volume snapshots must be copied with the new KMS key so the new cluster can work with the snapshots.

Install Velero

Velero can be installed on the Kubernetes cluster using helm.

  1. Generate a velero-values file as follows (taken from the official source with defaults removed for brevity):

    $ cat > velero-values.yaml <<EOF
    resources:
      requests:
        cpu: 500m
        memory: 128Mi
      limits:
        cpu: 1000m
        memory: 512Mi
    
    initContainers:
      - name: velero-plugin-for-aws
        image: velero/velero-plugin-for-aws:v1.6.1
        imagePullPolicy: IfNotPresent
        volumeMounts:
          - mountPath: /target
            name: plugins
    
    podSecurityContext:
      runAsNonRoot: true
      runAsUser: 1000
      runAsGroup: 1000
    
    upgradeCRDs: true
    
    cleanUpCRDs: false
    
    configuration:
      # Cloud provider being used (e.g. aws, azure, gcp).
      provider: aws
    
      backupStorageLocation:
        # name is the name of the backup storage location where backups should be stored.
        name: <insert-bsl-name-here>
        provider: aws
        # bucket is the name of the bucket to store backups in. Required.
        bucket: <insert-s3-bucket-name-here>
        config:
          region: us-east-2
    
      volumeSnapshotLocation:
        # name is the name of the volume snapshot location where snapshots are being taken. Required.
        name: <insert-vsl-name-here>
        config:
          region: us-east-2
    
      # These are server-level settings passed as CLI flags to the `velero server` command.
      logLevel: debug
      namespace: <insert-velero-namespace-here>
    
    rbac:
      # Whether to create the Velero role and role binding to give all permissions to the namespace to Velero.
      create: true
      # Whether to create the cluster role binding to give administrator permissions to Velero
      clusterAdministrator: true
      # Name of the ClusterRole.
      clusterAdministratorName: cluster-admin
    
    # Information about the Kubernetes service account Velero uses.
    serviceAccount:
      server:
        create: true
        name: velero
        annotations:
          eks.amazonaws.com/sts-regional-endpoints: "true"
          eks.amazonaws.com/role-arn: <insert-IAM-Role-ARN-here>
        labels:
    
    credentials:
      useSecret: false
    
    backupsEnabled: true
    
    snapshotsEnabled: true
    
    deployNodeAgent: false
    EOF
    
  2. Install the Velero helm chart with the above values file:

    $ velero_namespace=<insert-velero-namespace-here>
    $ helm install velero vmware-tanzu/velero \
      --create-namespace \
      --namespace $velero_namespace \
      --version 3.2.0 \
      -f velero-values.yaml
    
    Arthur recommends installing Velero in a different namespace from Arthur, so Velero can be managed separately from Arthur.
    

📘

Where to install velero

Arthur recommends installing Velero in a different namespace from Arthur, so Velero can be managed separately from Arthur.

Verify Velero Installation

To confirm that Velero is installed and configured correctly:

  1. Open the Kots Admin Interface and navigate to the "Snapshots" tab
  2. Click the "Check for Velero" button (see the screenshot below)

Validate the Backup Storage Location

The Backup Storage Location is a Velero resource that points to the S3 Bucket where backups will be stored. Use kubectl to validate connectivity/access to AWS S3, which should say "Available".

$ velero_namespace=<insert-velero-namespace-here>
$ kubectl get backupstoragelocation -n $velero_namespace

Please do not proceed until the Backup Storage Location is Available.

Configuring clickhouse-backup

Configuring clickhouse-backup to store backups in remote storage (e.g., S3) can be done in the Admin Console.
Once your cluster is set up for Backup and Restore, you should see the "Enable OLAP Database Backup Capabilities" option in the "OLAP Database" section.

Ensure that:

  1. The configuration that points to the bucket is correct
    • The Bucket Name
    • The Bucket Region
  2. The ServiceAccount is the same ServiceAccount that you've configured with the IRSA annotation (if you are not sure, enter the default value)
  3. The IAM Role that you are using for the IRSA annotation has the appropriate permissions to read/write/list from the S3 bucket
  4. The S3 Path is where you want to be storing backups