Using Stash with Rook Storage Service

This tutorial will show you how to use Stash to backup and restore a Kubernetes volume in Rook storage service. Here, we are going to backup the /source/data folder of a busybox pod into AWS S3 compatible Rook Object Storage. Then, we will show how to recover this data into a PersistentVolumeClaim of Rook Block Storage. We will also re-deploy deployment using this recovered volume.

Before You Begin

At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using Minikube. Now, install Stash in your cluster following the steps here.

You should have understanding the following Stash concepts:

Then, you will need to have a Rook Storage Service with Object Storage and Block Storage configured. If you do not already have a Rook Storage Service configured, you can create one by following this quickstart guide.

Backup

First, deploy the following busybox Deployment in your cluster. Here we are using a git repository as a source volume for demonstration purpose.

$ kubectl apply -f ./docs/examples/tutorial/busybox.yaml
deployment "stash-demo" created

Definition of busybox deployment:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  labels:
    app: stash-demo
  name: stash-demo
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: stash-demo
      name: busybox
    spec:
      containers:
      - args:
        - sleep
        - "3600"
        image: busybox
        imagePullPolicy: IfNotPresent
        name: busybox
        volumeMounts:
        - mountPath: /source/data
          name: source-data
      restartPolicy: Always
      volumes:
      - gitRepo:
          repository: https://github.com/appscode/stash-data.git
        name: source-data

Run the following command to confirm that busybox pods are running.

$ kubectl get pods -l app=stash-demo
NAME                         READY     STATUS    RESTARTS   AGE
stash-demo-b66b9cdfd-j7rb5   1/1       Running   0          49s

You can check that the /source/data/ directory of pod is populated with data from the volume source using this command,

$  kubectl exec stash-demo-b66b9cdfd-j7rb5 -- ls -R /source/data/
/source/data/:
stash-data

/source/data/stash-data:
Eureka-by-EdgarAllanPoe.txt
LICENSE
README.md

Now, let’s backup the directory into a AWS S3 compatible Rook Object Storage.

At first, we need to create a secret for Restic crd. Create secret for Restic using following command,

$ echo -n 'changeit' > RESTIC_PASSWORD
$ echo -n '<your-rook-access-key-id-here>' > AWS_ACCESS_KEY_ID
$ echo -n '<your-rook-secret-access-key-here>' > AWS_SECRET_ACCESS_KEY
$ kubectl create secret generic rook-restic-secret \
      --from-file=./RESTIC_PASSWORD \
      --from-file=./AWS_ACCESS_KEY_ID \
      --from-file=./AWS_SECRET_ACCESS_KEY
secret "rook-restic-secret" created

Verify that the secret has been created successfully,

$ kubectl get secret rook-restic-secret -o yaml
apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: <base64 encoded rook access key>
  AWS_SECRET_ACCESS_KEY: <base64 encoded rook secret key>
  RESTIC_PASSWORD: Y2hhbmdlaXQ=
kind: Secret
metadata:
  creationTimestamp: 2018-04-12T10:32:14Z
  name: rook-restic-secret
  namespace: default
  resourceVersion: "2414"
  selfLink: /api/v1/namespaces/default/secrets/rook-restic-secret
  uid: c454391b-3e3c-11e8-a7b6-080027672508
type: Opaque

Now, we can create Restic crd. This will create a repository stash-backup-repo in Rook Object Storage bucket and start taking periodic backup of /source/data/ folder.

$ kubectl apply -f ./docs/examples/backends/rook/rook-restic.yaml
restic "rook-restic" created

Definition of Restic crd for Rook Object Storage backend,

apiVersion: stash.appscode.com/v1alpha1
kind: Restic
metadata:
  name: rook-restic
  namespace: default
spec:
  selector:
    matchLabels:
      app: stash-demo # Must match with the label of busybox pod we have created before.
  fileGroups:
  - path: /source/data
    retentionPolicyName: 'keep-last-5'
  backend:
    s3:
      endpoint: 'http://rook-ceph-rgw-my-store.rook' # Use your own rook object storage end point.
      bucket: stash-backup  # Give a name of the bucket where you want to backup.
      prefix: demo  # . Path prefix into bucket where repository will be created.(optional).
    storageSecretName: rook-restic-secret
  schedule: '@every 1m'
  volumeMounts:
  - mountPath: /source/data
    name: source-data
  retentionPolicies:
  - name: 'keep-last-5'
    keepLast: 5
    prune: true

If everything goes well, A Repository crd with name deployment.stash-demo will be created for the respective repository in Rook Object Storage backend. Verify that, Repository is created successfully using this command,

$ kubectl get repository deployment.stash-demo
NAME                    AGE
deployment.stash-demo   1m

Restic will take backup of the volume periodically with a 1-minute interval. You can verify that backup is taking successfully by,

$ kubectl get snapshots -l repository=deployment.stash-demo
NAME                             AGE
deployment.stash-demo-c1014ca6   10s

Here, deployment.stash-demo-c1014ca6 represents the name of the successful backup Snapshot taken by Stash in deployment.stash-demo repository.

Recover to PersistentVolumeClaim

Rook Block Storage allow the users to mount persistent volume into pod using PersistentVolumeClaim. Here, we will recover our backed up data into a PVC.

At first, delete Restic crd so that it does not lock the restic repository while we are trying to recover from it.

$ kubectl delete restic rook-restic
restic "rook-restic" deleted

Now, create a PersistentVolumeClaim for Rook Block Storage,

$ kubectl apply -f ./docs/examples/backends/rook/rook-pvc.yaml
persistentvolumeclaim "stash-recovered" created

Definition of PersistentVolumeClaim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: stash-recovered
  labels:
    app: stash-demo
spec:
  storageClassName: rook-block
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

Check cluster has provisioned the requested claim,

$ kubectl get pvc -l app=stash-demo
NAME              STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
stash-recovered   Bound     pvc-a7aa73fa-3e3f-11e8-a7b6-080027672508   2Gi        RWO            rook-block     36s

Look at the STATUS filed. stash-recovered PVC is bounded to volume pvc-a7aa73fa-3e3f-11e8-a7b6-080027672508.

Now, create a Recovery to recover backed up data in this PVC.

$ kubectl apply -f ./docs/examples/backends/rook/rook-recovery.yaml
recovery "rook-recovery" created

Definition of Recovery should look like below:

apiVersion: stash.appscode.com/v1alpha1
kind: Recovery
metadata:
  name: rook-recovery
  namespace: default
spec:
  repository:
    name: deployment.stash-demo
    namespace: default
  paths:
  - /source/data
  recoveredVolumes:
  - mountPath: /source/data
    persistentVolumeClaim:
      claimName: stash-recovered

Wait until Recovery job completed its task. To verify that recovery completed successfully run,

$ kubectl get recovery rook-recovery -o yaml
apiVersion: stash.appscode.com/v1alpha1
kind: Recovery
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
     {"apiVersion":"stash.appscode.com/v1alpha1","kind":"Recovery","metadata":{"name":"rook-recovery","namespace":"default"},"spec":{"repository":"deployment.stash-demo","paths":["/source/data"],"recoveredVolumes":[{"mountPath":"/source/data","persistentVolumeClaim":{"claimName":"stash-recovered"}}]}}
  clusterName: ""
  creationTimestamp: 2018-04-12T12:57:54Z
  generation: 0
  name: rook-recovery
  namespace: default
  resourceVersion: "2315"
  selfLink: /apis/stash.appscode.com/v1alpha1/namespaces/default/recoveries/rook-recovery
  uid: 1dbad356-3e51-11e8-b2bd-080027dbef96
spec:
  repository:
    name: deployment.stash-demo
    namespace: default
  paths:
  - /source/data
  recoveredVolumes:
  - mountPath: /source/data
    persistentVolumeClaim:
      claimName: stash-recovered
status:
  phase: Succeeded

Now, let’s re-deploy the busybox deployment using this recovered PVC. First, delete old deployment and recovery job.

$ kubectl delete deployment stash-demo
deployment "stash-demo" deleted

$ kubectl delete recovery rook-recovery
recovery "rook-recovery" deleted

Now, mount the recovered PersistentVolumeClaim in busybox deployment instead of gitRepo we had mounted before then re-deploy it,

$ kubectl apply -f ./docs/examples/backends/rook/restored-deployment.yaml
deployment "stash-demo" created
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  labels:
    app: stash-demo
  name: stash-demo
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: stash-demo
      name: busybox
    spec:
      containers:
      - args:
        - sleep
        - "3600"
        image: busybox
        imagePullPolicy: IfNotPresent
        name: busybox
        volumeMounts:
        - mountPath: /source/data
          name: source-data
      restartPolicy: Always
      volumes:
      - name: source-data
        persistentVolumeClaim:
          claimName: stash-recovered

Get the pod of new deployment,

$  kubectl get pod -l app=stash-demo
NAME                          READY     STATUS    RESTARTS   AGE
stash-demo-5bc57fbcfb-b45k8   1/1       Running   0          1m

Check the backed up data is restored in /source/data/ directory of busybox pod.

$ kubectl exec stash-demo-5bc57fbcfb-b45k8 -- ls -R /source/data/
/source/data/:
lost+found
stash-data

/source/data/lost+found:

/source/data/stash-data:
Eureka-by-EdgarAllanPoe.txt
LICENSE
README.md

Cleanup

$ kubectl delete pvc stash-recovered
$ kubectl delete deployment stash-demo
$ kubectl delete repository deployment.stash-demo

Uninstall Stash following the instructions here.

Take your team where it needs to go.

Create your cluster in minutes. Our team is here to help and would be happy to chat with you.