New to Stash? Please start here.

Using Stash with TLS secured Minio Server

Minio is an open source object storage server compatible with Amazon S3 cloud storage service. You can deploy Minio server in docker container locally, in a kubernetes cluster, Microsoft Azure, GCP etc. You can find a guide for Minio server here. This tutorial will show you how to use Stash to backup a Kubernetes Deployment in a TLS secure Minio Server. It will also show you how to recover this backed up data.

Before You Begin

At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using Minikube. Now, install Stash in your cluster following the steps here.

You should have understanding the following Stash terms:

Then, you will need a TLS secure Minio server to store backed up data. You can deploy a TLS secure Minio server in your cluster by following this official guide or these steps below:

Create self-signed SSl certificate

A Certificate is used to verify the identity of server or client. Usually, a certificate issued by trusted third party is used to verify identity. We can also use a self-signed certificate. In this tutorial, we will use a self-signed certificate to verify the identity of Minio server.

You can generate self-signed certificate easily with our onessl tool.

Here is an example how we can generate a self-signed certificate using onessl tool.

First install onessl by,

$ curl -fsSL -o onessl curl -fsSL -o onessl \
  && chmod +x onessl \
  && sudo mv onessl /usr/local/bin/

Now generate CA’s root certificate,

$ onessl create ca-cert

This will create two files ca.crt and ca.key.

Now, generate certificate for server,

$ onessl create server-cert --domains minio-service.default.svc

This will generate two files server.crt and server.key.

Minio server will start TLS secure service if it find public.crt and private.key files in /root/.minio/certs/ directory of the docker container. The public.crt file is concatenation of server.crt and ca.crt where private.key file is only the server.key file.

Let’s generate public.crt and private.key file,

$ cat {server.crt,ca.crt} > public.crt
$ cat server.key > private.key

Be sure about the order of server.crt and ca.crt. The order will be server's certificate, any intermediate certificates and finally the CA's root certificate. The intermediate certificates are required if the server certificate is created using a certificate which is not the root certificate but signed by the root certificate. onessl use root certificate by default to generate server certificate if no certificate path is specified by --cert-dir flag. Hence, the intermediate certificates are not required here.

We will create a kubernetes secret with this public.crt and private.key files and mount the secret to /root/.minio/certs/ directory of minio container.

Minio server will not trust a self-signed certificate by default. We can mark the self-signed certificate as a trusted certificate by adding public.crt file in /root/.minio/certs/CAs directory.

Create Secret

Now, let’s create a secret from public.crt and private.key files,

$ kubectl create secret generic minio-server-secret \
                              --from-file=./public.crt \
secret "minio-server-secret" created

$ kubectl label secret minio-server-secret app=minio -n default

Now, verify that the secret is created successfully

$ kubectl get secret minio-server-secret -o yaml

If secret is created successfully then you will see output like this,

apiVersion: v1
  private.key: <base64 encoded private.key data>
  public.crt: <base64 encoded public.key data>
kind: Secret
  creationTimestamp: 2018-01-26T12:02:09Z
  name: minio-server-secret
  namespace: default
  resourceVersion: "40701"
  selfLink: /api/v1/namespaces/default/secrets/minio-server-secret
  uid: bc57add7-0290-11e8-9a26-080027b344c9
    app: minio
type: Opaque

Create Persistent Volume Claim

Minio server needs a Persistent Volume to store data. Let’s create a Persistent Volume Claim to request Persistent Volume from the cluster.

$ kubectl apply -f ./docs/examples/backends/minio/minio-pvc.yaml
persistentvolumeclaim "minio-pvc" created

YAML for PersistentVolumeClaim,

apiVersion: v1
kind: PersistentVolumeClaim
  # This name uniquely identifies the PVC. Will be used in minio deployment.
  name: minio-pvc
    app: minio
  storageClassName: standard
    - ReadWriteOnce
    # This is the request for storage. Should be available in the cluster.
      storage: 2Gi

Create Deployment

Minio deployment creates pod where the Minio server will run. Let’s create a deployment for minio server by,

$ kubectl apply -f ./docs/examples/backends/minio/minio-deployment.yaml
deployment "minio-deployment" created

YAML for minio-deployment

apiVersion: extensions/v1beta1
kind: Deployment
  # This name uniquely identifies the Deployment
  name: minio-deployment
    app: minio
    type: Recreate # If pod fail, we want to recreate pod rather than restarting it.
        # Label is used as a selector in the service.
        app: minio-server
      # Refer to the PVC have created earlier
      - name: storage
          # Name of the PVC created earlier
          claimName: minio-pvc
      - name: minio-certs
          secretName: minio-server-secret
          - key: public.crt
            path: public.crt
          - key: private.key
            path: private.key
          - key: public.crt
            path: CAs/public.crt # mark self signed certificate as trusted
      - name: minio
        # Pulls the default Minio image from Docker Hub
        image: minio/minio
        - server
        - --address
        - ":443"
        - /storage
        # Minio access key and secret key
        - name: MINIO_ACCESS_KEY
          value: "<your minio access key(any string)>"
        - name: MINIO_SECRET_KEY
          value: "<your minio secret key(any string)>"
        - containerPort: 443
          # This ensures containers are allocated on separate hosts. Remove hostPort to allow multiple Minio containers on one host
          hostPort: 443
        # Mount the volumes into the pod
        - name: storage # must match the volume name, above
          mountPath: "/storage"
        - name: minio-certs
          mountPath: "/root/.minio/certs"

Create Service

Now, the final touch. Minio server is running in the cluster. Let’s create a service so that other pods can access the server.

$ kubectl apply -f ./docs/examples/backends/minio/minio-service.yaml
service "minio-service" created

YAML for minio-service

apiVersion: v1
kind: Service
  name: minio-service
    app: minio
  type: LoadBalancer
    - port: 443
      targetPort: 443
      protocol: TCP
    app: minio-server # must match with the label used in the deployment

Verify that the service is created successfully,

$ kubectl get service minio-service
NAME            TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
minio-service   LoadBalancer   <pending>     443:30722/TCP   49s


In this tutorial, we are going to backup the /source/data folder of a busybox pod into a Minio backend. Then, we will recover the data to another HostPath volume form backed up snapshots.


First, deploy the following busybox Deployment in your cluster. Here we are using a git repository as a source volume for demonstration purpose.

$  kubectl apply -f ./docs/examples/tutorial/busybox.yaml
deployment "stash-demo" created

YAML for busybox deplyment,

apiVersion: apps/v1beta1
kind: Deployment
    app: stash-demo
  name: stash-demo
  namespace: default
  replicas: 1
        app: stash-demo
      name: busybox
      - command:
        - sleep
        - "3600"
        image: busybox
        imagePullPolicy: IfNotPresent
        name: busybox
        - mountPath: /source/data
          name: source-data
      restartPolicy: Always
      - gitRepo:
        name: source-data

Run the following command to confirm that busybox pods are running.

$ kubectl get pods -l app=stash-demo
NAME                          READY     STATUS    RESTARTS   AGE
stash-demo-69d9dd8d76-bz2bz   1/1       Running   0          12s

You can check that the /source/data/ directory of pod is populated with data from the volume source using this command,

$ kubectl exec stash-demo-69d9dd8d76-bz2bz -- ls -R /source/data/


Now, let’s backup the directory into a Minio server.

At first, we need to create a secret for Restic crd. To configure this backend, following secret keys are needed:

Key Description
RESTIC_PASSWORD Required. Password used to encrypt snapshots by restic
AWS_ACCESS_KEY_ID Required. Minio access key ID
AWS_SECRET_ACCESS_KEY Required. Minio secret access key
CA_CERT_DATA Required. Root certificate by which Minio server certificate is signed

Create secret for Restic crd,

$ echo -n 'changeit' > RESTIC_PASSWORD
$ echo -n '<your-minio-access-key-id-here>' > AWS_ACCESS_KEY_ID
$ echo -n '<your-minio-secret-access-key-here>' > AWS_SECRET_ACCESS_KEY
$ cat ./directory/of/root/certificate/ca.crt > CA_CERT_DATA
$ kubectl create secret generic minio-restic-secret \
    --from-file=./RESTIC_PASSWORD \
    --from-file=./AWS_ACCESS_KEY_ID \
    --from-file=./AWS_SECRET_ACCESS_KEY \
secret "minio-restic-secret" created

Verify that the secret has been created successfully,

$ kubectl get secret minio-restic-secret -o yaml
apiVersion: v1
  CA_CERT_DATA: <base64 endoded ca.crt data>
kind: Secret
  creationTimestamp: 2018-01-29T11:20:35Z
  name: minio-restic-secret
  namespace: default
  resourceVersion: "7773"
  selfLink: /api/v1/namespaces/default/secrets/minio-restic-secret
  uid: 6d70a2c1-04e6-11e8-b4cd-0800279de528
type: Opaque

Now, we can create Restic crd. This will create a repository in Minio server and start taking periodic backup of /source/data/ folder.

$ kubectl apply -f ./docs/examples/backends/minio/minio-restic.yaml
restic "minio-restic" created

YAML of Restic crd for Minio backend,

kind: Restic
  name: minio-restic
  namespace: default
      app: stash-demo # Must match with the label of busybox pod we have created before.
  - path: /source/data
    retentionPolicyName: 'keep-last-5'
      endpoint: 'https://minio-service.default.svc' # Use your own Minio server address.
      bucket: stash-qa  # Give a name of the bucket where you want to backup.
      prefix: demo  # . Path prefix into bucket where repository will be created.(optional).
    storageSecretName: minio-restic-secret
  schedule: '@every 1m'
  - mountPath: /source/data
    name: source-data
  - name: 'keep-last-5'
    keepLast: 5
    prune: true

If everything goes well, a Repository crd with name deployment.stash-demo will be created for the respective repository in Minio backend. Verify that, Repository is created successfully using this command,

$ kubectl get repository deployment.stash-demo
NAME                    AGE
deployment.stash-demo   1m

Restic will take backup of the volume periodically with a 1-minute interval. You can verify that backup is taking successfully by,

$ kubectl get snapshots -l repository=deployment.stash-demo
NAME                             AGE
deployment.stash-demo-c1014ca6   10s

Here, deployment.stash-demo-c1014ca6 represents the name of the successful backup Snapshot taken by Stash in deployment.stash-demo repository.


Now, it is time to recover the backed up data. At first, delete Restic crd so that it does not lock the restic repository while we are trying to recover from it.

$ kubectl delete restic minio-restic
restic "minio-restic" deleted

Now, create a Recovery crd.

$ kubectl apply -f ./docs/examples/backends/minio/minio-recovery.yaml
recovery "minio-recovery" created

YAML for Recovery crd

kind: Recovery
  name: minio-recovery
  namespace: default
    name: deployment.stash-demo
    namespace: default
  - /source/data
  - mountPath: /source/data # where the volume will be mounted
    name: stash-recovered-volume
    hostPath: # volume source, where the recovered data will be stored.
      path: /data/stash-recovered/ # directory in volume source where recovered data will be stored

Wait until Recovery job completed its task. To verify that recovery completed successfully run,

$ kubectl get recovery minio-recovery -o yaml
kind: Recovery
  annotations: |
  clusterName: ""
  creationTimestamp: 2018-01-30T06:54:18Z
  generation: 0
  name: minio-recovery
  namespace: default
  resourceVersion: "10060"
  selfLink: /apis/
  uid: 64c12ff7-058a-11e8-9976-08002750604b
    name: deployment.stash-demo
    namespace: default
  - /source/data
  - hostPath:
      path: /data/stash-recovered/
    mountPath: /source/data
    name: stash-recovered-volume
  phase: Succeeded

status.phase: Succeeded indicates that the recovery was successful. Now, we can check /data/stash-recovered/ directory of HostPath to see the recovered data. If you are using minikube for cluster then you can check by,

$ minikube ssh
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ sudo su
$ cd /
$ ls -R /data/stash-recovered/


Eureka-by-EdgarAllanPoe.txt  LICENSE

You can mount this recovered volume into any pod.


To cleanup the Kubernetes resources created by this tutorial, run:

$ kubectl delete deployment stash-demo
$ kubectl delete restic minio-restic
$ kubectl delete recovery minio-recovery
$ kubectl delete secret minio-restic-secret
$ kubectl delete repository deployment.stash-demo

To cleanup the minio server, run:

$ kubectl delete deployment minio-deployment
$ kubectl delete service minio-service
$ kubectl delete pvc minio-pvc
$ kubectl delete secret minio-server-secret

Take your team where it needs to go.

Create your cluster in minutes. Our team is here to help and would be happy to chat with you.