New to Kubed? Please start here.

Forward Cluster Events

Kubed can send notifications via Email, SMS or Chat for various cluster events. This document will show you how to use Kubed to setup an event forwarder.

Before You Begin

At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using Minikube.

Deploy Kubed

To enable config syncer, you need a cluster config like below.

$ cat ./docs/examples/event-forwarder/config.yaml

clusterName: unicorn
eventForwarder:
  receivers:
  - notifier: Mailgun
    to:
    - ops@example.com
  rules:
  # notify for warning events in kube-system namespace
  - namespaces:
    - kube-system
    operations:
    - CREATE
    resources:
    - group: "" # core API group
      resources:
      - events
  # notify for both CREATE and DELETE operations in any namespace
  - resources:
    - group: ""  # core API group
      resources:
      - nodes
      - persistentvolumes
      - persistentvolumeclaims
    - group: storage.k8s.io
      resources:
      - storageclasses
    - group: extensions
      resources:
      - ingresses
    - group: voyager.appscode.com
      resources:
      - ingresses
    - group: certificates.k8s.io
      resources:
      - certificatesigningrequests
notifierSecretName: notifier-config

The configuration format is inpired by audit policy file format. The policy is defined here. The matcher logic is implemented here.

NB: The event forwarder configuration format has been redesigned in 0.8.0 and should be updates accordingly if you are upgrading from a previous version.

Now, create a Secret with the Kubed cluster config under config.yaml key.

$ kubectl create secret generic kubed-config -n kube-system \
    --from-file=./docs/examples/event-forwarder/config.yaml
secret "kubed-config" created

# apply app=kubed label to easily cleanup later
$ kubectl label secret kubed-config app=kubed -n kube-system
secret "kubed-config" labeled

$ kubectl get secret kubed-config -n kube-system -o yaml
apiVersion: v1
data:
  config.yaml: ZXZlbnRGb3J3YXJkZXI6CiAgbm9kZUFkZGVkOgogICAgaGFuZGxlOiB0cnVlCiAgc3RvcmFnZUFkZGVkOgogICAgaGFuZGxlOiB0cnVlCiAgaW5ncmVzc0FkZGVkOgogICAgaGFuZGxlOiB0cnVlCiAgd2FybmluZ0V2ZW50czoKICAgIGhhbmRsZTogdHJ1ZQogICAgbmFtZXNwYWNlczoKICAgIC0ga3ViZS1zeXN0ZW0KICByZWNlaXZlcjoKICAgIG5vdGlmaWVyOiBtYWlsZ3VuCiAgICB0bzoKICAgIC0gb3BzQGV4YW1wbGUuY29tCm5vdGlmaWVyU2VjcmV0TmFtZToga3ViZWQtbm90aWZpZXIK
kind: Secret
metadata:
  creationTimestamp: 2017-07-27T05:35:54Z
  labels:
    app: kubed
  name: kubed-config
  namespace: kube-system
  resourceVersion: "70583"
  selfLink: /api/v1/namespaces/kube-system/secrets/kubed-config
  uid: 753220c3-728d-11e7-87f5-08002738e55e
type: Opaque

Now, deploy Kubed operator in your cluster following the steps here. Once the operator pod is running, go to the next section.

Test Forwarder

To keep things isolated, this tutorial uses a separate namespace called demo throughout this tutorial. Run the following command to prepare your cluster for this tutorial:

$ kubectl create namespace demo
namespace "demo" created

~ $ kubectl get namespaces
NAME          STATUS    AGE
default       Active    6h
kube-public   Active    6h
kube-system   Active    6h
demo          Active    4m

Forward Storage Added Event

In this section, a PVC will be used to show how event forwarder feature can be used. Create a PVC called myclaim in the demo namespace.

$ kubectl apply -f ./docs/examples/event-forwarder/demo-0.yaml
persistentvolumeclaim "myclaim" configured
$ kubectl get pvc myclaim -n demo -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"a56b7269-71ef-11e7-af79-08002738e55e","leaseDurationSeconds":15,"acquireTime":"2017-07-27T01:24:08Z","renewTime":"2017-07-27T01:24:10Z","leaderTransitions":0}'
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"myclaim","namespace":"demo"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"50Mi"}},"storageClassName":"standard"}}
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
  creationTimestamp: 2017-07-27T01:24:08Z
  name: myclaim
  namespace: demo
  resourceVersion: "58641"
  selfLink: /api/v1/namespaces/demo/persistentvolumeclaims/myclaim
  uid: 49b9851c-726a-11e7-af79-08002738e55e
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 50Mi
  storageClassName: standard
  volumeName: pvc-49b9851c-726a-11e7-af79-08002738e55e
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Mi
  phase: Bound

Now, assuming you configured a GMail account as the receiver for events, you should see an email like below:

PVC Added Notification

Forward Warning Events

In this section, a Busybox pod will be used to show how warning events are forwarded. Create a Pod called busybox in the demo namespace.

$ cat ./docs/examples/event-forwarder/demo-1.yaml

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: demo
spec:
  restartPolicy: Never
  containers:
  - name: busybox
    image: busybox
    imagePullPolicy: IfNotPresent
    command:
      - bad
      - "3600"
$ kubectl apply -f ./docs/examples/event-forwarder/demo-1.yaml
pod "busybox" created

$ kubectl get pods -n demo --show-all
NAME      READY     STATUS                                                                                                                                                                                                      RESTARTS   AGE
busybox   0/1       rpc error: code = 2 desc = failed to start container "bcc25386c0c9421b04ce9c574405917fc4940a0b324a2b062f02978c46463f07": Error response from daemon: Container command 'bad' not found or does not exist.   0          10m

Here, the busybox pod fails to start because it uses a missing command called bad. This results in 2 Warning events. Now, check your GMail account. You should receive 2 emails like below.

Pod Failed Pod FailedSync

Supported Kubernetes Objects

Following Kubernetes objects are supported by event forwarder:

  • v1:
    • ConfigMap
    • Event
    • LimitRange
    • Namespace
    • Node
    • PersistentVolume
    • PersistentVolumeClaim
    • ReplicationController
    • Secret
    • Service
    • ServiceAccount
  • apps/v1beta1:
    • Deployment
    • StatefulSet
  • batch/v1:
    • Job
  • batch/v1beta1:
    • CronJob
  • extensions/v1beta1:
    • Deployment
    • Ingress
    • ReplicaSet
  • networking.k8s.io/v1:
    • NetworkPolicy
  • kubedb/v1alpha1:
    • DormantDatabase
    • Elasticsearch
    • Memcached
    • MongoDB
    • MySQL
    • Postgres
    • Redis
    • Snapshot
  • monitoring.coreos.com/v1:
    • Prometheus
    • ServiceMonitor
    • Alertmanager
  • rbac/v1:
    • ClusterRole
    • ClusterRoleBinding
    • Role
    • RoleBinding
  • monitoring.appscode.com/v1alpha1:
    • ClusterAlert
    • NodeAlert
    • PodAlert
  • stash.appscode.com/v1alpha1:
    • Restic
    • Recovery
  • storage/v1:
    • StorageClass
  • voyager.appscode.com/v1beta1:
    • Certificate
    • Ingress

To add support for additional object types, please file an issue.

Disable Event Forwarder

If you would like to disable this feature, remove the eventForwarder portion of your Kubed cluster config. Then update the kubed-config Secret and restart Kubed operator pod(s).

Cleaning up

To cleanup the Kubernetes resources created by this tutorial, run the following commands:

$ kubectl delete pvc myclaim -n demo
persistentvolumeclaim "myclaim" deleted

$ kubectl delete ns demo
namespace "demo" deleted

To uninstall Kubed operator, please follow the steps here.

Next Steps

  • See the list of supported notifiers here.
  • Learn how to use Kubed to protect your Kubernetes cluster from disasters here.
  • Need to keep configmaps/secrets synchronized across namespaces or clusters? Try Kubed config syncer.
  • Out of disk space because of too much logs in Elasticsearch or metrics in InfluxDB? Configure janitors to delete old data.
  • Wondering what features are coming next? Please visit here.
  • Want to hack on Kubed? Check our contribution guidelines.

Take your team where it needs to go.

Create your cluster in minutes. Our team is here to help and would be happy to chat with you.