New to Kubed? Please start here.

Supported Notifiers

Kubed can send notifications via Email, SMS or Chat for various operations using appscode/go-notify library. To connect to these services, you need to create a Secret with the appropriate keys. Then pass the secret name to Kubed by setting notifierSecretName field in Kubed cluster config. Also, set clusterName to a meaningful name for you. This cluster name will be prefixed to any notification sent via Email/SMS/Chat so that you can identify the source easily.

Hipchat

To receive chat notifications in Hipchat, create a Secret with the following key:

NameDescription
HIPCHAT_AUTH_TOKENRequired Hipchat api access token. You can use room notification tokens, if you are planning to send notifications to a single room.
HIPCHAT_BASE_URLOptional Base url of Hipchat server
HIPCHAT_CA_CERT_DATAOptional PEM encoded CA certificate used by Hipchat server
HIPCHAT_INSECURE_SKIP_VERIFYOptional If set to true, skips SSL verification
HIPCHAT_NOTIFYOptional If set to true, a notification will be triggered
HIPCHAT_COLOROptional Set color of message sent. Possible values: yellow, green, red, purple, gray, random.
$ echo -n 'your-hipchat-auth-token' > HIPCHAT_AUTH_TOKEN
$ kubectl create secret generic notifier-config -n kube-system \
    --from-file=./HIPCHAT_AUTH_TOKEN
secret "notifier-config" created
apiVersion: v1
data:
  HIPCHAT_AUTH_TOKEN: eW91ci1oaXBjaGF0LWF1dGgtdG9rZW4=
kind: Secret
metadata:
  creationTimestamp: 2017-07-25T01:54:37Z
  name: notifier-config
  namespace: kube-system
  resourceVersion: "2244"
  selfLink: /api/v1/namespaces/kube-system/secrets/notifier-config
  uid: 372bc159-70dc-11e7-9b0b-080027503732
type: Opaque

Now, to receive notifications via Hipchat, configure receiver as below:

  • notifier: Hipchat
  • to: a list of chat room names

    clusterName: unicorn
    eventForwarder:
    receivers:
    - notifier: Hipchat
    to:
    - ops-alerts
    notifierSecretName: notifier-config
    

Stride

To receive chat notifications in Stride, create a Secret with the following key:

NameDescription
STRIDE_CLOUD_IDRequired You can get stride cloud_id from your organization’s URL
STRIDE_ROOM_TOKENOptional Stride room token

If you login into your stride web app, you will see an URL with this format https://app.stride.com/{{cloud ID}}/lobby. From here, you will get cloud_id.

$ echo -n 'your-stride-cloud-id' > STRIDE_CLOUD_ID
$ echo -n 'your-stride-room-token' > STRIDE_ROOM_TOKEN
$ kubectl create secret generic notifier-config -n demo \
    --from-file=./STRIDE_CLOUD_ID \
    --from-file=./STRIDE_ROOM_TOKEN
secret "notifier-config" created
apiVersion: v1
data:
  STRIDE_CLOUD_ID: eW91ci1zdHJpZGUtY2xvdWQtaWQ=
  STRIDE_ROOM_TOKEN: eW91ci1zdHJpZGUtcm9vbS10b2tlbg==
kind: Secret
metadata:
  creationTimestamp: 2017-07-25T01:54:37Z
  name: notifier-config
  namespace: demo
  resourceVersion: "2244"
  selfLink: /api/v1/namespaces/demo/secrets/notifier-config
  uid: 372bc159-70dc-11e7-9b0b-080027503732
type: Opaque

Now, to receiver notifications via Stride, configure receiver as below:

  • notifier: Stride
  • to: a single room name

    clusterName: unicorn
    eventForwarder:
    receivers:
    - notifier: Stride
    to:
    - ops-alerts
    notifierSecretName: notifier-config
    

Using Stride App

You can also use Stride App as client to send notification. This will allow you to send notifications to multiple rooms.

You need to do followings:

To receive chat notifications in stride using App, create a Secret with the following key:

NameDescription
STRIDE_CLOUD_IDRequired You can get stride cloud_id from your organization’s URL
STRIDE_CLIENT_IDOptional Stride app clientId
STRIDE_CLIENT_SECRETOptional Stride app clientSecret
$ echo -n 'your-stride-cloud-id' > STRIDE_CLOUD_ID
$ echo -n 'your-stride-app-client-id' > STRIDE_CLIENT_ID
$ echo -n 'your-stride-app-client-secret' > STRIDE_CLIENT_SECRET

$ kubectl create secret generic notifier-config -n demo \
    --from-file=./STRIDE_CLOUD_ID \
    --from-file=./STRIDE_CLIENT_ID \
    --from-file=./STRIDE_CLIENT_SECRET
secret "notifier-config" created
apiVersion: v1
data:
  STRIDE_CLOUD_ID: eW91ci1zdHJpZGUtY2xvdWQtaWQ=
  STRIDE_CLIENT_ID: eW91ci1zdHJpZGUtYXBwLWNsaWVudC1pZA==
  STRIDE_CLIENT_SECRET: eW91ci1zdHJpZGUtYXBwLWNsaWVudC1zZWNyZXQ=
kind: Secret
metadata:
  creationTimestamp: 2017-07-25T01:54:37Z
  name: notifier-config
  namespace: demo
  resourceVersion: "2244"
  selfLink: /api/v1/namespaces/demo/secrets/notifier-config
  uid: 372bc159-70dc-11e7-9b0b-080027503732
type: Opaque

Now, to receiver notifications via Stride App, configure receiver as below:

  • notifier: Stride
  • to: a list of room names where this app is installed

    clusterName: unicorn
    eventForwarder:
    receivers:
    - notifier: Stride
    to:
    - ops-alerts
    - dev-alerts
    notifierSecretName: notifier-config
    

Mailgun

To receive email notifications via Mailgun, create a Secret with the following keys:

NameDescription
MAILGUN_DOMAINRequired domain name for Mailgun account
MAILGUN_API_KEYRequired Mailgun API Key
MAILGUN_FROMRequired sender email address
MAILGUN_PUBLIC_API_KEYOptional Mailgun public API Key
$ echo -n 'your-mailgun-domain' > MAILGUN_DOMAIN
$ echo -n 'no-reply@example.com' > MAILGUN_FROM
$ echo -n 'your-mailgun-api-key' > MAILGUN_API_KEY
$ echo -n 'your-mailgun-public-api-key' > MAILGUN_PUBLIC_API_KEY
$ kubectl create secret generic notifier-config -n kube-system \
    --from-file=./MAILGUN_DOMAIN \
    --from-file=./MAILGUN_FROM \
    --from-file=./MAILGUN_API_KEY \
    --from-file=./MAILGUN_PUBLIC_API_KEY
secret "notifier-config" created
apiVersion: v1
data:
  MAILGUN_API_KEY: eW91ci1tYWlsZ3VuLWFwaS1rZXk=
  MAILGUN_DOMAIN: eW91ci1tYWlsZ3VuLWRvbWFpbg==
  MAILGUN_FROM: bm8tcmVwbHlAZXhhbXBsZS5jb20=
  MAILGUN_PUBLIC_API_KEY: bWFpbGd1bi1wdWJsaWMtYXBpLWtleQ==
kind: Secret
metadata:
  creationTimestamp: 2017-07-25T01:31:24Z
  name: notifier-config
  namespace: kube-system
  resourceVersion: "714"
  selfLink: /api/v1/namespaces/kube-system/secrets/notifier-config
  uid: f8e91037-70d8-11e7-9b0b-080027503732
type: Opaque

Now, to receive notifications via Mailgun, configure receiver as below: - notifier: Mailgun - to: a list of email addresses

clusterName: unicorn
eventForwarder:
  receivers:
  - notifier: Mailgun
    to:
    - ops-alerts@example.com
notifierSecretName: notifier-config

SMTP

To configure any email provider, use SMTP notifier. To receive email notifications via SMTP services, create a Secret with the following keys:

NameDescription
SMTP_HOSTRequired Host address of smtp server
SMTP_PORTRequired Port of smtp server
SMTP_INSECURE_SKIP_VERIFYRequired If set to true, skips SSL verification
SMTP_USERNAMERequired Username
SMTP_PASSWORDRequired Password
SMTP_FROMRequired Sender email address
$ echo -n 'your-smtp-host' > SMTP_HOST
$ echo -n 'your-smtp-port' > SMTP_PORT
$ echo -n 'your-smtp-insecure-skip-verify' > SMTP_INSECURE_SKIP_VERIFY
$ echo -n 'your-smtp-username' > SMTP_USERNAME
$ echo -n 'your-smtp-password' > SMTP_PASSWORD
$ echo -n 'your-smtp-from' > SMTP_FROM
$ kubectl create secret generic notifier-config -n kube-system \
    --from-file=./SMTP_HOST \
    --from-file=./SMTP_PORT \
    --from-file=./SMTP_INSECURE_SKIP_VERIFY \
    --from-file=./SMTP_USERNAME \
    --from-file=./SMTP_PASSWORD \
    --from-file=./SMTP_FROM
secret "notifier-config" created

To configure Kubed to send email notifications using a GMail account, set the Secrets like below:

$ echo -n 'smtp.gmail.com' > SMTP_HOST
$ echo -n '587' > SMTP_PORT
$ echo -n 'your-gmail-adress' > SMTP_USERNAME
$ echo -n 'your-gmail-password' > SMTP_PASSWORD
$ echo -n 'your-gmail-address' > SMTP_FROM

Now, to receive notifications via SMTP, configure receiver as below:

  • notifier: SMTP
  • to: a list of email addresses

    clusterName: unicorn
    eventForwarder:
    receivers:
    - notifier: SMTP
    to:
    - ops-alerts@example.com
    notifierSecretName: notifier-config
    

Twilio

To receive SMS notifications via Twilio, create a Secret with the following keys:

NameDescription
TWILIO_ACCOUNT_SIDRequired Twilio account SID
TWILIO_AUTH_TOKENRequired Twilio authentication token
TWILIO_FROMRequired Sender mobile number
$ echo -n 'your-twilio-account-sid' > TWILIO_ACCOUNT_SID
$ echo -n 'your-twilio-auth-token' > TWILIO_AUTH_TOKEN
$ echo -n 'your-twilio-from' > TWILIO_FROM
$ kubectl create secret generic notifier-config -n kube-system \
    --from-file=./TWILIO_ACCOUNT_SID \
    --from-file=./TWILIO_AUTH_TOKEN \
    --from-file=./TWILIO_FROM
secret "notifier-config" created
apiVersion: v1
data:
  TWILIO_ACCOUNT_SID: eW91ci10d2lsaW8tYWNjb3VudC1zaWQ=
  TWILIO_AUTH_TOKEN: eW91ci10d2lsaW8tYXV0aC10b2tlbg==
  TWILIO_FROM: eW91ci10d2lsaW8tZnJvbQ==
kind: Secret
metadata:
  creationTimestamp: 2017-07-26T17:38:38Z
  name: notifier-config
  namespace: kube-system
  resourceVersion: "27787"
  selfLink: /api/v1/namespaces/kube-system/secrets/notifier-config
  uid: 41f57a61-7229-11e7-af79-08002738e55e
type: Opaque

Now, to receive notifications via Twilio, configure receiver as below:

  • notifier: Twilio
  • to: a list of receiver mobile numbers

    clusterName: unicorn
    eventForwarder:
    receivers:
    - notifier: Twilio
    to:
    - +1-999-888-1234
    notifierSecretName: notifier-config
    

Slack

To receive chat notifications in Slack, create a Secret with the following keys:

NameDescription
SLACK_AUTH_TOKENRequired Slack legacy auth token
$ echo -n 'your-slack-auth-token' > SLACK_AUTH_TOKEN
$ kubectl create secret generic notifier-config -n kube-system \
    --from-file=./SLACK_AUTH_TOKEN
secret "notifier-config" created
apiVersion: v1
data:
  SLACK_AUTH_TOKEN: eW91ci1zbGFjay1hdXRoLXRva2Vu
kind: Secret
metadata:
  creationTimestamp: 2017-07-25T01:58:58Z
  name: notifier-config
  namespace: kube-system
  resourceVersion: "2534"
  selfLink: /api/v1/namespaces/kube-system/secrets/notifier-config
  uid: d2571817-70dc-11e7-9b0b-080027503732
type: Opaque

Now, to receive notifications via Slack, configure receiver as below:

  • notifier: Slack
  • to: a list of chat room names

    clusterName: unicorn
    eventForwarder:
    receivers:
    - notifier: Slack
    to:
    - '#ops-alerts'
    notifierSecretName: notifier-config
    

Plivo

To receive SMS notifications via Plivo, create a Secret with the following keys:

NameDescription
PLIVO_AUTH_IDRequired Plivo auth ID
PLIVO_AUTH_TOKENRequired Plivo authentication token
PLIVO_FROMRequired Sender mobile number
$ echo -n 'your-plivo-auth-id' > PLIVO_AUTH_ID
$ echo -n 'your-plivo-auth-token' > PLIVO_AUTH_TOKEN
$ echo -n 'your-plivo-from' > PLIVO_FROM
$ kubectl create secret generic notifier-config -n kube-system \
    --from-file=./PLIVO_AUTH_ID \
    --from-file=./PLIVO_AUTH_TOKEN \
    --from-file=./PLIVO_FROM
secret "notifier-config" created
apiVersion: v1
data:
  PLIVO_AUTH_ID: eW91ci1wbGl2by1hdXRoLWlk
  PLIVO_AUTH_TOKEN: eW91ci1wbGl2by1hdXRoLXRva2Vu
  PLIVO_FROM: eW91ci1wbGl2by1mcm9t
kind: Secret
metadata:
  creationTimestamp: 2017-07-25T02:00:02Z
  name: notifier-config
  namespace: kube-system
  resourceVersion: "2606"
  selfLink: /api/v1/namespaces/kube-system/secrets/notifier-config
  uid: f8dade1c-70dc-11e7-9b0b-080027503732
type: Opaque

Now, to receive notifications via Plivo, configure receiver as below:

  • notifier: Plivo
  • to: a list of receiver mobile numbers

    clusterName: unicorn
    eventForwarder:
    receivers:
    - notifier: Plivo
    to:
    - +1-999-888-1234
    notifierSecretName: notifier-config
    

Pushover.net

To receive push notifications via Pushover.net, create a Secret with the following keys:

NameDescription
PUSHOVER_TOKENRequired Pushover.net token.
PUSHOVER_USER_KEYRequired User key or group key.
PUSHOVER_TITLEOptional Message’s title, otherwise your app’s name is used.
PUSHOVER_URLOptional A supplementary URL to show with your message.
PUSHOVER_URL_TITLEOptional A title for the supplementary URL, otherwise just the URL is shown.
PUSHOVER_PRIORITYOptional Send as -2 to generate no notification/alert, -1 to always send as a quiet notification, 1 to display as high-priority and bypass the user’s quiet hours, or 2 to also require confirmation from the user.
PUSHOVER_SOUNDOptional The name of one of the sounds supported by device clients to override the user’s default sound choice.
$ echo -n 'your-pushover-token' > PUSHOVER_TOKEN
$ echo -n 'your-pushover-user-key' > PUSHOVER_USER_KEY
$ echo -n 'your-pushover-title' > PUSHOVER_TITLE
$ echo -n 'your-pushover-url' > PUSHOVER_URL
$ echo -n 'your-pushover-url-title' > PUSHOVER_URL_TITLE
$ echo -n 'your-pushover-priority' > PUSHOVER_PRIORITY
$ echo -n 'your-pushover-sound' > PUSHOVER_SOUND
$ kubectl create secret generic notifier-config -n kube-system \
    --from-file=./PUSHOVER_TOKEN \
    --from-file=./PUSHOVER_USER_KEY \
    --from-file=./PUSHOVER_TITLE \
    --from-file=./PUSHOVER_URL \
    --from-file=./PUSHOVER_URL_TITLE \
    --from-file=./PUSHOVER_PRIORITY \
    --from-file=./PUSHOVER_SOUND
secret "notifier-config" created
apiVersion: v1
data:
  PUSHOVER_PRIORITY: eW91ci1wdXNob3Zlci1wcmlvcml0eQ==
  PUSHOVER_SOUND: eW91ci1wdXNob3Zlci1zb3VuZA==
  PUSHOVER_TITLE: eW91ci1wdXNob3Zlci10aXRsZQ==
  PUSHOVER_TOKEN: eW91ci1wdXNob3Zlci10b2tlbg==
  PUSHOVER_URL: eW91ci1wdXNob3Zlci11cmw=
  PUSHOVER_URL_TITLE: eW91ci1wdXNob3Zlci11cmwtdGl0bGU=
  PUSHOVER_USER_KEY: eW91ci1wdXNob3Zlci11c2VyLWtleQ==
kind: Secret
metadata:
  creationTimestamp: 2017-08-04T05:13:07Z
  name: notifier-config
  namespace: kube-system
  resourceVersion: "33711872"
  selfLink: /api/v1/namespaces/kube-system/secrets/notifier-config
  uid: 99df75a8-78d3-11e7-acfa-42010af00141
type: Opaque

Now, to receive notifications via Pushover.net, configure receiver as below:

  • notifier: Pushover
  • to: a list of devices where notifications will be sent. If list is empty, all devices will be notified.

    clusterName: unicorn
    eventForwarder:
    receivers:
    - notifier: Pushover
    to:
    - my-phone
    notifierSecretName: notifier-config
    

Telegram

To receive notifications in your Telegram app, please follow the steps below:

  • Sign up for Telegram.
  • Now, use @botfather to create your own Telegram Bot and get the authorization token.
  • Create a Kubernetes Secret with the following keys:
NameDescription
TELEGRAM_TOKENRequired Telegram Bot authorization token.
$ echo -n 'your-telegram-bot-token' > TELEGRAM_TOKEN
$ kubectl create secret generic notifier-config -n kube-system \
    --from-file=./TELEGRAM_TOKEN
secret "notifier-config" created
apiVersion: v1
data:
  TELEGRAM_TOKEN: NDkxoooooooooooooooooxJ
kind: Secret
metadata:
  creationTimestamp: 2018-01-16T13:41:04Z
  name: notifier-config
  namespace: kube-system
  resourceVersion: "768"
  selfLink: /api/v1/namespaces/kube-system/secrets/notifier-config
  uid: e6066076-fac2-11e7-b3e7-0800276ee39b
type: Opaque
  • Add the Bot as an Admin for your channel where notifications will be sent.

telegram-bot

Now, to receiver notifications in your public Telegram channels, configure receiver as below:

  • notifier: Telegram
  • to: a list of channels where notifications will be sent.

    clusterName: unicorn
    eventForwarder:
    receivers:
    - notifier: Telegram
    to:
    - '@my-channel'
    notifierSecretName: notifier-config
    

To receive notifications if your private channel, follow the steps below:

  • Create a new public channel or change your private channel to public temporarily and assign a link.
  • Now, run the following command to determine the id for your channel.

    TOKEN=<token>
    curl -X POST https://api.telegram.org/bot${TOKEN}/sendMessage -d "chat_id=@channelusername&text=hello"
    
    {"ok":true,"result":{"message_id":10,"chat":{"id":-1001210429328,"title":"mytest","username":"mytest123489","type":"channel"},"date":1516103121,"text":"hello"}}
    
  • Make your channel private.

Now, to receiver notifications in your private Telegram channels, configure channle id as receiver like below:

clusterName: unicorn
eventForwarder:
  receivers:
  - notifier: Telegram
    to:
    - '-1001210429328'
notifierSecretName: notifier-config

Webhook Notifier

To receive chat notifications via Webhook, create a Secret with the following keys:

NameDescription
WEBHOOK_URLRequired URL of webhook server where notification is sent
WEBHOOK_USERNAMEOptional Username for basic auth
WEBHOOK_PASSWORDOptional Password for basic auth
WEBHOOK_TOKENOptional Token for bearer auth
WEBHOOK_CA_CERT_DATAOptional PEM encoded CA certificate used by Webhook server
WEBHOOK_CLIENT_CERT_DATAOptional PEM encoded client certificate used to authenticate to Webhook server
WEBHOOK_CLIENT_KEY_DATAOptional PEM encoded client private key used to authenticate to Webhook server
WEBHOOK_INSECURE_SKIP_VERIFYOptional If set to true, skips SSL verification of Webhook server certificate
$ echo -n '' > WEBHOOK_URL
$ echo -n '' > WEBHOOK_TOKEN
$ echo -n '' > WEBHOOK_CA_CERT_DATA
$ kubectl create secret generic notifier-config -n kube-system \
    --from-file=./WEBHOOK_URL \
    --from-file=./WEBHOOK_TOKEN \
    --from-file=./WEBHOOK_CA_CERT_DATA
secret "notifier-config" created
apiVersion: v1
data:
  WEBHOOK_URL: eW91ci1oaXBjaGF0LWF1dGgtdG9rZW4=
  WEBHOOK_TOKEN: eW91ci1oaXBjaGF0LWF1dGgtdG9rZW4=
  WEBHOOK_CA_CERT_DATA: eW91ci1oaXBjaGF0LWF1dGgtdG9rZW4=
kind: Secret
metadata:
  creationTimestamp: 2017-07-25T01:54:37Z
  name: notifier-config
  namespace: kube-system
  resourceVersion: "2244"
  selfLink: /api/v1/namespaces/kube-system/secrets/notifier-config
  uid: 372bc159-70dc-11e7-9b0b-080027503732
type: Opaque

Now, to receiver notifications via Webhook, configure receiver as below:

  • notifier: Webhook
  • to: a list of receiver names

    clusterName: unicorn
    eventForwarder:
    receivers:
    - notifier: Webhook
    to:
    - ops-alerts
    notifierSecretName: notifier-config
    

Using multiple notifiers

Kubed supports using different notifiers in different scenarios. First add the credentials for the different notifiers in the same Secret notifier-config and deploy that to Kubernetes. Then in the Kubed cluster config, specify the appropriate notifier for each feature.

apiVersion: v1
data:
  MAILGUN_API_KEY: eW91ci1tYWlsZ3VuLWFwaS1rZXk=
  MAILGUN_DOMAIN: eW91ci1tYWlsZ3VuLWRvbWFpbg==
  MAILGUN_FROM: bm8tcmVwbHlAZXhhbXBsZS5jb20=
  SLACK_AUTH_TOKEN: eW91ci1zbGFjay1hdXRoLXRva2Vu
kind: Secret
metadata:
  creationTimestamp: 2017-07-25T01:58:58Z
  name: notifier-config
  namespace: kube-system
  resourceVersion: "2534"
  selfLink: /api/v1/namespaces/kube-system/secrets/notifier-config
  uid: d2571817-70dc-11e7-9b0b-080027503732
type: Opaque


clusterName: unicorn
eventForwarder:
  warningEvents:
    handle: true
    namespaces:
    - kube-system
  receivers:
  - notifier: Mailgun
    to:
    - ops@example.com
  - notifier: Slack
    to:
    - '#ops-alerts'
notifierSecretName: notifier-config

Next Steps

  • Want to keep an eye on your cluster with automated notifications? Setup Kubed event forwarder.
  • Learn how to use Kubed to protect your Kubernetes cluster from disasters here.
  • Need to keep configmaps/secrets synchronized across namespaces or clusters? Try Kubed config syncer.
  • Out of disk space because of too much logs in Elasticsearch or metrics in InfluxDB? Configure janitors to delete old data.
  • Wondering what features are coming next? Please visit here.
  • Want to hack on Kubed? Check our contribution guidelines.