New to Searchlight? Please start here.
Supported Notifiers
Searchlight can send notifications via Email, SMS or Chat for alerts using appscode/go-notify library. To connect to these services, you need to create a Secret with the appropriate keys. Then pass the secret name to Searchlight by setting spec.notifierSecretName field in ClusterAlert/NodeAlert/PodAlert objects. This Secret must exist in the same namespace where the Alert object exists. To easily synchronize this Secret across all current and future namespaces of a Kubernetes cluster, you can use kubed.
Hipchat
To receive chat notifications in Hipchat, create a Secret with the following key:
| Name | Description |
|---|
| HIPCHAT_AUTH_TOKEN | Required Hipchat api access token. You can use room notification tokens, if you are planning to send notifications to a single room. |
| HIPCHAT_BASE_URL | Optional Base url of Hipchat server |
| HIPCHAT_CA_CERT_DATA | Optional PEM encoded CA certificate used by Hipchat server |
| HIPCHAT_INSECURE_SKIP_VERIFY | Optional If set to true, skips SSL verification |
$ echo -n 'your-hipchat-auth-token' > HIPCHAT_AUTH_TOKEN
$ kubectl create secret generic notifier-config -n demo \
--from-file=./HIPCHAT_AUTH_TOKEN
secret "notifier-config" created
apiVersion: v1
data:
HIPCHAT_AUTH_TOKEN: eW91ci1oaXBjaGF0LWF1dGgtdG9rZW4=
kind: Secret
metadata:
creationTimestamp: 2017-07-25T01:54:37Z
name: notifier-config
namespace: demo
resourceVersion: "2244"
selfLink: /api/v1/namespaces/demo/secrets/notifier-config
uid: 372bc159-70dc-11e7-9b0b-080027503732
type: Opaque
Now, to receiver notifications via Hipchat, configure receiver as below:
- notifier:
Hipchat - to: a list of chat room names
apiVersion: monitoring.appscode.com/v1alpha1
kind: ClusterAlert
metadata:
name: check-ca-cert
namespace: demo
spec:
check: ca_cert
vars:
warning: 240h
critical: 72h
checkInterval: 30s
alertInterval: 2m
notifierSecretName: notifier-config
receivers:
- notifier: Hipchat
state: CRITICAL
to: ["ops-alerts"]
Mailgun
To receive email notifications via Mailgun, create a Secret with the following keys:
| Name | Description |
|---|
| MAILGUN_DOMAIN | Required domain name for Mailgun account |
| MAILGUN_API_KEY | Required Mailgun API Key |
| MAILGUN_FROM | Required sender email address |
| MAILGUN_PUBLIC_API_KEY | Optional Mailgun public API Key |
$ echo -n 'your-mailgun-domain' > MAILGUN_DOMAIN
$ echo -n 'no-reply@example.com' > MAILGUN_FROM
$ echo -n 'your-mailgun-api-key' > MAILGUN_API_KEY
$ echo -n 'your-mailgun-public-api-key' > MAILGUN_PUBLIC_API_KEY
$ kubectl create secret generic notifier-config -n demo \
--from-file=./MAILGUN_DOMAIN \
--from-file=./MAILGUN_FROM \
--from-file=./MAILGUN_API_KEY \
--from-file=./MAILGUN_PUBLIC_API_KEY
secret "notifier-config" created
apiVersion: v1
data:
MAILGUN_API_KEY: eW91ci1tYWlsZ3VuLWFwaS1rZXk=
MAILGUN_DOMAIN: eW91ci1tYWlsZ3VuLWRvbWFpbg==
MAILGUN_FROM: bm8tcmVwbHlAZXhhbXBsZS5jb20=
MAILGUN_PUBLIC_API_KEY: bWFpbGd1bi1wdWJsaWMtYXBpLWtleQ==
kind: Secret
metadata:
creationTimestamp: 2017-07-25T01:31:24Z
name: notifier-config
namespace: demo
resourceVersion: "714"
selfLink: /api/v1/namespaces/demo/secrets/notifier-config
uid: f8e91037-70d8-11e7-9b0b-080027503732
type: Opaque
Now, to receiver notifications via Mailgun, configure receiver as below:
- notifier:
Mailgun - to: a list of email addresses
apiVersion: monitoring.appscode.com/v1alpha1
kind: ClusterAlert
metadata:
name: check-ca-cert
namespace: demo
spec:
check: ca_cert
vars:
warning: 240h
critical: 72h
checkInterval: 30s
alertInterval: 2m
notifierSecretName: notifier-config
receivers:
- notifier: Mailgun
state: CRITICAL
to: ["ops-alerts@example.com"]
SMTP
To configure any email provider, use SMTP notifier. To receive email notifications via SMTP services, create a Secret with the following keys:
| Name | Description |
|---|
| SMTP_HOST | Required Host address of smtp server |
| SMTP_PORT | Required Port of smtp server |
| SMTP_INSECURE_SKIP_VERIFY | Required If set to true, skips SSL verification |
| SMTP_USERNAME | Required Username |
| SMTP_PASSWORD | Required Password |
| SMTP_FROM | Required Sender email address |
$ echo -n 'your-smtp-host' > SMTP_HOST
$ echo -n 'your-smtp-port' > SMTP_PORT
$ echo -n 'your-smtp-insecure-skip-verify' > SMTP_INSECURE_SKIP_VERIFY
$ echo -n 'your-smtp-username' > SMTP_USERNAME
$ echo -n 'your-smtp-password' > SMTP_PASSWORD
$ echo -n 'your-smtp-from' > SMTP_FROM
$ kubectl create secret generic notifier-config -n demo \
--from-file=./SMTP_HOST \
--from-file=./SMTP_PORT \
--from-file=./SMTP_INSECURE_SKIP_VERIFY \
--from-file=./SMTP_USERNAME \
--from-file=./SMTP_PASSWORD \
--from-file=./SMTP_FROM
secret "notifier-config" created
To configure Searchlight to send email notifications using a GMail account, set the Secrets like below:
$ echo -n 'smtp.gmail.com' > SMTP_HOST
$ echo -n '587' > SMTP_PORT
$ echo -n 'your-gmail-adress' > SMTP_USERNAME
$ echo -n 'your-gmail-password' > SMTP_PASSWORD
$ echo -n 'your-gmail-address' > SMTP_FROM
Now, to receiver notifications via SMTP, configure receiver as below:
- notifier:
SMTP - to: a list of email addresses
apiVersion: monitoring.appscode.com/v1alpha1
kind: ClusterAlert
metadata:
name: check-ca-cert
namespace: demo
spec:
check: ca_cert
vars:
warning: 240h
critical: 72h
checkInterval: 30s
alertInterval: 2m
notifierSecretName: notifier-config
receivers:
- notifier: SMTP
state: CRITICAL
to: ["ops-alerts@example.com"]
NB: According to https://cloud.google.com/compute/docs/guides/sending-mail/ , Google Compute Engine does not allow outbound connections on ports 25, 465, and 587. So, try Mailgun to receive alert notifications via email with your GCE or GKE clusters.
Twilio
To receive SMS notifications via Twilio, create a Secret with the following keys:
| Name | Description |
|---|
| TWILIO_ACCOUNT_SID | Required Twilio account SID |
| TWILIO_AUTH_TOKEN | Required Twilio authentication token |
| TWILIO_FROM | Required Sender mobile number |
$ echo -n 'your-twilio-account-sid' > TWILIO_ACCOUNT_SID
$ echo -n 'your-twilio-auth-token' > TWILIO_AUTH_TOKEN
$ echo -n 'your-twilio-from' > TWILIO_FROM
$ kubectl create secret generic notifier-config -n demo \
--from-file=./TWILIO_ACCOUNT_SID \
--from-file=./TWILIO_AUTH_TOKEN \
--from-file=./TWILIO_FROM
secret "notifier-config" created
apiVersion: v1
data:
TWILIO_ACCOUNT_SID: eW91ci10d2lsaW8tYWNjb3VudC1zaWQ=
TWILIO_AUTH_TOKEN: eW91ci10d2lsaW8tYXV0aC10b2tlbg==
TWILIO_FROM: eW91ci10d2lsaW8tZnJvbQ==
kind: Secret
metadata:
creationTimestamp: 2017-07-26T17:38:38Z
name: notifier-config
namespace: demo
resourceVersion: "27787"
selfLink: /api/v1/namespaces/demo/secrets/notifier-config
uid: 41f57a61-7229-11e7-af79-08002738e55e
type: Opaque
Now, to receiver notifications via SMTP, configure receiver as below:
- notifier:
Twilio - to: a list of receiver mobile numbers
apiVersion: monitoring.appscode.com/v1alpha1
kind: ClusterAlert
metadata:
name: check-ca-cert
namespace: demo
spec:
check: ca_cert
vars:
warning: 240h
critical: 72h
checkInterval: 30s
alertInterval: 2m
notifierSecretName: notifier-config
receivers:
- notifier: Twilio
state: CRITICAL
to: ["1-999-888-1234"]
Slack
To receive chat notifications in Slack, create a Secret with the following keys:
$ echo -n 'your-slack-auth-token' > SLACK_AUTH_TOKEN
$ kubectl create secret generic notifier-config -n demo \
--from-file=./SLACK_AUTH_TOKEN
secret "notifier-config" created
apiVersion: v1
data:
SLACK_AUTH_TOKEN: eW91ci1zbGFjay1hdXRoLXRva2Vu
kind: Secret
metadata:
creationTimestamp: 2017-07-25T01:58:58Z
name: notifier-config
namespace: demo
resourceVersion: "2534"
selfLink: /api/v1/namespaces/demo/secrets/notifier-config
uid: d2571817-70dc-11e7-9b0b-080027503732
type: Opaque
Now, to receiver notifications via Hipchat, configure receiver as below:
- notifier:
Slack - to: a list of chat room names
apiVersion: monitoring.appscode.com/v1alpha1
kind: ClusterAlert
metadata:
name: check-ca-cert
namespace: demo
spec:
check: ca_cert
vars:
warning: 240h
critical: 72h
checkInterval: 30s
alertInterval: 2m
notifierSecretName: notifier-config
receivers:
- notifier: Slack
state: CRITICAL
to: ["#ops-alerts"]
Plivo
To receive SMS notifications via Plivo, create a Secret with the following keys:
| Name | Description |
|---|
| PLIVO_AUTH_ID | Required Plivo auth ID |
| PLIVO_AUTH_TOKEN | Required Plivo authentication token |
| PLIVO_FROM | Required Sender mobile number |
$ echo -n 'your-plivo-auth-id' > PLIVO_AUTH_ID
$ echo -n 'your-plivo-auth-token' > PLIVO_AUTH_TOKEN
$ echo -n 'your-plivo-from' > PLIVO_FROM
$ kubectl create secret generic notifier-config -n demo \
--from-file=./PLIVO_AUTH_ID \
--from-file=./PLIVO_AUTH_TOKEN \
--from-file=./PLIVO_FROM
secret "notifier-config" created
apiVersion: v1
data:
PLIVO_AUTH_ID: eW91ci1wbGl2by1hdXRoLWlk
PLIVO_AUTH_TOKEN: eW91ci1wbGl2by1hdXRoLXRva2Vu
PLIVO_FROM: eW91ci1wbGl2by1mcm9t
kind: Secret
metadata:
creationTimestamp: 2017-07-25T02:00:02Z
name: notifier-config
namespace: demo
resourceVersion: "2606"
selfLink: /api/v1/namespaces/demo/secrets/notifier-config
uid: f8dade1c-70dc-11e7-9b0b-080027503732
type: Opaque
Now, to receiver notifications via SMTP, configure receiver as below:
- notifier:
Plivo - to: a list of receiver mobile numbers
apiVersion: monitoring.appscode.com/v1alpha1
kind: ClusterAlert
metadata:
name: check-ca-cert
namespace: demo
spec:
check: ca_cert
vars:
warning: 240h
critical: 72h
checkInterval: 30s
alertInterval: 2m
notifierSecretName: notifier-config
receivers:
- notifier: Plivo
state: CRITICAL
to: ["1-999-888-1234"]
Pushover.net
To receive push notifications via Pushover.net, create a Secret with the following keys:
| Name | Description |
|---|
| PUSHOVER_TOKEN | Required Pushover.net token. |
| PUSHOVER_USER_KEY | Required User key or group key. |
| PUSHOVER_TITLE | Optional Message’s title, otherwise your app’s name is used. |
| PUSHOVER_URL | Optional A supplementary URL to show with your message. |
| PUSHOVER_URL_TITLE | Optional A title for the supplementary URL, otherwise just the URL is shown. |
| PUSHOVER_PRIORITY | Optional Send as -2 to generate no notification/alert, -1 to always send as a quiet notification, 1 to display as high-priority and bypass the user’s quiet hours, or 2 to also require confirmation from the user. |
| PUSHOVER_SOUND | Optional The name of one of the sounds supported by device clients to override the user’s default sound choice. |
$ echo -n 'your-pushover-token' > PUSHOVER_TOKEN
$ echo -n 'your-pushover-user-key' > PUSHOVER_USER_KEY
$ echo -n 'your-pushover-title' > PUSHOVER_TITLE
$ echo -n 'your-pushover-url' > PUSHOVER_URL
$ echo -n 'your-pushover-url-title' > PUSHOVER_URL_TITLE
$ echo -n 'your-pushover-priority' > PUSHOVER_PRIORITY
$ echo -n 'your-pushover-sound' > PUSHOVER_SOUND
$ kubectl create secret generic notifier-config -n demo \
--from-file=./PUSHOVER_TOKEN \
--from-file=./PUSHOVER_USER_KEY \
--from-file=./PUSHOVER_TITLE \
--from-file=./PUSHOVER_URL \
--from-file=./PUSHOVER_URL_TITLE \
--from-file=./PUSHOVER_PRIORITY \
--from-file=./PUSHOVER_SOUND
secret "notifier-config" created
apiVersion: v1
data:
PUSHOVER_PRIORITY: eW91ci1wdXNob3Zlci1wcmlvcml0eQ==
PUSHOVER_SOUND: eW91ci1wdXNob3Zlci1zb3VuZA==
PUSHOVER_TITLE: eW91ci1wdXNob3Zlci10aXRsZQ==
PUSHOVER_TOKEN: eW91ci1wdXNob3Zlci10b2tlbg==
PUSHOVER_URL: eW91ci1wdXNob3Zlci11cmw=
PUSHOVER_URL_TITLE: eW91ci1wdXNob3Zlci11cmwtdGl0bGU=
PUSHOVER_USER_KEY: eW91ci1wdXNob3Zlci11c2VyLWtleQ==
kind: Secret
metadata:
creationTimestamp: 2017-08-04T05:13:07Z
name: notifier-config
namespace: demo
resourceVersion: "33711872"
selfLink: /api/v1/namespaces/demo/secrets/notifier-config
uid: 99df75a8-78d3-11e7-acfa-42010af00141
type: Opaque
Now, to receiver notifications via Pushover.net, configure receiver as below:
- notifier:
Pushover - to: a list of devices where notifications will be sent. If list is empty, all devices will be notified.
apiVersion: monitoring.appscode.com/v1alpha1
kind: ClusterAlert
metadata:
name: check-ca-cert
namespace: demo
spec:
check: ca_cert
vars:
warning: 240h
critical: 72h
checkInterval: 30s
alertInterval: 2m
notifierSecretName: notifier-config
receivers:
- notifier: Pushover
state: CRITICAL
to: ["my-phone"]
Telegram
To receive notifications in your Telegram app, please follow the steps below:
- Sign up for Telegram.
- Now, use @botfather to create your own Telegram Bot and get the authorization token.
- Create a Kubernetes Secret with the following keys:
| Name | Description |
|---|
| TELEGRAM_TOKEN | Required Telegram Bot authorization token. |
$ echo -n 'your-telegram-token' > TELEGRAM_TOKEN
$ kubectl create secret generic notifier-config -n demo \
--from-file=./TELEGRAM_TOKEN
secret "notifier-config" created
apiVersion: v1
data:
TELEGRAM_TOKEN: NDkxoooooooooooooooooxJ
kind: Secret
metadata:
creationTimestamp: 2018-01-16T13:41:04Z
name: notifier-config
namespace: demo
resourceVersion: "768"
selfLink: /api/v1/namespaces/demo/secrets/notifier-config
uid: e6066076-fac2-11e7-b3e7-0800276ee39b
type: Opaque
- Add the Bot as an Admin for your channel where notifications will be sent.
Now, to receiver notifications in your public Telegram channels, configure receiver as below:
- notifier:
Telegram - to: a list of channels where notifications will be sent.
apiVersion: monitoring.appscode.com/v1alpha1
kind: ClusterAlert
metadata:
name: check-ca-cert
namespace: demo
spec:
check: ca_cert
vars:
warning: 240h
critical: 72h
checkInterval: 30s
alertInterval: 2m
notifierSecretName: notifier-config
receivers:
- notifier: Telegram
state: CRITICAL
to: ["@my-channel"]
To receive notifications if your private channel, follow the steps below:
- Create a new public channel or change your private channel to public temporarily and assign a link.
- Now, run the following command to determine the
id for your channel.
TOKEN=<token>
curl -X POST https://api.telegram.org/bot${TOKEN}/sendMessage -d "chat_id=@channelusername&text=hello"
{"ok":true,"result":{"message_id":10,"chat":{"id":-1001210429328,"title":"mytest","username":"mytest123489","type":"channel"},"date":1516103121,"text":"hello"}}
- Make your channel private.
Now, to receiver notifications in your private Telegram channels, configure channle id as receiver like below:
apiVersion: monitoring.appscode.com/v1alpha1
kind: ClusterAlert
metadata:
name: check-ca-cert
namespace: demo
spec:
check: ca_cert
vars:
warning: 240h
critical: 72h
checkInterval: 30s
alertInterval: 2m
notifierSecretName: notifier-config
receivers:
- notifier: Telegram
state: CRITICAL
to: ["-1001210429328"]
Using multiple notifiers
Searchlight supports using different notifiers in different states. First add the credentials for the different notifiers in the same Secret notifier-config and deploy that to Kubernetes. Then in the Alert object, specify the appropriate notifier for each feature.
apiVersion: v1
data:
MAILGUN_API_KEY: eW91ci1tYWlsZ3VuLWFwaS1rZXk=
MAILGUN_DOMAIN: eW91ci1tYWlsZ3VuLWRvbWFpbg==
MAILGUN_FROM: bm8tcmVwbHlAZXhhbXBsZS5jb20=
SLACK_AUTH_TOKEN: eW91ci1zbGFjay1hdXRoLXRva2Vu
kind: Secret
metadata:
creationTimestamp: 2017-07-25T01:58:58Z
name: notifier-config
namespace: demo
resourceVersion: "2534"
selfLink: /api/v1/namespaces/demo/secrets/notifier-config
uid: d2571817-70dc-11e7-9b0b-080027503732
type: Opaque
apiVersion: monitoring.appscode.com/v1alpha1
kind: ClusterAlert
metadata:
name: check-ca-cert
namespace: demo
spec:
check: ca_cert
vars:
warning: 240h
critical: 72h
checkInterval: 30s
alertInterval: 2m
notifierSecretName: notifier-config
receivers:
- notifier: Mailgun
state: WARNING
to: ["ops-alerts@example.com"]
- notifier: Slack
state: CRITICAL
to: ["#ops-alerts"]
Next Steps
- To periodically run various checks on your Kubernetes cluster, use ClusterAlerts.
- To periodically run various checks on nodes in a Kubernetes cluster, use NodeAlerts.
- To periodically run various checks on pods in a Kubernetes cluster, use PodAlerts.
- See the list of supported notifiers here.
- Wondering what features are coming next? Please visit here.
- Want to hack on Searchlight? Check our contribution guidelines.
