New to Searchlight? Please start here.
A PodAlert is a Kubernetes Custom Resource Definition (CRD). It provides declarative configuration of Icinga services for Pods in a Kubernetes native way. You only need to describe the desired check command and notifier in a PodAlert object, and the Searchlight operator will create Icinga2 hosts, services and notifications to the desired state for you.
As with all other Kubernetes objects, a PodAlert needs apiVersion, kind, and metadata fields. It also needs a .spec section. Below is an example PodAlert object.
apiVersion: monitoring.appscode.com/v1alpha1
kind: PodAlert
metadata:
name: nginx-webstore
namespace: demo
spec:
selector:
matchLabels:
app: nginx
check: pod-volume
vars:
volumeName: webstore
warning: '70'
critical: '95'
checkInterval: 5m
alertInterval: 3m
notifierSecretName: notifier-config
receivers:
- notifier: Mailgun
state: Warning
to: ["ops@example.com"]
- notifier: Twilio
state: Critical
to: ["+1-234-567-8901"]
This object will do the followings:
app=nginx in demo namespace.pod-volume will be applied on volume named webstore.Warning state and emails will be sent to ops@example.com via Mailgun as notification.Critical state and SMSes will be sent to +1-234-567-8901 via Twilio as notification.Any PodAlert object has 3 main sections:
Any PodAlert can specify pods in 2 ways:
spec.podName can be used to specify a pod by name. Use this if you are creating pods directly.
spec.selector is a label selector for pods. This should be used if pods are created by workload controllers like Deployment, ReplicaSet, StatefulSet, DaemonSet, ReplicationController, etc. Searchlight operator will update Icinga as pods with matching labels are created/deleted by workload controllers.
Check commands are used by Icinga to periodically test some condition. If the test return positive appropriate notifications are sent. The following check commands are supported for pods: - pod-exec - To check Kubernetes exec command. Returns OK if exit code is zero, otherwise, returns Critical - pod-status - To check Kubernetes pod status. - pod-volume - To check Pod volume usage stat.
Each check command has a name specified in spec.check field. Optionally each check command can take one or more parameters. These are specified in spec.vars field. To learn about the available parameters for each check command, please visit their documentation. spec.checkInterval specifies how frequently Icinga will perform this check. Some examples are: 30s, 5m, 6h, etc.
When a check fails, Icinga will keep sending notifications until acknowledged via IcingaWeb dashboard. spec.alertInterval specifies how frequently notifications are sent. Icinga can send notifications to different targets based on alert state. spec.receivers contains that list of targets:
| Name | Description |
|---|---|
spec.receivers[*].state | Required Name of state for which notification will be sent |
spec.receivers[*].to | Required To whom notifications will be sent |
spec.receivers[*].method | Required How this notification will be sent |
You can skip this section if you are unfamiliar with how Icinga works. Searchlight operator watches for PodAlert objects and turns them into Icinga objects accordingly. For each Kubernetes Pod which has an PodAlert configured, an Icinga Host is created with the name {namespace}@pod@{pod-name} and address matching the IP of the Pod. Now for each PodAlert, an Icinga service is created with name matching the PodAlert name.
You can pause a PodAlert by setting spec.pause to true. If you already have a PodAlert created, you can edit it to set spec.pause. Searchlight operator will delete all Icinga Services related to this PodAlert. That’s how, periodical checks by Icinga will be stopped.
spec:
pause: true
You can resume the process again by setting spec.pause to false. Then Searchlight operator will create Icinga Services again for this PodAlert.