New to Voyager? Please start here.

Monitor HAProxy using Prometheus

This tutorial will show you how to monitor Voyager managed HAProxy pods using builtin Prometheus scraper.

Before You Begin

At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using Minikube.

Now, deploy Voyager operator following instructions here.

To keep things isolated, this tutorial uses a separate namespace called demo throughout this tutorial. Run the following command to prepare your cluster for this tutorial:

$ kubectl create namespace demo
namespace "demo" created

$ kubectl get ns
NAME          STATUS    AGE
default       Active    45m
demo          Active    10s
kube-public   Active    45m
kube-system   Active    45m

Note that the yaml files that are used in this tutorial, stored in docs/examples folder in GitHub repository appscode/voyager.

Create Ingress

We are going to use a nginx server as the backend. To deploy nginx server, run the following commands:

kubectl run nginx --image=nginx -n demo
kubectl expose deployment nginx --name=web --port=80 --target-port=80 -n demo

Now create Ingress ing.yaml

$ kubectl apply -f
ingress "stats-ing" created
kind: Ingress
  name: stats-ing
  namespace: demo
  annotations: 'NodePort' 'true' ''
  - host: voyager.appscode.test
      - path: /
          serviceName: web
          servicePort: 80

Voyager operator watches for Ingress objects using Kubernetes api. When a Ingress object is created, Voyager operator will create a new HAProxy deployment and a NodePort Service with name voyager-{ingress-name}. Since annotation was configured, a stats service object is configured accordingly. Here,

Keys Value Default Description bool "false" Required. If set, HAProxy stats will be exposed string Required. Indicates the monitoring agent used. Here built-in scraper in Prometheus is used to monitor the HAProxy pods. Voyager operator will configure the stats service in a way that the Prometheus server will automatically find out the service endpoint and scrape metrics from exporter.

You can verify it running the following commands:

$ kubectl get pods,svc -n demo
NAME                                    READY     STATUS    RESTARTS   AGE
po/nginx-8586cf59-r2m59                 1/1       Running   0          1m
po/voyager-stats-ing-5bf6b54949-5zs4x   2/2       Running   0          1m

NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)               AGE
svc/voyager-stats-ing         NodePort   <none>        80:31094/TCP          1m
svc/voyager-stats-ing-stats   ClusterIP   <none>        56789/TCP,56790/TCP   1m
svc/web                       ClusterIP   <none>        80/TCP                1m
$ kubectl get svc -n demo voyager-stats-ing-stats -o yaml

apiVersion: v1
kind: Service
  annotations: stats-ing / "56790" "true"
  creationTimestamp: 2018-02-25T21:48:24Z
    feature: stats
    origin: voyager
    origin-name: stats-ing
  name: voyager-stats-ing-stats
  namespace: demo
  resourceVersion: "317"
  selfLink: /api/v1/namespaces/demo/services/voyager-stats-ing-stats
  uid: 9ac02c7c-1a75-11e8-a133-080027640ad5
  - name: stats
    port: 56789
    protocol: TCP
    targetPort: stats
  - name: http
    port: 56790
    protocol: TCP
    targetPort: http
    origin: voyager
    origin-name: stats-ing
  sessionAffinity: None
  type: ClusterIP
  loadBalancer: {}

We can see that the service contains these specific annotations. The Prometheus server will discover the exporter using these specifications. ... ... ...

Deploy and configure Prometheus Server

The Prometheus server is needed to configure so that it can discover endpoints of services. If a Prometheus server is already running in cluster and if it is configured in a way that it can discover service endpoints, no extra configuration will be needed. If there is no existing Prometheus server running, rest of this tutorial will create a Prometheus server with appropriate configuration.

The configuration file to Prometheus-Server will be provided by ConfigMap. The below config map will be created:

apiVersion: v1
kind: ConfigMap
  name: prometheus-server-conf
    name: prometheus-server-conf
  namespace: demo
  prometheus.yml: |-
      scrape_interval: 5s
      evaluation_interval: 5s
    - job_name: 'kubernetes-service-endpoints'

      - role: endpoints

      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name
$ kubectl create -f
configmap "prometheus-server-conf" created

Now, the below yaml is used to deploy Prometheus in kubernetes :

apiVersion: apps/v1
kind: Deployment
  name: prometheus-server
  namespace: demo
  replicas: 1
      app: prometheus-server
        app: prometheus-server
        - name: prometheus
          image: prom/prometheus:v2.1.0
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
            - containerPort: 9090
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
        - name: prometheus-config-volume
            defaultMode: 420
            name: prometheus-server-conf
        - name: prometheus-storage-volume
          emptyDir: {}

Now, run the following command to deploy prometheus in kubernetes:

$ kubectl create -f
clusterrole "prometheus-server" created
serviceaccount "prometheus-server" created
clusterrolebinding "prometheus-server" created
deployment "prometheus-server" created
service "prometheus-service" created

Prometheus Dashboard

Now to open prometheus dashboard on Browser:

$ kubectl get svc -n demo
NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)               AGE
kubedb               ClusterIP      None             <none>        <none>                59m
mgo-mon-prometheus   ClusterIP    <none>        27017/TCP,56790/TCP   59m
prometheus-service   LoadBalancer   <pending>     9090:30901/TCP        8s

$ minikube ip

$ minikube service prometheus-service -n demo --url

Now, open your browser and go to the following URL: http://{minikube-ip}:{prometheus-svc-nodeport} to visit Prometheus Dashboard. According to the above example, this URL will be

Now, if you go the Prometheus Dashboard, you should see that the HAProxy pod as one of the targets.


Cleaning up

To cleanup the Kubernetes resources created by this tutorial, run:

$ kubectl delete ns demo
namespace "demo" deleted

Take your team where it needs to go.

Create your cluster in minutes. Our team is here to help and would be happy to chat with you.