Prometheus and Grafana may as well be buddies in monitoring, especially in DevOps, where system monitoring is a major matter. This article shall focus on some basic points regarding Prometheus and Grafana, as well as configuration methods for the monitoring of a Jenkins system on Kubernetes.

Prometheus and Grafana

Prometheus is an open source tool for surveillance and alarms, and was first developed by SoundCloud, before becoming open source and community-developed.

While Prometheus is mostly a metrics gathering tool, Grafana is used to potray those data in charts, thus providing a visual representation.

Example of Grafana display:

For theoretical details on Prometheus and Grafana are readily available on Google, this article will take a more practical approach.

Setup Prometheus and Grafana

Installation and configuration of Prometheus and Grafana for monitoring a 1 Jenkins cluster.

In this practical example, Prometheus and Grafana are installed with 1 cluster K8s. and Prometheus shall monitor the Jenkins cluster’s various statuses including node status, namespace, memory usage…

Data gathered by Prometheus shall be presented on Grafana. Should any problem arisen, the system shall issue alarms, push them to Altermanager, and notify users via channels like Slack or Webhook.

Assume that Jenkins (Master + slaves) is already installed on a cluster K8s. You can use this very cluster for installation of Prometheus and Grafana.

In reality, instead of manual installation, projects will utilize Infra as Code.

On K8S, Prometheus can be installed with a yaml file as below:

The most important file is *-deployment, which defines Prometheus metrics including images, port, volume.

After getting connected to cluster, start Prometheus deployment on k8s using the following command:

kubectl apply –f prometheus –n monitoring

Prometheus shall be deployed as 1 k8s pod and service like below:

Prometheus service

apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: "true"
  name: prometheus-service
  namespace: monitoring
  labels:
    name: prometheus
spec:
  selector:
    app: prometheus-server
  ports:
    - name: prometheus
      port: 8080
      targetPort: prometheus
  #type: LoadBalancer

Grafana shall then present metrics derived from Prometheus service.

Repeat the same procedure with grafana.

Like Prometheus, grafana shall also be deployed on k8s as a docker container, port 3000.

Pod grafana will be exposed with service LoadBalancer.

apiVersion: v1
kind: Service
metadata:
  labels:
    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
    # If you are NOT using this as an addon, you should comment out this line.
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: grafana
 #   cloud.google.com/load-balancer-type: "Internal"
  name: grafana
  namespace: monitoring
spec:
  type: LoadBalancer
  loadBalancerSourceRanges:
  - 10.10.43.0/23

You can also limit access with loadBalancerSourceRanges sets using CIDIR.

For pod and Jenkin K8S’s node monitoring, you can use a kube-state-metrics Add-on agent to generate and expose cluster-level metrics. i.e. an add-on agent that can gather information metrics on clusters (pod, node…)

Use similar configurations for Alertmanager.

There is a definition called “config map” in K8S, which means mutual config information between pods, used for separation of codes and configuration files.

After installation you may see services and pods installed on k8s.

Now log in on grafana using the above IP/Port for status monitoring of Jenkins clusters, including memory usage, container status….

When problems arise in the system, Alertmanager will send alarms via channels like mail or slack.

Do Trong Nguyen 

Related posts: