How To Setup Prometheus (Operator) and Grafana Monitoring on Kubernetes

This tutorial shows you how to begin a modern monitoring stack with Prometheus (Operator) and Grafana on Kubernetes

Guillaume Vincent
Better Programming

--

Tourist viewer so they can see the city from afar
Photo by Matthijs van Schuppen on Unsplash

Monitoring is a key pillar of DevOps best practices. This gives vital information on the performance and health of a platform. In a distributed environment such as Kubernetes and microservices, it is even more true.

One of the great strengths of Kubernetes is the ability to scale your services and applications. When you reach thousand of applications, monitoring them manually or using scripts is not viable. You need to adopt a monitoring system that scales too! That’s where Prometheus and Grafana come into the scene.

Prometheus will collect, store and allow you to leverage your platform metrics. On the other hand, Grafana will plug into Prometheus and allow you to create beautiful dashboards and charts.

Today, we’ll go over what Prometheus is and the best way to deploy it on Kubernetes — using the operator. We’ll see how to set up a monitoring platform with Prometheus and Grafana.

This tutorial will give you a good starting point with observability and go further!

Prometheus

Timeline

Prometheus is a free and open source application for event monitoring and alerting built at SoundCloud in 2012. Soon after, many companies and organizations adopted it and contributed. In 2016, the Cloud Native Computing Foundation (CNCF) incubated the Prometheus project after Kubernetes.

The following timeline shows the evolution of the Prometheus project:

2012 — built at SoundCloud 2013 — use in SoundCloud’s production May 2016 — CNCF incubated Prometheus July 2016 — Prometheus 1.0 released Nov 2017 — Prometheus 2.0 released Aug 2018 — CNCF graduated Prometheus production-ready
The Prometheus project timeline

Concepts

Prometheus is considered the default monitoring solution for Kubernetes and was inspired by Borgman from Google. It collects metrics from applications and infrastructure using the HTTP pull requests. Targets are discovered via service discovery or static configuration. Pushing time series is also supported via an intermediary gateway.

Example of a metric exposes to Prometheus over HTTP
Example of a metric exposes to Prometheus over HTTP

Metrics exposed by a Prometheus target has the following format: <metric name>{<label name>=<label value>, ...}

Prometheus records real-time metrics in a time-series database (TSDB), which brings dimensional data models, operational simplicity, and scalable data collection. It provides PromQL, a flexible query language to leverage this dimensionality.

Architecture

The Prometheus architecture from https://prometheus.io/

The architecture diagram shows Prometheus is a multi-component monitoring system. The following pieces are integrated into the Prometheus deployment:

  • Prometheus server scraping and stores time-series data. It also provides a user interface to query the metrics.
  • Client libraries for instrumenting application code.
  • Pushgateway supports metrics collection from short-lived jobs
  • Exporters for services that do not instrument Prometheus metrics directly.
  • Alertmanager handles real-time alerts based on triggers

Why Choose the Prometheus Operator?

Kubernetes provides a lot of objects to deploy your applications: Pod, Deployment, Service, Ingress, etc… These are native resources, and as they are generic they do not behave like the final application. Kubernetes allows you to create custom resources through Custom Resource Definition (CRD).

CRD objects implement the behavior of the final application. This allows better maintainability and reduces the deployment effort. When using the Prometheus Operator, each component of the architecture comes from a CRD. This makes the Prometheus setup more straightforward than a classical installation.

In the classic Prometheus installation, adding new metric endpoints requires an update of the server configuration. This allows registering a new endpoint as a target to collect metrics. The Prometheus Operator uses Monitor objects (PodMonitor, ServiceMonitor) to discover dynamically endpoints and scrape metrics.

Using the Prometheus Operator gains you time on the Prometheus installation and maintainability. It provides you Monitor objects to dynamically collect metrics without updating the Prometheus configuration.

Deploying Prometheus With the Operator

The kube-prometheus-stack is a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules. It provides easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Operator.

This collection is available and deployable through a Helm chart. You can deploy the monitoring stack in a single command line. New to Helm? Have a look at this tutorial to get started!

Installing Helm

$ brew install helm

Creating the dedicated monitoring namespace

In Kubernetes, namespaces provide a mechanism for isolating groups of resources within a single cluster. We create a namespace named monitoring to prepare the new deployment:

$ kubectl create namespace monitoring

Installing kube-prometheus-stack with Helm

Add the Prometheus chart repository and update the local cache with these commands:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 
$ helm repo update

Deploy the kube-stack-prometheus chart in the namespace monitoring with Helm:

$ helm upgrade --namespace monitoring --install kube-stack-prometheus prometheus-community/kube-prometheus-stack --set prometheus-node-exporter.hostRootFsMount.enabled=false

hostRootFsMount.enabled is set to false to work on Docker Desktop on Mac

CRDs are now installed in the namespace. You can verify this with the following kubectl command:

$ kubectl get -n monitoring crds                                                           
NAME CREATED AT
alertmanagerconfigs.monitoring.coreos.com 2022-03-15T10:54:41Z
alertmanagers.monitoring.coreos.com 2022-03-15T10:54:42Z
podmonitors.monitoring.coreos.com 2022-03-15T10:54:42Z
probes.monitoring.coreos.com 2022-03-15T10:54:42Z
prometheuses.monitoring.coreos.com 2022-03-15T10:54:42Z
prometheusrules.monitoring.coreos.com 2022-03-15T10:54:42Z
servicemonitors.monitoring.coreos.com 2022-03-15T10:54:42Z
thanosrulers.monitoring.coreos.com 2022-03-15T10:54:42Z

Here is what we have running now in the namespace:

$ kubectl get pods -n monitoring                                                                    
NAME READY STATUS RESTARTS AGE
alertmanager-kube-stack-prometheus-kube-alertmanager-0 2/2 Running 0 2m36s
kube-stack-prometheus-grafana-6994bd6c69-h6s9z 3/3 Running 0 13h
kube-stack-prometheus-kube-operator-86667b5cdf-cqndt 1/1 Running 0 13h
kube-stack-prometheus-kube-state-metrics-fc9878699-dpgh6 1/1 Running 0 13h
kube-stack-prometheus-prometheus-node-exporter-vrjsl 1/1 Running 0 13h
prometheus-kube-stack-prometheus-kube-prometheus-0 2/2 Running 0 13h

The chart has installed Prometheus components, Operator, Grafana, and the following exporters:

Our monitoring stack with Prometheus and Grafana is up and ready!

Connecting to Prometheus’s Web Interface

The Prometheus web UI is accessible through port-forward with this command:

$ kubectl port-forward --namespace monitoring svc/kube-stack-prometheus-kube-prometheus 9090:9090

Opening a browser tab on http://localhost:9090 shows the Prometheus web UI. We can retrieve the metrics collected from exporters:

Dashboard in the Prometheus web UI showing time-series metrics
Dashboard in the Prometheus web UI showing time-series metrics

Going to the “Status>Targets” and you can see all the metric endpoints discovered by the Prometheus server:

The metric endpoints discovered by Prometheus are visible in Status>Targets menu
The metric endpoints discovered by Prometheus are visible in Status>Targets menu
Few target endpoints discovered by Prometheus server
Few target endpoints discovered by Prometheus server

Connecting To Grafana

The credentials to connect to the Grafana web interface are stored in a Kubernetes Secret and encoded in base64. We retrieve the username and password with these two commands:

$ kubectl get secret --namespace monitoring kube-stack-prometheus-grafana -o jsonpath='{.data.admin-user}' | base64 -d 
$ kubectl get secret --namespace monitoring kube-stack-prometheus-grafana -o jsonpath='{.data.admin-password}' | base64 -d

We create the port-forward to Grafana with the following command:

$ kubectl port-forward --namespace monitoring svc/kube-stack-prometheus-grafana 8080:80

Open your browser and go to http://localhost:8080 and fill in previous credentials:

The Grafana login interface
The Grafana login interface

The kube-stack-prometheus deployment has provisioned Grafana dashboards:

List of the provisioned Grafana dashboards
List of the provisioned Grafana dashboards

Here we can see one of them showing compute resources of Kubernetes pods:

A Grafana dashboard showing Kubernetes pod compute resources

All right, that’s how you set up Prometheus and Grafana monitoring on Kubernetes.

Thanks for reading! 🙏 I hope this article was helpful. Want to learn more about DevOps? Check out more articles like this one down below! ⬇️

--

--