fbpx
Unlock the power of choice with CloudPlex and DigitalOcean Get CloudPlex free for 3 months, and Digital Ocean for 1 month

How to use Prometheus and Grafana to Monitor Kubernetes – Part 2

tutorial
If you are new to Kubernetes, Prometheus, Grafana, and monitoring Kubernetes using these tools. We highly recommend you go through the first blog post of this series How to use Prometheus and Grafana to Monitor Kubernetes – Part 1 before moving forward. In this part, we are going to move to the installation steps.

Deploying Prometheus and Grafana to Kubernetes:

We will start from the very start and install Prometheus on our Kubernetes cluster. We will then continue to explore our current dashboard and learn how to create a new one. The construction of dashboards would then lead to new metrics. We will discuss which of the metrics are the most important and how to customize them. Hint: most of them are very important, and you should at least familiarize yourself with them all.

The very first step to using Kubernetes with Grafana and Prometheus is to install and run a Kubernetes cluster. You can run any of the clusters, but in this tutorial, we are using GKE.

There are different ways through which you can install Prometheus on your Kubernetes cluster.

To deploy Grafana and Prometheus, we will use Helm. And a Helm component called Helm charts will help in defining, upgrading, and installing Kubernetes applications.

To install Helm, you need to first download the latest version of it and then unpack it. You need to then move “Helm Binary” to “/usr/local/bin/helm”. You need to install it according to your operating system.

Now, using Helm, in a different namespace, we will deploy our Prometheus using the command below:

kubectl create ns monitoring

Next, create a different namespace and deploy Prometheus. And then will install the Prometheus operator:

helm install  [stable/prometheus-operator] --namespace monitor

where:

  • prometheus-operation: is the name of the release, and it can be changed as you want.
  • monitor: is the namespace where we will deploy the operator.
  • stable/ prometheus-operator: It is the name of the Helm chart.

Verify your installation using:

kubectl get pods -n monitoring

Now your pods are up and running successfully. And we have an option to run Prometheus from the local machines with:

kubectl port-forward -n monitoring prometheus-prometheus-operation-prometheus-0 9090

To see the Prometheus dashboard and confirm your Prometheus server is up and running ok, use this URL: http://127.0.0.1:9090

Similar to what we just did to make Prometheus accessible, do the same to make the Grafana dashboard accessible:

kubectl port-forward $(kubectl get pods --selector=app=grafana -n monitoring --output=jsonpath="{.items..metadata.name}") -n monitor 3000

You can then view your Grafana dashboards with http://127.0.0.1:3000

There are going to be multiple dashboards, and to proceed further from this point, you need to go with the “admin” and “prom-operation” as a password, and they can be easily found in the secret object. The object will provide you a coded username and password, and you will have to decode it then to make use of it.

Grafana Preconfigured dashboards:

Grafana has numerous, useful preconfigured dashboards. Once done with the installation, some of the dashboards below will be visible on the Grafana homepage. You can select and use any of them to avoid having to create dashboards from scratch.

  • Kubernetes / Compute Resources / Cluster
  • Kubernetes / Nodes
  • Kubernetes / Networking / Namespace (Workload)
  • Kubernetes / Networking / Namespace (Pods)
  • Kubernetes / Nodes

Prometheus dashboard:

Let’s now delve further into how Prometheus works, using the dashboard we just created.

First set up port-forwarding for the Prometheus dashboard on Kubernetes using this command:

kubectl port-forward -n monitoring prometheus-prometheus-operation-prometheus-0 9090

Next, navigate to http://127.0.0.1:9090/metrics

Prometheus uses the PromQL language, which is a Prometheus-specific query language. It is a query language that lets you aggregate time-series data in real-time. PromQL needs time and effort to master, but you can grasp the basics fairly quickly.

It uses the four basic data types: scalar, string, range vector, string vector with other literals.

The stable Helm Chart we used for the Prometheus deployment also installed the following:

  • Grafana
  • kube-controller-manager
  • kube-dns/coredns
  • Prometheus
  • Prometheus operator
  • Node exporter
  • Kube proxy
  • Etcd
  • Kube scheduler
  • Alertmanager
  • Service monitors
  • Kube state Metrics
  • Kube apiserver

Watching Metrics in production

Grafana’s data source is now set as Prometheus. The next question is, what metrics do we want to watch, and how?

The default dashboards in Grafana make life easier as they are already named according to their actions. Kubernetes mostly runs more than one node, and it is important to configure monitoring for your nodes one at a time.

If none of the preconfigured dashboards works for you, create your own dashboard using the Grafana dashboard interface or using a configMap manifest. In fact it doesn’t have to be a dashboard; you can choose between a dashboard for visualization or query. If you want to go with visualization, you will then be asked to select your graph type and add a data source. You can either add Prometheus as the data source or a PromQL query that will return the necessary data.

While setting up the name of the dashboard, you can also set up alerts. For example, Resident Set Size is the portion of memory which holds the process in RAM. It is a significant metric for Kubernetes; hence this is a good candidate to set up alerts for. While monitoring your system containers, there are chances for Resident Set Size to be overpopulated.

Your container’s total number of restarts is another metric to watch closely.

Some other useful and common metrics you should monitor are:

  • Kubernetes Container CPU and Memory Usage
  • Kubernetes Pod CPU and Memory Usage
  • Kubernetes Node CPU and Memory Usage
  • Kubernetes Resource Capacity (total node CPU capacity, total cluster memory, total cluster disk capacity, etc)
  • Kubernetes Resource Utilization (comparing Resource Usage to total available Capacity)

There are, of course, hundreds of other metrics you can watch, but this depends on your very use case. Make sure to choose the most important metrics to watch, depending on your technical requirements, business use case and goals.

Asad Faizi

Founder CEO

CloudPlex.io, Inc

asad@cloudplex.io

Start building app

Start building your cloud native application

111540cookie-checkHow to use Prometheus and Grafana to Monitor Kubernetes – Part 2