A Comprehensive Guide to Monitoring Linux Infrastructure with Prometheus and Grafana

Prometheus and Grafana have become a ubiquitous open source monitoring stack for modern infrastructures. This comprehensive guide will teach you how to install, configure and operate Prometheus and Grafana to monitor Linux infrastructure.

We will cover the following topics in-depth:

  • Installing Prometheus and Grafana on major Linux distros – CentOS, Ubuntu, Debian
  • Configuring Prometheus to scrape metrics from exporters
  • Using node_exporter to instrument Linux host metrics
  • Enabling additional node_exporter collectors
  • Setting up other major exporters – MySQL, Nginx, Redis, etc
  • Creating Grafana dashboards and visualizing metrics
  • Using Prometheus query language and analyzing metrics
  • Comparing Prometheus vs InfluxDB, Graphite and other tools
  • Prometheus instrumentation best practices – metrics and labels
  • Scaling out monitoring for multiple servers
  • Integrating logging data sources into Grafana
  • Monitoring Kubernetes clusters and containers
  • Sending alerts from Prometheus and Grafana
  • Troubleshooting common issues

We will install the latest versions of Prometheus and Grafana on CentOS 8. This also works similarly on RHEL 8, Fedora, AlmaLinux, Rocky Linux.

For Debian/Ubuntu, the commands would be mostly the same except using apt instead of yum.

Installing dependencies

Prometheus and Grafana require some additional dependencies to be installed first:

sudo yum install wget git httpd-tools -y

Downloading and Extracting

Next, download the latest stable versions using wget:

wget https://github.com/prometheus/prometheus/releases/download/v2.36.2/prometheus-2.36.2.linux-amd64.tar.gz
wget https://dl.grafana.com/oss/release/grafana-9.0.5-1.x86_64.rpm

tar xvfz prometheus-*.tar.gz

Configuring Prometheus

Before starting the Prometheus server, we need to create a user and configure some directories…

Now let‘s see how we can visualize Prometheus metrics in Grafana…