Grafana & Prometheus: Observability for your servers – Part 1

---

What is Grafana? What is Prometheus? And why is so important to know how to use it? In this post, we are going to learn about both technologies. We are going to install it with Docker and search for Dashboard creates by the community.

First, we are going to start differentiating monitoring from observability. Monitoring it is reviewing and analyzing a server, we are looking for availability, performance and security. And observability is the ability to measure the state of a server by its outputs.

Grafana as his web define “query, visualize, alert on, and understand your data no matter where it’s stored. With Grafana you can create, explore and share all of your data through beautiful, flexible dashboards.”. So with Grafana we are going to visualize with dashboards all the data we want from our server based on Prometheus. This is very important in the life of a server because we need to check the way the servers are working, also we can prevent incidents and security breaks sooner.

Prometheus it’s almost a standard in the Docker/Kubernetes monitoring services: “Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company.”

We are going to use node_exporter to obtain the server metrics, and cAdvisor for container metrics

So we are going to start editing our docker-compose file, I’m going to explain every step of the yaml on the same file:

version: "3.8"

#Storage for the data
volumes:
  grafana-data:
  prometheus-data:    

services:
  grafana:
    image: grafana/grafana
    container_name: grafana
    restart: unless-stopped
    volumes:
    - grafana-data:/var/lib/grafana
    ports:
    - 3000:3000

  prometheus:
    image: prom/prometheus
    container_name: prometheus
    restart: unless-stopped
    volumes:
    # Set the prometheus.yaml location (conf in the next block)
    - /path/to/prometheus.yml:/etc/prometheus/prometheus.yml
    - prometheus-data:/prometheus
    ports:
    - 9090:9090
    # Commands 
    command:
    - '--config.file=/etc/prometheus/prometheus.yml'
    - '--storage.tsdb.path=/prometheus'
    - '--storage.tsdb.retention.time=1y'
    - '--web.enable-lifecycle'

  node_exporter:
    image: quay.io/prometheus/node-exporter
    container_name: node_exporter
    restart: unless-stopped
    ports:
    - 9100:9100
    command:
    - '--path.rootfs=/host'
    pid: host
    volumes:
    - '/:/host:ro,rslave'
    
        
  cadvisor:
    image: gcr.io/cadvisor/cadvisor
    container_name: cadvisor
    restart: unless-stopped
    expose:
    - 8080
    volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro

# https://github.com/brancz/prometheus-example-app
  app_example:
    image: quay.io/brancz/prometheus-example-app:v0.3.0
    container_name: app_example
    restart: unless-stopped
    ports:
    - 80:8080

We also need the prometheus.yaml configuration file:

global:
  scrape_interval:     15s
# Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s 
# Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['prometheus:9090']

  - job_name: 'cadvisor'
    static_configs:
    - targets: ['cadvisor:8080']

  - job_name: 'node_exporter'
    static_configs:
    - targets: ['node_exporter:9100']

  - job_name: 'app_example'
    static_configs:
    - targets: ['app_example:8080']
    #metrics_path: '/metrics'

I’m going to be using Portainer, for this tutorial. We can avoid Portainer, but we recently learn how to use it, so:

Create a new stack on “Stacks”. First name it, then upload the docker-compose.yaml file or just copy the content. After that, click on “Deploy”

If the deployment works fine, we are going to start looking our example app. My server it’s working on 10.0.0.20 so now on, I’m going to be explaining with that IP :

---

After that if we see that our app it’s working good, we can now go to metrics, and we are going to find something like this:

Every “http_request…” are metrics from Prometheus. We are going to graph this on Grafana. The first time we open Grafana, we are going to introduce the default user: admin-admin, then we need to change the password.

Now in Grafana the first thing we need to do is linked Prometheus data source, we are going to click on “Add your first data source”:

Select Prometheus:

##This is a message from the future, you have to add “http://” because later when we need to create alerts, this is going to make an error and alerts aren’t going to be sent it:

On URL type “http://prometheus:9090″ that’s the way we declare it on our compose. Finally, click on “Save & test”:

If everything it’s okay, you are going to see a green box telling you the DB was successfully linked. Now It’s time to install some dashboards. We are going to navigate in Grafana Labs, finding the dashboard we like. In Grafana Labs, we are going to see there are a lot of dashboards already created by the community. You can create your own dashboards, but I think this is a topic for a other post. In this case, I’m going to search for node_exporter dashboards:

I found one that I like, we can check the number of reviews, downloads and screenshots to give us an idea of what the dashboard can offer. Then we are going to copy the ID of the dashboard:

Now we are going back to Grafana, click on “Create” (“+” icon) and click on “Import”. Paste the dashboard ID and click “Load”:

We are going to see that all the information about the dashboard we select. We need to point the Prometheus we are going to be using. On the bottom select “Prometheus” and click on Import:

And voilà! The complete dashboard ready to analyze:

But wait! In this first part we are going to install everything. Part 2 will be focus on checking some graphs and set some useful options. After installing node_exporter dashboard, we are going to find a dashboard for cAdvisor the same way. I’m going to be using 13946:

All these dashboards are very complex and have a lot of information, so we need to take it easy and studied every dashboard we need slowly. We also need to know that all information it’s running on the same server we are running Grafana & Prometheus, so we need to export all of this information to another safe place. If we click on “Search” we can see all the dashboard we imported:

---

So now we have installed one of the most professional combos for observability and monitoring. On part 2 we are going to see how to read all this data, how to back up it, how to create alert if something isn’t right, and much more. A lot of information was extracted from Pelado Nerd and Caos Binario youtube videos, if you speak spanish, I recommend these 2 great channels.