Monitor Servers/Serices Via Grafana & Prometheus

David(Zhen Zhang)

2020-11-22

Focusing Grafana and Prometheus For years since 2014 when I was working for Sony Mobile, and built a service monitor soltion for them.

These days, I built a dashboard for my homeserver and services again, let me record it, a bit long.

Frist look:

Install Prometheus & Grafana

Install them on my home server via docker

Install Grafana

1 2	docker run -d --name=grafana -p 3000:3000 grafana/grafana-oss:8.1.3

prepare prometheus.yml at somewhere

global:
  scrape_interval:     15s
  evaluation_interval: 15s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

Install Prometheus

1	docker run -p 9090:9090 -v /a_path_for_prometheus/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Have a look Servers

Grafana (http://your-server:3000/), you will be asked to set username and password first time.

Prometheus (http://your-server:9090/)

Install Prometheus Exporters and Setup

Proxmox(PVE) Server

Create an api user type

1 2	veum roleadd Monitoring pveum rolemod Monitoring --privs Sys.Modify,Sys.Audit,VM.Monitor,Datastore.Audit,VM.Audit

Add users named “monitoring” and “prometheus”

pveum useradd monitoring@pve --comment "The ICINGA 2 monitoring user"
pveum useradd prometheus@pve --comment "The prometheus monitoring user"
pveum passwd prometheus@pve
pveum passwd monitoring@pve

assign role to above users

1 2	pveum aclmod / -user monitoring@pve -role Monitoring pveum aclmod / -user prometheus@pve -role Monitoring

Install python and pip, via apt or other way as you wish

1	apt-get install -y python python-pip

Install prometheus-pve-exporter

1	pip install prometheus-pve-exporter

Setup systemd service

[Unit]
Description=Prometheus exporter for Proxmox VE

[Service]
Restart=always
User=prometheus@pve
#ExecStart=/opt/prometheus-pve-exporter/bin/pve_exporter /etc/prometheus/pve.yml
ExecStart= /usr/local/bin/pve_exporter /etc/prometheus/pve.yml

[Install]
WantedBy=multi-user.target

Config file

1 2	mkdir -p /etc/prometheus-pve-exporter/ nano /etc/prometheus-pve-exporter/pve.yml

update pve.yml as following:

default:
    user: prometheus@pve
    password: <your password>
    verify_ssl: false

Start prometheus-pve-exporter service

1 2	systemctl daemon-reload systemctl start prometheus-pve-exporter

Added a job to Prometheus Server

scrape_configs:
  - job_name: 'pve-xxx'
    static_configs:
      - targets:
        - xx.xx.xx.xx:9221  # Proxmox VE node with PVE exporter.

Restart Prometheus Server

Check the export server and it on Prometheus server

open browser, goto http://xx.xx.xx.xx:9221/pve, then you can see:

access your Prometheus http://xx.xx.xx.xx:9090/targets, it should be listed.

Setup/Import Grafana Dashboard

If you want to copy or modify a dashboard from other, find it on Grafana Hub (https://grafana.com/grafana/dashboards/)

Because we are using prometheus service(collects metrics from server, only choose the “prometheus”, but others )

find one your favorite, copy it’s ID, or copy the json file.

Goto the management page of your Grafana, and import it

paste the ID, or paste the json content above, then load.

Choose your server, it will filter fitable servers automatically.

Then, you can modify as you wish, wow, get a rest now :).

Exporters/Dashboard for PM2, Nginx, Docker

If you wana monitor a server or service, just find google the name with prometheus exporter, maybe someone have share it and find a dashboard exist on Grafana hub, otherwise you have to create it yourself.

Reflection

I was spending weeks building dashboards(same solution as above) for company projects years ago, nowadays only cost an afternoon(Thanks guys who opensource these exporters and dashboards).