How To Monitor Linux Servers Using Prometheus Node Exporter

There are multiple exporters that can be used to collect all the Linux server related metrics and statistics for monitoring. However, Node is the best way to monitor Linux Servers Using Prometheus

In this article we will learn how to setup Prometheus node exporter. This will be used to export all node level from the Prometheus server.

Table of Contents

Pre-requisites:

Prometheus Server: In order to use the Prometheus Node Exporter you need a Prometheus Server running, please see the Prometheus setup guide for Linux.
Port 9100: Opened in server firewall as Prometheus reads metrics on this port.

Setup Node Exporter Binary

Step 1: Go to the official download page for the latest version and use wget to download the latest node exporter package.

wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz

Step 2: Unpack it

tar -xvf node_exporter-1.0.1.linux-amd64.tar.gz

Step 3: Move the node export binary to /usr/local/bin

mv node_exporter-1.0.1.linux-amd64/node_exporter /usr/local/bin/

Create a Custom Node Exporter Service

Step 1: Create a node_exporter user to run the node exporter service.

useradd -rs /bin/false node_exporter

Step 2: Create a node_exporter service file under systemd.

vi /etc/systemd/system/node_exporter.service

Step 3: Add the following service file content to the service file and save it.

[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target

Step 4: Reload the system daemon and start the node exporter service.

systemctl daemon-reload
systemctl start node_exporter

Step 5: Check the node exporter status to make sure it is running in the active state.

systemctl status node_exporter

Step 6: Enable the node exporter service to the system startup.

systemctl enable node_exporter

Now, node exporter would be exporting metrics on port 9100.

You can see all the server metrics by visiting your server URL on /metrics as shown below.

https://<server-IP>:9100/metrics

Something like this:

Configure the Server as Target on Prometheus Server

If everything is working upto this point, Congratulations! You have configured a node exporter, it is up and running on the server, we have to add this server a target on the Prometheus server configuration.

Note: This configuration should be done on the Prometheus server. If you have followed along the previous tutorial it should be that machine, if not please configure it before moving ahead.

Step 1: Login to the Prometheus server and open the prometheus.yml file.

vi /etc/prometheus/prometheus.yml

Step 2: Inside the scrape config change the scrape config section add the node exporter target as shown below. Change 10.1.213.2 with your server IP where you have setup node exporter. Job name can be your server hostname or IP for identification purposes.

– job_name: ‘node_test’
    scrape_interval: 5s
    static_configs:
      – targets: [‘10.1.213.2:9100’]

Step 3: Restart the prometheus service for the configuration changes to take place.

systemctl restart prometheus

Now, if you check the target in prometheus web UI (http://<prometheus-IP>:9090/targets) , you will be able to see the status as shown below.

Also, you can use the Prometheus expression browser to query for node related metrics. Following are the few key node metrics you can use to find its statistics.

node_memory_MemFree_bytes
node_cpu_seconds_total
node_filesystem_avail_bytes
rate(node_cpu_seconds_total{mode=”system”}[1m])
rate(node_network_receive_bytes_total[1m])