# Tools to monitore your validator

Hello friends! I consider it necessary to make a general analysis of useful software which you will need to make the validator always work.<br>

Here I will present how monitoring is performed using 3 utilities:

* 1st Part - Prometheus
* 2nd Part - Grafana cloud
* 3d Part - Node exporter
* 4th Part - Dashborad setting up
* 5th Part - Conclusion

{% hint style="info" %}
Before we start - deploy and connecting to a new server
{% endhint %}

## Update dependencies

```bash
sudo apt update && sudo apt upgrade -y
sudo apt install nano
```

## Setup Prometheus

### 1 Create a dedicated user and group for Prometheus on your server

```bash
groupadd --system prometheus
useradd -s /sbin/nologin --system -g prometheus prometheus
```

### 1.2 Download the latest version of Prometheus

```bash
wget 
https://github.com/prometheus/prometheus/releases/download/v2.51.2/prometheus-2.51.2.linux-amd64.tar.gz
```

### 1.3 Extract Prometheus

```bash
tar -xvf prometheus*.tar.gz
```

### 1.4 Change the directory to the extracted directory

```bash
cd prometheus-2.51.2.linux-amd64
```

### 1.5 Create some required directories

```bash
mkdir /etc/prometheus
mkdir /var/lib/prometheus
```

### 1.6 Copy the required files

```bash
mv prometheus.yml /etc/prometheus/prometheus.yml
mv consoles/ console_libraries/ /etc/prometheus/
mv prometheus promtool /usr/local/bin/
```

### 1.7 Create a systemd service file

```bash
sudo tee /etc/systemd/system/prometheus.service > /dev/null <<EOF
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.external-url=
SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target
EOF
```

### 1.8 Set proper ownership and permission to the Prometheus directory

```bash
sudo chown -R prometheus:prometheus /etc/prometheus/
sudo chmod -R 775 /etc/prometheus/
sudo chown -R prometheus:prometheus /var/lib/prometheus/
```

## Setup Grafana Cloud

### 2 Create account and api keys grafana free service

```bash
https://grafana.com/auth/sign-up/
```

### 2.1 Head over to your Grafana Cloud Portal and select Send Metrics on Prometheus. If you scroll above, you should see the section for API Key.

Click on Generate now and create an API Key with the Role MetricsPublisher.\
Copy the Prometheus config and save it locally. The url and username should be unique for every user.\
The password in both snippet should be filled with your API key.

<figure><img src="https://1191442972-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FsocsWgfdMBUSXoC8Z3Gv%2Fuploads%2FPaCfvZ4YL6DNirEYKQbc%2Fimage.png?alt=media&#x26;token=967d53e4-3af5-461c-95e8-6990b27d5ca3" alt=""><figcaption><p>API Key</p></figcaption></figure>

<figure><img src="https://1191442972-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FsocsWgfdMBUSXoC8Z3Gv%2Fuploads%2Fhuka7LNdQmNuatTVAFJE%2Fimage.png?alt=media&#x26;token=8409a128-3168-47bc-b733-d71484351f6f" alt=""><figcaption><p>API  Key Name and Role</p></figcaption></figure>

### 2.2 Change prometheus config change url, password and username in config

<figure><img src="https://1191442972-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FsocsWgfdMBUSXoC8Z3Gv%2Fuploads%2F2lvfsxEjYtK7cyepOdPy%2Fimage.png?alt=media&#x26;token=7739760f-55e1-4ef1-8bdc-b5be5a5da69c" alt=""><figcaption></figcaption></figure>

<figure><img src="https://1191442972-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FsocsWgfdMBUSXoC8Z3Gv%2Fuploads%2FnLx2jJdtvaPZfW7z6WIm%2Fimage.png?alt=media&#x26;token=c61d09f6-5e37-44fa-8470-50bfd16a86f5" alt=""><figcaption></figcaption></figure>

```bash
nano /etc/prometheus/prometheus.yml
```

> Change 5 lines by yours (***origin\_prometheus***, ***url***, ***username***, ***password***, ***job\_name exporter targets***)&#x20;

<pre class="language-bash"><code class="lang-bash"># Sample config for Prometheus.

global:
  scrape_interval: 60s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 60s # Evaluate rules every 15 seconds. The default is every 1 minute.

  # scrape_timeout is set to the global default (10s).
  # external systems (federation, remote storage, Alertmanager).

  external_labels:
    monitor: 'example'
    origin_prometheus: &#x3C;AnyName>
remote_write:
- url: https://prometheus-prod-12-prod-us-central-4.grafana.net/api/prom/push
  basic_auth:
    username: 77777
    password: AOHSDJASHDKASDUhkasjdhauKSADHausdhaskj
# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  #- "first_rules.yml"
  #- "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label 'job=&#x3C;job_name>' to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 60s
    scrape_timeout: 60s

<strong>    #metrics_path defaults to '/metrics'
</strong>    #scheme defaults to 'http'.

<strong>    static_configs:
</strong><strong>      - targets: ['localhost:9090']
</strong>
  - job_name: 'exporter'
    # If prometheus-node-exporter is installed, grab stats about the local
<strong>    # machine by default.
</strong>    static_configs:
      - targets: ['localhost:9100']

  - job_name: &#x3C;AnyName>
    static_configs:
      - targets: ['localhost:9101']
</code></pre>

### 2.3 Run prometheus

```bash
systemctl daemon-reload
systemctl start prometheus
systemctl enable prometheus
```

## Setup Node Exporter

{% hint style="info" %}
Next, go to the server where your Validator Node is installed and install Node Exporter
{% endhint %}

### 3 Download node\_exporter

```bash
wget 
https://github.com/prometheus/node_exporter/releases/download/v1.8.0/node_exporter-1.8.0.linux-amd64.tar.gz
```

### 3.2 Extract Node Exporter

```bash
tar -xvzf node_exporter-1.8.0.linux-amd64.tar.gz
```

### 3.3 Move the extracted directory to the /etc/prometheus/

```bash
mv node_exporter-1.8.0.linux-amd64 /etc/prometheus/node_exporter
```

3.4 Set proper ownership

```bash
sudo chown -R prometheus:prometheus /etc/prometheus/node_exporter
```

### 3.5 Create a systemd service file

```bash
sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<EOF
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
ExecStart=/etc/prometheus/node_exporter/node_exporter

[Install]
WantedBy=default.target
EOF
```

### 3.6 Run Node exporter

```bash
systemctl daemon-reload
systemctl start node_exporter
systemctl enable node_exporter
```

## Dashboard setting up

<figure><img src="https://1191442972-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FsocsWgfdMBUSXoC8Z3Gv%2Fuploads%2Fk1dVvz6EroT5ZcWBObZT%2Fimage.png?alt=media&#x26;token=6640c712-8f9a-4802-9d74-4570bd6bd224" alt=""><figcaption><p>Launch grafana cloud</p></figcaption></figure>

Now go to [grafana.net](http://grafana.net/) → dashboard → import dashboard → import your desired dashboard + you can import exporter dashboard with the detailed server info for example 11074.

Also in that dashboard you can add any statistic about your node which was collected by prometheus.

## Useful commands:

Check status

```bash
systemctl status prometheus
systemctl status node_exporter
```

Stop prometheus and exporter

```bash
systemctl stop prometheus && systemctl disable prometheus
systemctl stop node_exporter && systemctl disable node_exporter
```

That’s all you need to monitor your node - please remember that alerts is really important part as well since need to instantly react on what’s happening on logs. I hope that guide was helpful for you to understand what tools do you need to be aware of your validator health. Enjoy your day!