Monitoring tool

Includes grafana server, node_exporter, prometheus, alertmanager. And dashboards.

1. Install Docker

sudo apt -q update
sudo apt -qy install ca-certificates curl gnupg lsb-release

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list >/dev/null

sudo apt -q update
sudo apt -qy install docker-ce docker-ce-cli containerd.io docker-compose-plugin

sudo usermod -aG docker $USER

2. Clone the repo

cd $HOME
git clone https://github.com/nodejumper-org/monitoring-tool.git

3. Create configuration files from examples

cd monitoring-tool
cp prometheus/prometheus.yml.example prometheus/prometheus.yml
cp alertmanager/config.yml.example alertmanager/config.yml

4. Start containers

sudo docker compose up -d
Open in browser http://<your_server_ip>:3000
default user: admin
default password: admin

5. Add your servers with installed node_exporter or installed cosmos-based node with enabled prometheus port to file prometheus/prometheus.yml

  # example for servers with node_exporter installed
  - job_name: "my-servers"
    static_configs:
    - targets: ["172.0.0.1:9100"]
      labels:
        instance: "server1"
    - targets: ["172.0.0.2:9100"]
      labels:
        instance: "server2"
    
  # example for servers with node_exporter and cosmos-based node installed
  - job_name: "cosmos-validator-nodes"
    static_configs:
    - targets: ["192.0.0.1:9100","192.0.0.1:26660"]
      labels:
        instance: "validator1"
    - targets: ["192.0.0.2:9100","192.0.0.2:26660"]
      labels:
        instance: "validator2"
    - targets: ["192.0.0.3:9100","192.0.0.3:26660"]
      labels:
        instance: "validator3"

Next, install the node exporter on the server with the validator:

bash <(curl https://raw.githubusercontent.com/nodejumper-org/monitoring-tool/main/utils/install_node_exporter.sh)

How to update

cd monitoring-tool
sudo docker compose down
git pull
sudo docker compose pull
sudo docker compose up -d

Telegram notifications

Server alerts

• Server down

• Out of memory (<10%)

• Out of disk space (<10%)

• Out of disk space within 24h

• High CPU load (>85%)

Cosmos-based validator node alerts

• Missing blocks

• Degraded syncing (sync less than 40 blocks in last 5 min)

• Low peers count (<5)

In order to enable telegram notifications, create your own bot and fill in the following fields in the file alertmanager/config.yml

chat_id=1111111                 # your telegram user id
bot_token=11111111:AAG_XXXXXXX  # your telegram bot token