Configuring Prometheus, Grafana, and Node Exporter on Amazon EC2 Ubunt

Setting Up Your EC2 Instance

Launch an EC2 instance with the desired specifications (e.g., t2.medium, 30GB storage and security group with inbound rule ssh, 9100, 9090, 3000 port open).
Connect to your EC2 instance using SSH.

ssh -i your-key.pem ubuntu@your-ec2-ip

Update the Linux System and Install Docker

Ensure your system is up to date and install Docker:

sudo apt update
sudo apt install -y docker.io

Install Docker Compose

Install Docker Compose:

sudo apt install -y docker-compose

Adjust Docker Permissions

Adjust Docker permissions to allow the current user to access the Docker daemon:

sudo chown $USER /var/run/docker.sock

Setting Up Directory Structure

Create a directory structure for Docker containers:

mkdir Docker_Container
cd Docker_Container
mkdir grafana prometheus node-exporter

Prometheus Configuration

Create a docker-compose.yml file in the prometheus directory:

cd prometheus
mkdir prometheus
nano docker-compose.yml

Copy the below yaml file & paste in docker-compose.yml then save the file.

version: '3.8'

# Define the services
services:
  # Prometheus service for monitoring and alerting
  prometheus:
    # Use the official Prometheus Docker image
    image: prom/prometheus
    # Define the container name
    container_name: prometheus
    # Command to specify Prometheus configuration file
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    # Expose Prometheus's web interface on port 9090
    ports:
      - "9090:9090"
    # Ensure the container restarts unless explicitly stopped
    restart: unless-stopped
    # Mount Prometheus's configuration and data volumes
    volumes:
      - ./prometheus:/etc/prometheus
      - prom_data:/prometheus

# Define the volume for Prometheus's data storage
volumes:
  prom_data:

Create a prometheus.yml file in the prometheus directory:

cd prometheus
nano prometheus.yml

Copy the below yaml file & paste in prometheus.yml change target-ip to your server-ip then save the file.

# Global configuration for Prometheus
global:
  # How frequently to scrape targets by default
  scrape_interval:     15s
  # Timeout for individual scrape requests
  scrape_timeout:      10s
  # How frequently to evaluate rules by default
  evaluation_interval: 15s

# Alerting configuration
alerting:
  # Configuration for Alertmanager
  alertmanagers:
    - static_configs:
        # No targets defined for Alertmanager
        - targets: []
      scheme: http
      timeout: 10s
      api_version: v1

# Scrape configurations for monitoring targets
scrape_configs:
  # Job configuration for scraping Prometheus itself
  - job_name: prometheus
    honor_timestamps: true
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    static_configs:
      # Targeting localhost where Prometheus is running
      - targets:
          - localhost:9090

  # Job configuration for scraping the my_application service
  - job_name: my_application
    static_configs:
      # Targeting the IP and port of the my_application service
      - targets: ['target-ip:9100']

Grafana Configuration

cd /home/ubuntu/Docker_Container/grafana
nano docker-compose.yml

Create a docker-compose.yml file in the grafana directory, paste the following yaml file and save it:

version: '3.8'

# Define the services
services:
  # Grafana service for data visualization
  grafana: 
    # Use the official Grafana Docker image
    image: grafana/grafana
    # Define the container name
    container_name: grafana
    # Ensure the container restarts unless explicitly stopped
    restart: unless-stopped
    # Expose Grafana's web interface on port 3000
    ports:
      - "3000:3000"
    # Mount Grafana's storage volume
    volumes:
      - grafana-storage:/var/lib/grafana

# Define the volume for Grafana's data storage
volumes:
  grafana-storage:

Setting Up Node Exporter

Download Node Exporter

cd /home/ubuntu/Docker_Container/node-exporter/

Download Node Exporter using wget:

wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz

Note: Ensure you are using the latest version of Node Exporter and the correct architecture build for your server. The provided link is for amd64. For the latest releases, check here - Prometheus Node Exporter Releases

Extract and Move Binary

Extract the contents and move the Node Exporter binary:

tar xvf node_exporter-1.7.0.linux-amd64.tar.gz
cd node_exporter-1.7.0.linux-amd64
sudo cp node_exporter /usr/local/bin
cd ..
rm -rf ./node_exporter-1.7.0.linux-amd64

Create Node Exporter User

Create a dedicated user for running Node Exporter:

sudo useradd --no-create-home --shell /bin/false node_exporter
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Configure the Service

Create a systemd service for Node Exporter:

sudo nano /etc/systemd/system/node_exporter.service

Paste the following configuration:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Save and exit the editor.

Enable and Start the Service

Reload the systemd daemon, enable, and start the Node Exporter service:

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

To confirm the service is running properly, check its status:

sudo systemctl status node_exporter.service

cd /home/ubuntu/Docker_Container/prometheus
docker-compose up -d
cd /home/ubuntu/Docker_Container/grafana
docker-compose up -d

docker ps

Copy the ip address of ec2 web server and paste in browser:

http://your_server_ip_address:9090 for Prometheus

http://your_server_ip_address:3000 for grafana

Add the data source:

Create a Dashboard:

1860 is the dashboard id for Node Exporter Full data source:

This guide should help you set up Prometheus, Grafana, and Node Exporter on your Amazon EC2 instance running Ubuntu using Docker. Adjustments can be made as needed for your specific environment.

In conclusion, this guide provides a comprehensive walk through for configuring Prometheus, Grafana, and Node Exporter on an Amazon EC2 Ubuntu instance using Docker. By following these steps, you can effectively monitor your system's performance metrics and visualize the data through Grafana's intuitive dashboard capabilities.

Here's a brief summary of the key points covered:

System Preparation: Ensure your Ubuntu system is up to date, install Docker and Docker Compose, and adjust Docker permissions to facilitate easy container management.
Directory Structure Setup: Organize your Docker setup by creating separate directories for Prometheus, Grafana, and Node Exporter within a parent directory named Docker_Container.
Prometheus Configuration: Create a docker-compose.yml file for Prometheus, specifying its image, container name, ports for web interface access, and volumes for configuration and data storage.
Grafana Configuration: Similarly, set up a docker-compose.yml file for Grafana, defining its image, container name, ports for web interface access, and volumes for data storage.
Node Exporter Installation: Download and install Node Exporter, create a dedicated user for running it, configure a systemd service to ensure automatic startup, and verify its status to ensure proper functionality.

By following these steps, you'll establish a robust monitoring and visualization environment, empowering you to gain valuable insights into your system's performance and health. With Prometheus collecting metrics, Grafana presenting them in visually appealing dashboards, and Node Exporter providing system-level data, you'll have a powerful toolkit at your disposal for effective monitoring and troubleshooting.

References:

Configuring Prometheus, Grafana, and Node Exporter on Amazon EC2 Ubuntu using Docker