Application Log Monitoring Set Up using Grafana Loki and Promtail

Our deployment involves two distinct servers: one housing Grafana Loki, and the other hosting a Dockerized application with Promtail installed.

Architecture:

Installing Grafana

1. Setting up the prerequisites
2. - sudo apt-get install -y apt-transport-https
- sudo apt-get install -y software-properties-common wget
- wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

3. Adding the repository to grafana
4. - echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
- sudo apt-get update

5. Installing grafana
6. sudo apt-get install grafana

7. Verify the installation
8. grafana-server -version

9. Access the Loki http://:3000

Installing Grafana Loki

1. Update and Upgrade System Packages sudo apt-get update
2. sudo apt-get upgrade

3. Create Keyring Directory and Add Grafana GPG Key mkdir -p /etc/apt/keyrings/
4. wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor > /etc/apt/keyrings/grafana.gpg

5. Add Grafana Repository
6. echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | tee /etc/apt/sources.list.d/grafana.list

7. Install Loki
8. sudo apt-get install loki

9. Create Loki systemd Service cd /etc/systemd/system/
10. sudo nano loki.service


Copy and paste the following into the loki.service file:

[Unit]
Description=Loki service After=network.target
[Service]
Type=simple
User=loki
ExecStart=/usr/bin/loki -config.file /etc/loki/config.yml


Give a reasonable amount of time for the server to start up/shut down TimeoutSec=120 Restart=on-failure

RestartSec=2
[Install] WantedBy=multi-user.target


1. Configure Loki**
2. sudo nano /etc/loki/config.yml

Copy and paste the following configuration into the config.yml file: auth_enabled: false

server:
http\_listen\_port: 3100
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory replication\_factor: 1 final\_sleep: 0s
chunk\_idle\_period: 5m chunk\_retain\_period: 30s max\_transfer\_retries: 0
schema\_config:
configs:
  - from: 2018-04-15 store: boltdb object\_store: filesystem schema: v11
index:
prefix: index\_
period: 168h
storage\_config:
boltdb:
directory: /tmp/loki/index
filesystem:
directory: /tmp/loki/chunks
limits\_config:
enforce\_metric\_name: false reject\_old\_samples: true reject\_old\_samples\_max\_age: 168h
chunk\_store\_config:
max\_look\_back\_period: 0s
table\_manager:
retention\_deletes\_enabled: false retention\_period: 0s


1. Start Loki Service sudo systemctl start loki

Installing Promtail on an EC2 Ubuntu Instance

Follow these precise steps to seamlessly install Promtail on an EC2 Ubuntu instance:

• Establish SSH Connection to Your EC2 Instance:
• Utilise an SSH client to connect securely to your designated Ubuntu instance on AWS EC2.

Download the Promtail Binary:

Obtain the binary via the following wget command:

wget https://github.com/grafana/loki/releases/download/v2.8.2/promtail-linux-amd64.zip Extract the Binary:

• Unzip the downloaded file:
• unzip promtail-linux-amd64.zip
• Move the Binary to a Suitable Location:
  1. Place the Promtail binary in an appropriate directory: sudo mv promtail-linux-amd64 /usr/local/bin/promtail
  2. Establish a directory and configuration file for Promtail: sudo mkdir -p /etc/promtail
  3. Create a Configuration File for Promtail: sudo nano /etc/promtail/promtail.yaml
  4. Populate the configuration file (promtail.yaml) with the necessary parameters: server:

http\_listen\_port: 9080
clients:
  url: http://localhost:3100/loki/api/v1/push**
scrape\_configs:
  job\_name: system static\_configs:
     targets:
        localhost


1. Configure Promtail as a Service:
2. Develop a systemd service file for Promtail: sudo vim /etc/systemd/system/promtail.service

Insert the following content into the service file:

[Unit]
Description=Promtail service After=network.target
[Service]
User=root
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/promtail.yaml
[Install] WantedBy=multi-user.target


1. Start and Enable Promtail:
2. Initiate the Promtail service and set it to launch automatically during boot:

sudo systemctl start promtail sudo systemctl enable promtail

1. Verify the Status:
2. Confirm the operational status of Promtail: sudo systemctl status promtail
3. Configure Firewall (if necessary):
4. If a firewall is active, permit inbound traffic on port 9080

Dashboard:

• Access grafana at http://:3000
• Go to that address and login with the username “admin” and password “admin”
• And you’ll see this

• Click on “Add data source” and search for Loki and Click on it.
• Enter the Name as “Loki” (or whatever)and URL as “http://:3100” (because that’s the address at which our loki server is running)

Visualizations:

Edit the promtail-config.yml file:

scrape\_configs:
- job\_name: docker-logs
static\_configs:
- targets:
 - localhost
labels:
job: my-container
host: localhost
path: /var/lib/docker/containers/container\_id/container\_id.log


• Click on this button (top-right) to create a new panel in the same dashboard and then, click on “Add New Panel”
• Add
• {job=”my-container”, host=”localhost”}
• Here, the {job=”my-container”} is fetching all the logs of the container.
• Edit the panel title and apply it.

‍