Skip to content

Commit 75a5d8d

Browse files
prathamesh-sonpatkiprathamesh-sonpatki
authored
Add example of docker monitoring (#63)
1 parent 26b2cd5 commit 75a5d8d

File tree

5 files changed

+317
-0
lines changed

5 files changed

+317
-0
lines changed

‎docker-compose/README.md

+152
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# Monitoring Docker Containers with OpenTelemetry and Last9
2+
3+
A guide for setting up Docker container monitoring using OpenTelemetry Collector with Last9. It collects container metrics, and logs from docker containers and sends them to Last9.
4+
5+
## Installation
6+
7+
### 1. Prerequisites
8+
9+
Ensure Docker and Docker Compose are installed on your system:
10+
11+
```bash
12+
# Check Docker installation
13+
docker --version
14+
15+
# Check Docker Compose installation
16+
docker compose version
17+
```
18+
19+
### 2. Configure OpenTelemetry Collector
20+
21+
The setup uses the `otel-config.yaml` file which defines:
22+
- Docker stats receiver for container metrics
23+
- TCP log receiver for container logs
24+
- Processors for batch processing and resource detection
25+
- Last9 exporter configuration
26+
27+
Before proceeding, update the Last9 authorization token:
28+
29+
```bash
30+
# Edit the config file
31+
nano otel-config.yaml
32+
```
33+
34+
In the `exporters` section, replace `<Last9 Basic Auth Token>` with your actual Last9 authorization auth header. You can get the auth header from [Last9 Integrations](https://app.last9.io/integrations).
35+
36+
### 3. Set Up with Docker Compose
37+
38+
This setup demonstrates how to monitor multiple Docker containers using multiple Docker Compose files to create a complete monitoring stack:
39+
40+
#### Start Apache Server
41+
42+
```bash
43+
docker compose -f apache-compose.yaml up -d
44+
```
45+
46+
This launches an Apache httpd server on port 8002.
47+
48+
#### Start Nginx Server (Optional)
49+
50+
```bash
51+
docker compose -f nginx-compose.yaml up -d
52+
```
53+
54+
This launches an Nginx server on port 8000.
55+
56+
#### Start OpenTelemetry Collector
57+
58+
```bash
59+
docker compose -f otel-compose.yaml up -d
60+
```
61+
62+
This starts:
63+
- OpenTelemetry Collector with the configuration from `otel-config.yaml`
64+
- Logspout container that forwards logs from Docker containers to the collector
65+
66+
## Understanding the Setup
67+
68+
### Docker Networks
69+
70+
The setup creates three Docker networks:
71+
- `nginx_network`: Network for Nginx server
72+
- `apache_network`: Network for Apache server
73+
- `otel_network`: Network for OpenTelemetry components
74+
75+
Logspout connects to all three networks to collect logs.
76+
77+
### Container Metrics
78+
79+
The OpenTelemetry Collector is configured to collect various metrics:
80+
- CPU usage and utilization
81+
- Memory usage and limits
82+
- Network I/O statistics
83+
- Block I/O information
84+
- Container process information
85+
86+
### Log Collection
87+
88+
Container logs are collected using Logspout and sent to OpenTelemetry Collector for processing before being exported to Last9.
89+
90+
## Verification
91+
92+
1. Verify the containers are running:
93+
```bash
94+
docker ps
95+
```
96+
97+
2. Test Apache server:
98+
```bash
99+
curl http://localhost:8002
100+
```
101+
102+
3. Test Nginx server (if installed):
103+
```bash
104+
curl http://localhost:8000
105+
```
106+
107+
4. Check OpenTelemetry Collector logs:
108+
```bash
109+
docker logs otel-collector
110+
```
111+
112+
## Troubleshooting
113+
114+
1. Container issues:
115+
```bash
116+
# Check container status
117+
docker ps -a
118+
119+
# View container logs
120+
docker logs apache-server
121+
docker logs nginx-server
122+
docker logs otel-collector
123+
docker logs logspout
124+
```
125+
126+
2. Network issues:
127+
```bash
128+
# List networks
129+
docker network ls
130+
131+
# Inspect a network
132+
docker network inspect apache_network
133+
docker network inspect nginx_network
134+
docker network inspect otel_network
135+
```
136+
137+
3. OpenTelemetry Collector issues:
138+
```bash
139+
# Check configuration
140+
docker exec otel-collector cat /etc/otel-collector-config.yaml
141+
142+
# Restart collector
143+
docker compose -f otel-compose.yaml restart otel-collector
144+
```
145+
146+
## Stopping the Stack
147+
148+
```bash
149+
docker compose -f otel-compose.yaml down
150+
docker compose -f nginx-compose.yaml down
151+
docker compose -f apache-compose.yaml down
152+
```

‎docker-compose/apache-compose.yaml

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
services:
2+
apache:
3+
image: httpd:2.4
4+
container_name: apache-server
5+
ports:
6+
- "8002:80"
7+
volumes:
8+
- ./app:/usr/local/apache2/htdocs/
9+
restart: unless-stopped
10+
networks:
11+
- apache_network
12+
13+
networks:
14+
apache_network:
15+
name: apache_network

‎docker-compose/nginx-compose.yaml

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
services:
2+
nginx:
3+
image: nginx:latest
4+
container_name: nginx-server
5+
ports:
6+
- "8000:80"
7+
restart: unless-stopped
8+
labels:
9+
- "collect-logs=true"
10+
volumes:
11+
- nginx_logs:/var/log/nginx
12+
networks:
13+
- nginx_network
14+
15+
networks:
16+
nginx_network:
17+
name: nginx_network
18+
19+
volumes:
20+
nginx_logs:
21+
name: nginx_logs

‎docker-compose/otel-compose.yaml

+52
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
services:
2+
otel-collector:
3+
image: otel/opentelemetry-collector-contrib:0.118.0
4+
container_name: otel-collector
5+
command:
6+
[
7+
"--config=/etc/otel-collector-config.yaml",
8+
"--feature-gates=transform.flatten.logs"
9+
]
10+
volumes:
11+
- ./otel-config.yaml:/etc/otel-collector-config.yaml
12+
- /var/run/docker.sock:/var/run/docker.sock
13+
- nginx_logs:/var/log/nginx
14+
ports:
15+
- "4317:4317" # OTLP gRPC receiver
16+
- "4318:4318" # OTLP HTTP receiver
17+
restart: on-failure
18+
user: "0" # root user to access docker stats
19+
environment:
20+
- LOGSPOUT=ignore
21+
networks:
22+
- otel_network
23+
24+
logspout:
25+
image: "gliderlabs/logspout:v3.2.14"
26+
container_name: logspout
27+
volumes:
28+
- /etc/hostname:/etc/host_hostname:ro
29+
- /var/run/docker.sock:/var/run/docker.sock
30+
command: syslog+tcp://otel-collector:2255
31+
depends_on:
32+
- otel-collector
33+
restart: on-failure
34+
networks:
35+
- nginx_network
36+
- apache_network
37+
- otel_network
38+
39+
networks:
40+
nginx_network:
41+
external: true
42+
name: nginx_network
43+
apache_network:
44+
external: true
45+
name: apache_network
46+
otel_network:
47+
name: otel_network
48+
49+
volumes:
50+
nginx_logs:
51+
external: true
52+
name: nginx_logs

‎docker-compose/otel-config.yaml

+77
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
receivers:
2+
docker_stats:
3+
collection_interval: 30s
4+
metrics:
5+
container.cpu.usage.total:
6+
enabled: true
7+
container.cpu.throttling_data.periods:
8+
enabled: true
9+
container.cpu.throttling_data.throttled_periods:
10+
enabled: true
11+
container.cpu.utilization:
12+
enabled: true
13+
container.memory.usage.limit:
14+
enabled: true
15+
container.memory.usage.total:
16+
enabled: true
17+
container.memory.percent:
18+
enabled: true
19+
container.blockio.io_service_bytes_recursive:
20+
enabled: true
21+
container.network.io.usage.rx_bytes:
22+
enabled: true
23+
container.network.io.usage.tx_bytes:
24+
enabled: true
25+
container.network.io.usage.rx_dropped:
26+
enabled: true
27+
container.network.io.usage.tx_dropped:
28+
enabled: true
29+
container.network.io.usage.rx_errors:
30+
enabled: true
31+
container.network.io.usage.tx_errors:
32+
enabled: true
33+
container.network.io.usage.rx_packets:
34+
enabled: true
35+
container.network.io.usage.tx_packets:
36+
enabled: true
37+
container.pids.count:
38+
enabled: true
39+
tcplog/docker:
40+
listen_address: "0.0.0.0:2255"
41+
operators:
42+
- type: syslog_parser
43+
protocol: rfc5424
44+
processors:
45+
transform/docker_logs:
46+
error_mode: ignore
47+
flatten_data: true
48+
log_statements:
49+
- context: log
50+
statements:
51+
- set(body, attributes["message"])
52+
- delete_key(attributes, "message")
53+
- set(resource.attributes["service.name"], attributes["appname"])
54+
55+
batch:
56+
send_batch_size: 100000
57+
send_batch_max_size: 100000
58+
timeout: 20s
59+
resourcedetection:
60+
detectors: [env, system, gcp, ec2, azure, docker]
61+
exporters:
62+
debug:
63+
verbosity: detailed
64+
otlp/last9:
65+
endpoint: https://otlp.last9.io:443
66+
headers:
67+
Authorization: "Basic <Last9 Basic Auth Token>"
68+
service:
69+
pipelines:
70+
metrics:
71+
receivers: [docker_stats]
72+
processors: [batch, resourcedetection]
73+
exporters: [otlp/last9]
74+
logs:
75+
receivers: [tcplog/docker]
76+
processors: [batch, resourcedetection, transform/docker_logs]
77+
exporters: [otlp/last9]

0 commit comments

Comments
 (0)