Grafana Alloy 및 cAdvisor로 Docker 컨테이너 모니터링

TL;DR\
Host metrics tell you whether a server is healthy. cAdvisor tells you
which container isn’t. This article shows how to integrate cAdvisor
into a Grafana Alloy push architecture, avoid the common cardinality
trap, and build a few alerts that catch real problems.

Container Monitoring

In my previous posts, I compared Prometheus Agent Mode with Grafana
Alloy and walked through migrating from node_exporter to Alloy. Both
focused on the agent responsible for shipping host metrics and logs
upstream.

The natural next step is monitoring Docker containers.

One of my servers runs eighteen Docker containers spread across multiple
Compose files. Host-level CPU and memory metrics can tell me the server
is healthy, but they cannot tell me which container is consuming all
of the memory or unexpectedly restarting.

cAdvisor solves that problem. Originally developed by Google, it reads
container resource usage directly from Linux cgroups and namespaces
without requiring any instrumentation inside the containers themselves.
In this article I’ll show how it fits cleanly into a push-based Alloy
architecture.

Deployment

cAdvisor runs as its own container with several read-only mounts so it
can inspect Docker and the host’s cgroup state:

docker run \
  --name cadvisor \
  --privileged \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker:/var/lib/docker:ro \
  --volume=/dev/disk:/dev/disk:ro \
  --publish=<port>:8080 \
  gcr.io/cadvisor/cadvisor:latest

Enter fullscreen mode Exit fullscreen mode

Although --privileged often raises eyebrows, every mounted volume is
read-only. cAdvisor is observing host state, not modifying it.

I also recommend avoiding a hardcoded 8080 mapping. It is one of the
most frequently occupied ports on development machines and small
servers. My installer probes for an available port and falls back to
9338, then reports the selected port back so Alloy can generate the
correct scrape target automatically.

Wiring it into Alloy

Rather than exposing cAdvisor to a central Prometheus server, Alloy
scrapes it locally over localhost and forwards those metrics using the
same remote_write pipeline that already carries host metrics and logs.

                    Docker Host
┌──────────────────────────────────────────┐
│                                          │
│  Docker Containers                       │
│        │                                 │
│        ▼                                 │
│    cAdvisor                              │
│        │ localhost scrape                │
│        ▼                                 │
│   Grafana Alloy                          │
│        │ remote_write                    │
└────────┼─────────────────────────────────┘
         │
         ▼
┌───────────────────────────────┐
│ Central Monitoring            │
│                               │
│ Prometheus                    │
│ Loki                          │
│ Grafana                       │
│ Alertmanager                  │
└───────────────────────────────┘

Enter fullscreen mode Exit fullscreen mode

From the backend’s perspective, container metrics arrive exactly like
host metrics: already labeled, already authenticated, and without
requiring any inbound connectivity to the client.

This becomes especially valuable for clients behind CGNAT or dynamic
residential IP addresses. Once a push-based pipeline exists, adding
another local exporter is simply another scrape target—not another
monitoring system.

Avoiding the Cardinality Trap

One detail that many getting-started guides overlook is that cAdvisor
exports metrics for every cgroup it can see, not just your Docker
containers.

Without filtering, dashboards become cluttered with unnamed
infrastructure cgroups while your active series count grows for little
benefit.

The simplest fix is to consistently filter container metrics using a
PromQL label matcher:

container_cpu_usage_seconds_total{name!=""}

Enter fullscreen mode Exit fullscreen mode

Alternatively, you can drop unnamed series at scrape time using metric
relabeling in Alloy or Prometheus.

Three Useful Alerts

Restart loops:

changes(container_start_time_seconds{name!=""}[1h]) > 3

Enter fullscreen mode Exit fullscreen mode

High sustained CPU:

rate(container_cpu_usage_seconds_total{name!="",name!="POD"}[5m]) * 100 > 80

Enter fullscreen mode Exit fullscreen mode

Container memory as a percentage of total host memory:

(container_memory_usage_bytes{name!=""}
 / on(instance, tenant) group_left node_memory_MemTotal_bytes) * 100 > 85

Enter fullscreen mode Exit fullscreen mode

That final query uses a group_left join to compare a container-level
metric against host memory, producing a percentage that’s much easier to
reason about than raw bytes.

Where This Fits

In Irin, Docker monitoring is implemented as an optional module, but the
architecture described here works with any push-based monitoring stack.
Once you’ve adopted a push-based agent, adding exporters like cAdvisor
becomes incremental work.

The difficult part isn’t collecting the metrics—it’s deciding which
ones are worth keeping.

If you’re already running cAdvisor, I’d be interested to hear whether
you’ve run into the unnamed cgroup problem or found a filtering strategy
that works even better.