scrape docker metrics from inside container

  docker, iptables, networking, socat

I am trying to scrape docker metrics from inside a container. I have seen the creator of Prometheus doing it this way see video here. I am not able to reproduce it.

Configuration

I have exposed docker metrics page via json config.

{
  "metrics-addr" : "172.17.0.1:4999",
  "experimental" : true
}

I have also tried with 0.0.0.0. In any case I am able to curl from the host machine itself with no issues.

$ curl 172.17.0.1:4999/metrics | more

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP builder_builds_failed_total Number of failed image builds
# TYPE builder_builds_failed_total counter
builder_builds_failed_total{reason="build_canceled"} 0
builder_builds_failed_total{reason="build_target_not_reachable_error"} 0
...

From inside container

However, I am not able to get the same metrics page from inside a container. I have tried to basi/socat image, the same one from the video. I have also tried a handful of other ways like using curl from inside the container and so on.

The image does essentially just one thing

ENV IN="172.17.0.1:4999" 
    OUT="4999"

ENTRYPOINT socat -d -d TCP-L:$OUT,fork TCP:$IN

Start Socat image

docker run --rm -p 4998:4999 basi/socat
2021/04/02 16:27:47 socat[8] N listening on AF=2 0.0.0.0:4999

curl the images published port

curl localhost:4998/metrics

Connection just hangs

The socat images recognizes the connection, but it hangs for 1-2 minutes and times out.

2021/04/02 16:29:53 socat[8] N accepting connection from AF=2 172.17.0.1:57678 on AF=2 172.17.0.2:4999
2021/04/02 16:29:53 socat[8] N forked off child process 9
2021/04/02 16:29:53 socat[8] N listening on AF=2 0.0.0.0:4999
2021/04/02 16:29:53 socat[9] N opening connection to AF=2 172.17.0.1:4999
2021/04/02 16:32:04 socat[9] E connect(5, AF=2 172.17.0.1:4999, 16): Operation timed out
2021/04/02 16:32:04 socat[9] N exit(1)
2021/04/02 16:32:04 socat[8] N childdied(): handling signal 17
2021/04/02 16:32:04 socat[8] W waitpid(): child 9 exited with status 1
curl: (52) Empty reply from server

Firewall

I am slowly wondering if it could be something in my iptables rules. I have posted below INPUT and DOCKER-USER chain. Those are the only ones I have changed myself.

sudo iptables -L -n -v
Chain INPUT (policy DROP 32060 packets, 1810K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 716K 1484M ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
 599K 1069M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   16  3138 ACCEPT     all  --  eth0   *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
   12   624 DROP       all  --  eth0   *       0.0.0.0/0            0.0.0.0/0           
74921  139M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Source: Docker Questions

LEAVE A COMMENT