Is it possible to reach docker swarm stacks via IPv6?

  r/docker

I've been pulling my hair out for the past couple of days trying to figure this out. Hhopefully someone here knows how this works and can tell me what I'm doing wrong?

High Level Overview

I have a 2 node docker swarm on raspberry pi 4s running raspbian. 1 manager and 1 worker. Both nodes have IPv4 and IPv6 addresses, and I used the ipv6 address to advertise / join the swarm, but when I start a stack (portainer), I can only reach the web ui using ipv4.

Low Level Addendum

Each of the Pis is running yggdrasil, an overlay ipv6 network that provides each pi with an ipv6 address. All yggdrasil addresses are "global", so my pis can be located on different wans / be anywhere and still be able to communicate using the yggdrasil IP. Pretty much anything that supports listening on ipv6 works out of the box with yggdrasil – the tun interface just gets traffic like any other interface.

My understanding of the problem

As I understand it, the core issue is that docker by default only gives containers ipv4 connectivity, and that even if the docker host has an ipv6 address and even if docker is listening on ipv6, the containers need to have ipv6 as well for them to be routable without 6in4 or some voodoo magic. So my attempts to make this work mostly center around trying to enable ipv6 in docker and get containers to automatically get an ipv6 address. Before I tried anything, I confirmed that portainer was actually serving html over :9000 using curl 127.0.0.1:9000.

What I have tried

First and most obvious, I enabled ipv6 in /etc/docker/daemon.json and, like several ipv6 + docker guides tell you to, added a random ipv6 subnet for it to use. This seems kind of moot if I understand correctly because this subnet applies to the docker0 bridge, which I believe I read is only used if the container has no networking specified (which is not the case for portainer, which creates a portainer_agent_network network), and the docs say the default bridge is deprecated.

Now that ipv6 is enabled and docker seems happy with that, I inspected the portainer non-agent container (portainer/portainer) and saw that it connected to ingress and portainer_agent_network, so I guess I need to get ipv6 addresses on the ingress bridge and hopefully that'll do the trick – when I nmap via the ipv6 address of the host I can see the port 9000 for portainer open, it just doesn't respond via ipv6.

So in an attempt to get ipv6 on the ingress network, I removed ingress and recreated it with this command: sudo docker network create -d overlay --subnet=10.0.0.0/16 --gateway=10.0.0.1 --subnet=2001:db8:1::/64 --ipv6 --scope swarm --ingress --opt com.docker.network.driver.overlay.vxlanid_list=4096 ingress (the opt I copied from the premade ingress network in hopes that I would avoid breaking anything)

This seemed to work, as it creates this network:

[ { "Name": "ingress", "Id": "ts6nsqgd402mwdnzmmuugq21e", "Created": "2019-12-21T05:20:00.509822096Z", "Scope": "swarm", "Driver": "overlay", "EnableIPv6": true, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "2001:db8:1::/64", "Gateway": "2001:db8:1::1" }, { "Subnet": "10.0.0.0/16", "Gateway": "10.0.0.1" } ] }, "Internal": false, "Attachable": false, "Ingress": true, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": null, "Options": { "com.docker.network.driver.overlay.vxlanid_list": "4096,4098" }, "Labels": null } ] 

And I deploy portainer using sudo docker stack deploy --compose-file=portainer-agent-stack.yml portainer

version: '3.2' services: agent: image: portainer/agent volumes: - /var/run/docker.sock:/var/run/docker.sock - /var/lib/docker/volumes:/var/lib/docker/volumes networks: - agent_network deploy: mode: global placement: constraints: [node.platform.os == linux] portainer: image: portainer/portainer command: -H tcp://tasks.agent:9001 --tlsskipverify ports: - "9000:9000" - "8000:8000" volumes: - portainer_data:/data networks: - agent_network deploy: mode: replicated replicas: 1 placement: constraints: [node.role == manager] networks: agent_network: driver: overlay attachable: true volumes: portainer_data: 

But at this point, only the portainer agent container will run – it doesn't have any published ports so that makes sense. In systemctl status docker I see errors related to the ipv6 subnet I added to ingress:

Dec 21 05:23:27 mm-swarm-manager-1 dockerd[3170]: time="2019-12-21T05:23:27.442727510Z" level=warning msg="Peer operation failed:Unable to find the peerDB for nid:ts6nsqgd402mwdnzmmuugq21e op:&{3 ts6nsqgd402mwdnzmmuugq21e [] [] [] [] false false false func1}" Dec 21 05:23:27 mm-swarm-manager-1 dockerd[3170]: time="2019-12-21T05:23:27.547833747Z" level=warning msg="underweighting node z729yr0y1uf0kgnhmp6srpy5x for service jap5xql52qt02orm53a3wdu7j because it experienced 5 failures or rejections within 5m0s" module=node node.id=z729yr0y1uf0kgnhmp6srpy5x Dec 21 05:23:33 mm-swarm-manager-1 dockerd[3170]: time="2019-12-21T05:23:33.191382597Z" level=warning msg="Peer operation failed:Unable to find the peerDB for nid:ts6nsqgd402mwdnzmmuugq21e op:&{3 ts6nsqgd402mwdnzmmuugq21e [] [] [] [] false false false func1}" Dec 21 05:23:33 mm-swarm-manager-1 dockerd[3170]: time="2019-12-21T05:23:33.191409263Z" level=error msg="fatal task error" error="Invalid address 2001:db8:1::2: It does not belong to any of this network's subnets" module=node/agent/taskmanager node.id=z729yr0y1uf0kgnhmp6srpy5x service.id=jap5xql52qt02orm53a3wdu7j task.id=b4ljo5iuu5ietijujf Dec 21 05:23:36 mm-swarm-manager-1 dockerd[3170]: time="2019-12-21T05:23:36.870241757Z" level=warning msg="failed to deactivate service binding for container portainer_portainer.1.mul9xzxnwpir88071k6lupz9s" error="No such container: portainer_portainer.1.mul9xzxnwpir88071k6lupz9s" module=node/agent node.id=z729yr0y1uf0kgnhmp6srpy5x Dec 21 05:23:38 mm-swarm-manager-1 dockerd[3170]: time="2019-12-21T05:23:38.240879579Z" level=error msg="fatal task error" error="Invalid address 2001:db8:1::2: It does not belong to any of this network's subnets" module=node/agent/taskmanager node.id=z729yr0y1uf0kgnhmp6srpy5x service.id=jap5xql52qt02orm53a3wdu7j task.id=t7g7uz49t4ymjdyfzx Dec 21 05:23:38 mm-swarm-manager-1 dockerd[3170]: time="2019-12-21T05:23:38.242641048Z" level=warning msg="Peer operation failed:Unable to find the peerDB for nid:ts6nsqgd402mwdnzmmuugq21e op:&{3 ts6nsqgd402mwdnzmmuugq21e [] [] [] [] false false false func1}" 

But this is confusing because 2001:db8:1::2 should definitely be in the subnet I assigned to ingress (2001:db8:1::/64), so I'm not sure where to go from here. My docker ps -a looks like:

sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1142fb413d71 portainer/agent:latest "./agent" 21 minutes ago Up 21 minutes portainer_agent.z729yr0y1uf0kgnhmp6srpy5x.gn88171nczeit151c298tyxf6 

Is it possible to have ipv6 ingress or am I doing it wrong? I feel like this should be relatively straightforward, and I honestly half expected it to just work after enabling ipv6. I don't really care about more complex setups like delegating prefixes to containers, as long as it's automatic and lets me access services using my host's ip address. Any pointers would be greatly appreciated!

submitted by /u/Stephen304
[link] [comments]
Source: Reddit

LEAVE A COMMENT