Why is docker failing to create an endpoint gateway on network docker_gwbridge?

We’re trying to implement a backup script that archives docker volumes on a master swarm node. Any container that persists volume data is constrained to that master node. Our backup script, when run, does the following:


1. systemctrl stop docker
2. tar -cjvf ${BACKUP_PATH}/scuba-backup-$(date +%Y-%m-%d-%H%M).tar.bz /var/lib/docker/volumes/ /var/lib/docker/swarm/
3. systemctrl start docker

This has seemingly worked fine for a while. We had previously restored from the backups.

During testing this week we where unable to restore. We have a handful of containers that throw roughly the same message:

{"CurrentState":"Failed 32 seconds ago","DesiredState":"Shutdown","Error":""starting container failed: container d00fe20ee9f7e9255acb48ce0ed0e8cc308752ee8076b3ae8bf3cf0797b97e0b: endpoint create on GW Network failed: failed to create endpoint gateway_46b8f70a043c on network docker_gwbridge: adding interface veth236e904 to bridge docker_gwbridge failed: exchange full"","ID":"6biwbxin9id73whrk12mqmtux","Image":"scuba/as-backend:[email protected]:6b023f01970926699e327a0b7e773a1edd9bb063895240115f490f3b911eb0f0","Name":"scuba_analysis_backend.1","Node":"scuba-master-00","Ports":""}

We’ve run:

sudo docker container prune -f
sudo docker network prune -f
sudo docker system prune -f

What’s causing these backups to throw networking errors?

Source: StackOverflow