docker-compose –build taking forever, takes massive amount of space

  build, docker, docker-compose, docker-registry

Problem(2):

Performing a docker-compose --build or docker-compose up --build results in all containers taking very long to build. Also docker appears to be creating a very large tmp file in /tmp, which I believe is the current directory written to this file.

Trouble shooting:

  • tried upgrading docker to 20.10.6
  • Doing a watch df -H shows the main partition fill up, and then the system does nothing (docker-compose –build process hung, nothing happens).
  • I performed a docker system prune -a, that removed about 50GB.
  • The host has 200GB partition set up for / , and this is where all the docker volumes live. I extended this partition to 500GB, and tried to build again, at this point the first container image FINNALLY builds (took 20min). After that the next image starts, which takes 20min to build as well, etc for each other container, took about 2h to build everything.
  • I found the source of space exhaustion. Inside /tmp there is a file called "tmpsx30lncc" (i assume random chars at the end), its size is growing and matches the size consumed during the build process. Running file tmpsx30lncc shows its a POSIX tar archive. I have read online that this file is an archive of my current directory where docker-compose build was run. Transfer rate appears to vary. Based on watching the file increase, it is anywhere form 50-250MB a second.
  • At some point (I believe when all of the contents are archived to the tmp archive file in /tmp, df -H shows the size being returned slowly, while each container gets to it’s build step.
  • I did a du --block-size=M -a / | sort -n -r | head -n 20, the main culprits for large files the /tmp file (full size of directory where docker-compose was run), my docker-registry folder (~70GB), and lastly gitea (4GB), other containers have other folders that add up as well. The main directory where docker-compose is copying all the files is ~76GB.

System

  • OS: Ubuntu 18.04
  • Docker version: 19.03.3
  • Containers trying to build in docker-compose: gitea, postgres, drone, drone-runner, docker-registry
  • volumes: Are all located in my / partition in various locations. Gitea will build at some point, but the next image will exhibit the same behavior and take just as long and fill up the hard drive.

docker-info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.7.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-141-generic
 Operating System: Ubuntu 18.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 10
 Total Memory: 15.64GiB
 Name: taxmd01-gitjenkins-v
 ID: ZZYO:KANZ:3AH3:UO4U:HFHC:G6OW:4UFQ:S7Q5:QNVC:O5QH:33P7:MRMN
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

So forgive me, I am not an expert on docker, and I did not build this system initially, its used for our dev team. This system has been around for 2 years now, and has grown a lot space utilization wise. I can only assume the problem is the databases of the containers are getting filled up, or the docker-registry has many images that are taking up too much space.

Questions:

  • Why are each of my containers taking 20min to build each? (I assume because it needs to copy over all of the files located in docker-compose file is.)
  • Why is there this massive tmp file in my /tmp dir, and why does my system transfer to this file so slowly? Is there anyway to stop docker from creating this file?
  • Any advice on how I can reduce these ridiculously long build times?

Source: Docker Questions

LEAVE A COMMENT