#### docker-compose –build taking forever, takes massive amount of space

Problem(2):

Performing a docker-compose --build or docker-compose up --build results in all containers taking very long to build. Also docker appears to be creating a very large tmp file in /tmp, which I believe is the current directory written to this file.

Trouble shooting:

• tried upgrading docker to 20.10.6
• Doing a watch df -H shows the main partition fill up, and then the system does nothing (docker-compose –build process hung, nothing happens).
• I performed a docker system prune -a, that removed about 50GB.
• The host has 200GB partition set up for / , and this is where all the docker volumes live. I extended this partition to 500GB, and tried to build again, at this point the first container image FINNALLY builds (took 20min). After that the next image starts, which takes 20min to build as well, etc for each other container, took about 2h to build everything.
• I found the source of space exhaustion. Inside /tmp there is a file called "tmpsx30lncc" (i assume random chars at the end), its size is growing and matches the size consumed during the build process. Running file tmpsx30lncc shows its a POSIX tar archive. I have read online that this file is an archive of my current directory where docker-compose build was run. Transfer rate appears to vary. Based on watching the file increase, it is anywhere form 50-250MB a second.
• At some point (I believe when all of the contents are archived to the tmp archive file in /tmp, df -H shows the size being returned slowly, while each container gets to it’s build step.
• I did a du --block-size=M -a / | sort -n -r | head -n 20, the main culprits for large files the /tmp file (full size of directory where docker-compose was run), my docker-registry folder (~70GB), and lastly gitea (4GB), other containers have other folders that add up as well. The main directory where docker-compose is copying all the files is ~76GB.

System

• OS: Ubuntu 18.04
• Docker version: 19.03.3
• Containers trying to build in docker-compose: gitea, postgres, drone, drone-runner, docker-registry
• volumes: Are all located in my / partition in various locations. Gitea will build at some point, but the next image will exhibit the same behavior and take just as long and fill up the hard drive.

docker-info:

Client:
Context:    default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
scan: Docker Scan (Docker Inc., v0.7.0)

Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 19.03.3
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-141-generic
Operating System: Ubuntu 18.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 10
Total Memory: 15.64GiB
Name: taxmd01-gitjenkins-v
ID: ZZYO:KANZ:3AH3:UO4U:HFHC:G6OW:4UFQ:S7Q5:QNVC:O5QH:33P7:MRMN
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false


So forgive me, I am not an expert on docker, and I did not build this system initially, its used for our dev team. This system has been around for 2 years now, and has grown a lot space utilization wise. I can only assume the problem is the databases of the containers are getting filled up, or the docker-registry has many images that are taking up too much space.

Questions:

• Why are each of my containers taking 20min to build each? (I assume because it needs to copy over all of the files located in docker-compose file is.)
• Why is there this massive tmp file in my /tmp dir, and why does my system transfer to this file so slowly? Is there anyway to stop docker from creating this file?
• Any advice on how I can reduce these ridiculously long build times?

Source: Docker Questions