docker-compose --build or
docker-compose up --build results in all containers taking very long to build. Also docker appears to be creating a very large tmp file in /tmp, which I believe is the current directory written to this file.
- tried upgrading docker to 20.10.6
- Doing a
watch df -Hshows the main partition fill up, and then the system does nothing (docker-compose –build process hung, nothing happens).
- I performed a
docker system prune -a, that removed about 50GB.
- The host has 200GB partition set up for / , and this is where all the docker volumes live. I extended this partition to 500GB, and tried to build again, at this point the first container image FINNALLY builds (took 20min). After that the next image starts, which takes 20min to build as well, etc for each other container, took about 2h to build everything.
- I found the source of space exhaustion. Inside /tmp there is a file called "tmpsx30lncc" (i assume random chars at the end), its size is growing and matches the size consumed during the build process. Running
file tmpsx30lnccshows its a POSIX tar archive. I have read online that this file is an archive of my current directory where
docker-compose buildwas run. Transfer rate appears to vary. Based on watching the file increase, it is anywhere form 50-250MB a second.
- At some point (I believe when all of the contents are archived to the tmp archive file in /tmp,
df -Hshows the size being returned slowly, while each container gets to it’s build step.
- I did a
du --block-size=M -a / | sort -n -r | head -n 20, the main culprits for large files the /tmp file (full size of directory where docker-compose was run), my docker-registry folder (~70GB), and lastly gitea (4GB), other containers have other folders that add up as well. The main directory where docker-compose is copying all the files is ~76GB.
- OS: Ubuntu 18.04
- Docker version: 19.03.3
- Containers trying to build in docker-compose: gitea, postgres, drone, drone-runner, docker-registry
- volumes: Are all located in my / partition in various locations. Gitea will build at some point, but the next image will exhibit the same behavior and take just as long and fill up the hard drive.
Client: Context: default Debug Mode: false Plugins: app: Docker App (Docker Inc., v0.9.1-beta3) buildx: Build with BuildKit (Docker Inc., v0.5.1-docker) scan: Docker Scan (Docker Inc., v0.7.0) Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 19.03.3 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec init version: fec3683 Security Options: apparmor seccomp Profile: default Kernel Version: 4.15.0-141-generic Operating System: Ubuntu 18.04.5 LTS OSType: linux Architecture: x86_64 CPUs: 10 Total Memory: 15.64GiB Name: taxmd01-gitjenkins-v ID: ZZYO:KANZ:3AH3:UO4U:HFHC:G6OW:4UFQ:S7Q5:QNVC:O5QH:33P7:MRMN Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false
So forgive me, I am not an expert on docker, and I did not build this system initially, its used for our dev team. This system has been around for 2 years now, and has grown a lot space utilization wise. I can only assume the problem is the databases of the containers are getting filled up, or the docker-registry has many images that are taking up too much space.
- Why are each of my containers taking 20min to build each? (I assume because it needs to copy over all of the files located in docker-compose file is.)
- Why is there this massive tmp file in my /tmp dir, and why does my system transfer to this file so slowly? Is there anyway to stop docker from creating this file?
- Any advice on how I can reduce these ridiculously long build times?
Source: Docker Questions