My issue is reproducible in a rather specific setup, so let’s first give some context:
- Host OS: Windows 10
- Virtual Box 6.0.14 (latest)
- Guest OS: Ubuntu 19.10 x86_64 (kernel: 5.3.0-24-generic)
- Gitlab-runner: 12.5.0
- Docker: 19.03.2 (latest)
- CMake: 3.16 (latest)
- ninja-build: latest
- Dynamic VDI: 150Gig (max-use is ~ 70 Gig)
A gitlab-runner is executing shell commands inside a Linux guest OS in a Virtual Box on top of the host OS (Windows 10). The host disk is 250Gig in total, while the dynamic linux partition is set tot 150 Gig max, leaving about 60 Gig free Gitlab-runner syncs the project source from an external submodule git, builds a docker (based on top of gcc:8 and dependencies) and pushes it to a repository. It then builds a second docker that executes a number of CMake/ninja install build runs on top of the source. If too many commands are within that docker (like, more than 10 RUN layers) the host OS suddenly drops from 60 GB of free space to mere kilobytes, prompting the host OS (Windows 10) to display a blue warning box that disk-space is critically low. Virtual Box will then suspend the guest OS (Ubuntu), and everything halts.
If I keep an eye on the disk space in the Host OS during this process, the free space stays the same until it suddenly seems to disappear in one ‘event’. After that, Windows Explorer and the Disk Management tool report that the drive is almost full (Kb’s), while WinDirStat can’t find the missing gigabytes.
Strangely enough, I can recover from this by opening a windows console with admin rights, request a chckdsk /f and reboot, after which the free size is reported correctly again, and I can restart the VM and restart the build.
- Q1: What is causing this sudden disappearing of free disk space?
- Q2: Why is chkdsk /f able to fix this?
- Q3: Is there a way to fix it without having to reboot (and exit the VM)?
- Q4: Has anyone else experienced similar strangeness regarding the disk free size being reported wrong in certain cases?