Dealing with "No Space Left on Device" Errors in GitLab Runner

Encountering "no space left on device" errors while running tasks on your GitLab Runner due to low disk space can be a frustrating roadblock. In this guide, we'll explore not only how to identify and address this issue but also provide additional steps specific to Docker-related space consumption, ensuring a smooth and efficient CI/CD pipeline.

If you're looking for a hassle-free solution, remember that Cloud-Runner offers managed GitLab runners, taking care of such issues for you.

Step 1: Check Disk Usage

Begin by assessing the disk utilization on the machine where the GitLab Runner operates. You can do this by running the following command:

df -h

This command will display the disk usage for all mounted file systems. Identify the file system that's running low on space.

Step 2: Clean Up Unused Files

If disk usage is excessive, it's time to declutter. Remove unused files, such as logs, transient files, and cached files, to free up space. You can locate and delete files older than a specified number of days using the find program. For instance, to delete files older than seven days, use:

find /path/to/files -type f -mtime +7 -exec rm {} \;

This command will clear out files older than a week in the specified path.

Step 3: Deal with Docker Space Consumption

Docker can often be a space hog, with its images, containers, volumes, and build cache. To keep Docker from monopolizing your disk space, implement regular pruning. Schedule the following commands as cron jobs (e.g., daily at midnight):

#Prune old images:
0 0 * * * docker image prune -a --filter "until=96h" -f
#Prune volumes:
0 0 * * * docker volume prune -f
#Prune containers older than 48 hours:
0 0 * * * docker container prune --filter "until=48h" -f
#Prune build cache:
0 0 * * * docker buildx prune --force