Solving Docker Container Log Overflow and Disk Space Shortage!
When operating Docker, container logs can accumulate in large amounts and put pressure on disk space. Especially in environments using docker-compose, if you stick with the default log settings, logs will accumulate indefinitely, and you may find your storage full before you know it.
While investigating the cause of the disk space shortage, I confirmed that the containers had become bloated.
du -shc /var/lib/docker/containers/* -x | sort -rh
9.3G Total
7.5G ./bba63eea9d895b78284dc1ec7a4ba9d74185dd8ec441c5bc41da954c5d622ab1
808M ./0b01e61f5d2d745b3b0f496077cd6fd94e1c8d0d6aecb2965ad31f38ad639e9c
・・・
Therefore, in this article, we will explain how to manage logs using docker-compose.
Causes of Logs Pressuring Disk Space
In Docker’s default configuration, container logs accumulate indefinitely under /var/lib/docker/containers/ using the json-file driver. As a result, operating for a long period causes the logs to bloat, which becomes a cause of squeezing disk space.
Checking the Current Log Size
To check the log size for each container, use the following command.
sudo du -sh /var/lib/docker/containers/*/*.log
This command allows you to understand how large the log files are.
Log Settings in docker-compose
To limit logs in docker-compose.yml, add logging settings to each service.
For example, to limit the maximum log size to 10MB and the number of retained log files to 3, configure it as follows:
version: '3.8'
services:
my_service:
image: my_image
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
With this setting, when a log reaches 10MB, a new log file is created, and up to 3 log files are retained.
Where are the docker-compose.yml Logs Stored?
Even if you add logging settings to docker-compose.yml, the log storage location remains the same as the default. Typically, logs for each container are stored in the following directory:
/var/lib/docker/containers/<container_id>/<container_id>-json.log
You can check the container_id using docker ps -a.
For this reason, even after configuring logging, logs will continue to be saved in this directory as long as the json-file driver is used.
Using Other Log Drivers
Besides json-file, Docker allows you to use log drivers such as the following:
syslog: Send to the system’s syslogjournald: Send to systemd’s journalfluentd: Send to Fluentdawslogs: Send to AWS CloudWatch Logsnone: Disable logging
For example, when using syslog, configure it like this:
logging:
driver: "syslog"
options:
syslog-address: "udp://192.168.1.100:514"
Periodically Clearing Logs
To delete logs that have already bloated, you can clear the log files with the following command:
echo "" | sudo tee /var/lib/docker/containers/*/*.log
It is also possible to use cron to periodically delete logs.
Summary
- Since Docker accumulates logs indefinitely by default, reviewing the settings is necessary
- Specify
max-sizeandmax-filein theloggingconfiguration ofdocker-compose.yml - Even with added configurations, the log destination remains under
/var/lib/docker/containers/ - Integrating with external log management systems like syslog is also an option
- You can save disk space by deleting and managing existing logs
When operating Docker, make sure to configure appropriate log settings to use your disk space efficiently!