Disk full, server down: find the culprit, cap journald, logrotate and Docker

A server at 100% disk is not just slow: the database cannot write and anything that needs to write a file falls over with it. Most of the time the culprit is not real data but logs that grow forever because nobody set a cap. This article walks through finding what ate the disk, cleaning up safely, and capping every source so it does not happen again.

Tested on Ubuntu 24.04 (systemd 255, logrotate 3.21) with Docker Engine. Defaults were checked against the systemd 255 man pages, the logrotate man page and the Docker docs.

Step 1: where is it full?

df -h          # space per filesystem
df -i          # inodes: if df -h shows space but you cannot create files, look here

Find the filesystem whose Use% is near 100%, usually /, then drill down one level at a time:

sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -15
sudo du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail -15

-x stays on one filesystem; sort -h sorts human-readable sizes (K/M/G). If you can install it, ncdu is nicer: you move in and out of folders with the arrow keys.

sudo apt install ncdu
sudo ncdu -x /

The usual culprits

Culprit	How to measure	The default that lets it grow
journald	journalctl --disk-usage	SystemMaxUse = 10% of the filesystem (capped at 4G)
App logs	du -sh /var/log/* /srv/*/logs	no logrotate config = never rotated
Docker container logs (json-file)	sudo du -sh /var/lib/docker/containers/*/*-json.log	max-size = -1, unlimited
Docker images, build cache	docker system df	never removed automatically
Old kernels	dpkg -l 'linux-image-*' | grep ^ii	left behind on every kernel update
Deleted files still open	sudo lsof +L1	the process still holds the space

The last row fools people most often. rm a 10 GB log the app is still writing and the name disappears but the space does not come back until the process closes the file. To empty a file that is still being written, use sudo truncate -s 0 file instead.

Clean up what is in front of you

sudo journalctl --rotate --vacuum-size=500M   # archive the active file, keep at most 500 MB
docker system df                              # see what is using space first
docker image prune -a                         # remove images no container uses (asks first)
docker builder prune                          # clear build cache
sudo apt autoremove --purge                   # kernels and packages no longer needed
sudo apt clean                                # downloaded .deb files

--vacuum-size only removes archived journal files, hence the --rotate in front. Be careful with docker image prune -a: it removes every image no container references, including the one you would roll back to. If you keep an old tag for rollback, start with docker image prune (no -a), which removes only dangling images.

Cleaning without a cap just books the next outage for next month.

Cap 1: journald

By default the journal may use 10% of the filesystem, up to 4G. On a 50 GB disk that is 5 GB, so the real cap is 4 GB, which is a lot for a small server. Set it with a drop-in instead of editing the main file:

sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/size.conf >/dev/null <<'EOF'
[Journal]
SystemMaxUse=500M
EOF
sudo systemctl restart systemd-journald
journalctl --disk-usage

Cap 2: logrotate for your app’s logs

System packages ship their own configs in /etc/logrotate.d/. Logs your own app writes need one you add yourself:

# /etc/logrotate.d/myapp
/srv/myapp/logs/*.log {
    daily
    rotate 14
    maxsize 200M
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

Directive	Meaning
daily + rotate 14	rotate once a day, keep 14, delete older
maxsize 200M	rotate early if the file passes 200 MB (takes effect whenever logrotate runs)
compress + delaycompress	gzip old copies but leave the newest one uncompressed for a cycle, in case the app still writes to it
missingok, notifempty	no error if missing, skip empty files
copytruncate	copy then truncate the original; for apps that cannot reopen their log, at the cost of possibly losing a few lines during the copy

If the app can reopen its log (on a signal, say), use create + postrotate instead of copytruncate. Always test first:

sudo logrotate -d /etc/logrotate.d/myapp    # dry run, changes nothing
sudo logrotate -f /etc/logrotate.d/myapp    # force one real rotation

The simpler route: if the app runs as a systemd service, log to stdout and let journald keep it. Then there is only one cap to look after (how to write the service is in the systemd article).

Cap 3: Docker logs

The default json-file driver does not rotate at all, so a chatty container grows forever. Set defaults in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

sudo systemctl restart docker

Values in log-opts must be strings (quote the "3"), and they apply only to containers created after the restart; recreate the existing ones. Docker’s docs also recommend the local driver, which rotates by default (20 MB × 5 files, compressed, about 100 MB per container) and uses a more efficient file format; you still read it with docker logs. For other Docker settings on a small machine see the Docker hardening article.

Worked example: how many days until full?

A server with a 50 GB disk has 28 GB used. Writing down df once a day for a week gives these growth rates:

Source	Growth per day
App logs (no logrotate)	300 MB
Docker container logs (json-file, unlimited)	150 MB
Journal	50 MB
Total	500 MB/day

1. Free space = 50 − 28 = 22 GB
2. To the 80% alert (= 40 GB): 40 − 28 = 12 GB → 12 GB ÷ 0.5 GB/day = 24 days
3. To 100%: 22 GB ÷ 0.5 GB/day = 44 days

With all three caps in place, the maximum log footprint becomes a constant:

• journal ≤ 500 MB
• Docker 10 MB × 3 files = 30 MB per container; with 4 containers, 120 MB
• app logs 14 rotated + the current file = 15 × 300 MB = 4.5 GB worst case, ignoring compression (gzip usually shrinks text logs a lot further)

That totals at most about 5.1 GB and then stops growing. The disk graph goes flat instead of climbing every day. What still grows is the app’s real data, which needs its own retention plan.

Every log source needs a cap. If one still doesn’t have one, the disk will fill; the only question is when.

Alert at 80%

A short script that checks / and warns past 80%, run every 15 minutes from a systemd timer or cron:

#!/bin/sh
# /usr/local/bin/disk-alert.sh
LIMIT=80
USE=$(df --output=pcent / | tail -1 | tr -dc '0-9')
if [ "$USE" -ge "$LIMIT" ]; then
  logger -p user.warning "disk / at ${USE}%"
  # forward to Telegram/email, whatever you already use
fi

In the example above, 80% still leaves 20 days before full: enough to fix it during working hours, not at 3 AM. Sending the alert to Telegram is covered in the server monitoring article.

Common mistakes

• rm on a log that is still being written. The space does not come back; use truncate -s 0 or restart the process
• Deleting files in /var/log/journal by hand. Use journalctl --vacuum-* and let journald manage it
• Setting daemon.json without recreating containers. The cap does not reach existing containers
• Forgetting inodes. df -h shows space but you cannot create files: usually huge numbers of small files (sessions, cache). Check df -i
• Cleaning without finding the cause. Record the daily growth rate so you know where the cap is missing

Sources

• journald.conf (systemd 255) — freedesktop.org
• journalctl (systemd 255) — freedesktop.org
• logrotate(8) — man7.org
• JSON File logging driver — Docker Docs
• Configure logging drivers — Docker Docs
• Local file logging driver — Docker Docs
• logrotate package in Ubuntu Noble — Launchpad