Back to problems

problem hub

Read-only first

Linux disk full or no space left on device

Prove which filesystem is full, check inodes, then rank directories and files before deleting anything.

Safest first command

df -h

Before you run this

Expected output: Mounted filesystems with size, used, available, use percentage, and mount point.

When not to use it: Do not start cleanup from this page if you already know a database, volume, or backup job is actively writing; capture service-specific evidence first.

Expected output example

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        50G   49G  900M  99% /
/dev/vdb1       100G   42G   58G  42% /srv/data

How to read the result

Start with the mount point at the highest Use%. Clean inside that mounted filesystem, not the path where the application happened to report the error.

What to check next

One filesystem is 95%+ full

Means: The incident scope is that mount point, not every path on the server.

Next step: Rank direct children on that same filesystem before choosing a cleanup path.

Find Which Folder Is Filling the Disk

df -h has space but df -i is near 100%

Means: The filesystem is out of inodes, usually from many small files.

Next step: Count files by directory before deleting or rotating anything.

Check Inodes When Disk Space Looks Fine

Usage stays high after files were deleted

Means: A running process may still hold deleted files open.

Next step: Inspect deleted-open files and identify the owning process first.

Find Open Deleted Files with lsof

Fast path

Start with mount-level evidence. Disk-full symptoms often appear in an application path, but the full filesystem may be root, /var, /boot, a Docker volume, or a mounted data disk.

  1. df -h
  2. df -i
  3. du -xhd1 /var 2>/dev/null | sort -h

Disk-full decision tree

Prove the pressure type before cleanup. Byte pressure, inode pressure, deleted-open files, journal growth, and Docker layers each point to a different next command.

  1. df -h
  2. df -i
  3. du -xhd1 /var 2>/dev/null | sort -h
  4. sudo lsof +L1
  5. journalctl --disk-usage
  6. docker system df

Byte pressure vs inode pressure

High Use% means the filesystem is short on bytes. High IUse% means the filesystem is short on file entries, often from many tiny cache, session, mail, or queue files. Fixing one does not automatically fix the other.

Deleted-open files, logs, journal, and Docker

If usage stays high after deletion, check deleted-open files before deleting more. If /var is the hot path, compare application logs, journal size, package cache, and Docker usage before choosing a cleanup command.

  1. sudo lsof +L1
  2. journalctl --disk-usage
  3. du -xhd1 /var/log /var/lib /var/cache 2>/dev/null | sort -h
  4. docker system df

Do not delete random large files

Large logs, databases, uploads, backups, and Docker layers each have different cleanup paths. Rank first, then identify ownership and service impact.

Why this page is safe to share

The checklist starts with read-only evidence: bytes, inodes, directory ranking, and deleted-open files. Share the repair path rather than a cleanup command so ownership, expected output, and bad fixes stay attached.

Bad fixes to avoid

Do not delete the largest path just because it is large. Avoid pruning Docker, truncating logs, clearing package caches, or removing database files until the owning service and rollback path are clear.

Common causes

  • Large logs under /var/log
  • Package cache or old kernels
  • Docker images, layers, volumes, or container logs
  • Backups or uploads stored on the wrong mount
  • Deleted files still held open by a running process

What not to change yet

  • Do not delete files from /usr, /var/lib, /boot, or Docker volumes until you know what owns them.
  • Do not run cleanup commands before checking bytes, inodes, and deleted-open files.
  • Do not assume the largest file is safe to remove.
  • Do not prune Docker or vacuum journals before checking what is using the space.
  • Do not remove database, upload, or backup files without an owner and rollback path.

Stop and escalate if

  • The full filesystem contains production databases, customer uploads, or active backup targets.
  • Storage or kernel logs mention I/O errors, read-only remounts, or failing devices.
  • Large files are still being written by an active service and deleting them could corrupt data.

platform notes

Distro and service notes

Debian/Ubuntu

APT cache and old kernels can be part of the cleanup path, but inspect first.

Docker

Docker disk pressure needs Docker-specific inspection before prune commands.

systemd journal

Journal cleanup should come after checking journal size and retention needs.

supporting commands

Command path

Guides and drills