Lsof

$ lsof | grep -i del

       .
       .   (Output cut short)
       . 
       apache2 151110 www-data DEL REG 0,1 81229715 /dev/zero
       apache2 151110 www-data DEL REG 0,1 74060991 /dev/zero
       apache2 151110 www-data 8w REG 259,2 12017916902 512005 /var/log/apache2/access.log (deleted)
       apache2 151110 www-data 9ur REG 259,2 0 5925 /tmp/.ZendSem.SLoNkY (deleted)
       apache2 151112 www-data DEL REG 0,1 81229715 /dev/zero
       apache2 151112 www-data DEL REG 0,1 74060991 /dev/zero
       apache2 151112 www-data 8w REG 259,2 12017916902 512005 /var/log/apache2/access.log (deleted)
       apache2 151112 www-data 9u REG 259,2 0 5925 /tmp/.ZendSem.SLoNkY (deleted)

Its possible a running process opened a file (likely a log file) and kept it locked. At some moment that file was deleted (maybe a wrong logrotate or something) but the process is still running, consequently the file descriptor is kept. You can run: lsof | grep -i del. and check for processes with deleted files (I don’t remember if they will be flagged as deleted, DELETED, delete, etc…). if you find any, confirm if the path matches then we can make a decision (likely kill the process, restart the service that owns it, etc). yep, so the quick fix here would be running service httpd restart (or service apache2 restart depending on the system). something else deleted the files and didn’t let apache know. I would say that it were wrong logrotate rules, but it is just a guess. sometimes a full restart isn’t required, just a apache reload may release the file descriptors and free space. the reload keeps the connections established, so shouldn’t cause downtime (I never promised that though)

lsof +L1; Check if there are open files which have been deleted. Yeah these files still take up space but don't show in directory. Yep. Need to restart that process to make them go away for good. Needs better log rotating to avoid this issue. yeah when you rm a file, you just mark it for deletion. If its not open, it will delete instantly. But if a process still has the file open, it won’t be truly deleted until that process exits or releases the file.

Sometimes the OS holds onto a file even if its deleted. lsof +L1 (https://serverfault.com/questions/232525/df-in-linux-not-showing-correct-free-space-after-file-removal/232526).you can try to restart apache. they might need to look into their application on why its locking deleted files.

How to check what's listening on a port

# lsof -i -P -n | grep LISTEN
systemd-r  794 systemd-resolve   13u  IPv4  16800      0t0  TCP 127.0.0.53:53 (LISTEN)
sshd      1024            root    3u  IPv4  19859      0t0  TCP *:22 (LISTEN)
sshd      1024            root    4u  IPv6  19870      0t0  TCP *:22 (LISTEN)
java      1223 aws-replication   66u  IPv6  28058      0t0  TCP *:1500 (LISTEN)

(Alternate) You can also use netstat:

# sudo netstat -tulpn | grep LISTEN
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      794/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1024/sshd           
tcp6       0      0 :::22                   :::*                    LISTEN      1024/sshd           
tcp6       0      0 :::1500                 :::*                    LISTEN      1223/java