Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Recover disk space
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
kilburna
Tux's lil' helper
Tux's lil' helper


Joined: 20 Sep 2004
Posts: 107

PostPosted: Thu Aug 03, 2017 5:56 pm    Post subject: Recover disk space Reply with quote

Hi

I got a server that has run out of disk space. I have deleted about 30G of files, but the df still shows 100% and I cannot start some services. After googling mention is made to use lsof util, but I cannot install lsof due to the disk space.

How may I go about finding out what is causing this. I tried rebooting the server after freeing so disk space but still no go. This is ext4 file system.

Thanks
Kilburn

Code:
/dev/md2       953239052 920260416         0 100% /
devtmpfs           10240         0     10240   0% /dev
tmpfs             814584       812    813772   1% /run
shm              4072912         0   4072912   0% /dev/shm
cgroup_root        10240         0     10240   0% /sys/fs/cgroup
/dev/md0          126867     34278     86039  29% /boot


[Moderator edit: added [code] tags to preserve output layout. -Hu]
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2562

PostPosted: Thu Aug 03, 2017 6:27 pm    Post subject: Reply with quote

Can you wrap that text in code tags so it's easier to read? Thanks.
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2562

PostPosted: Thu Aug 03, 2017 6:32 pm    Post subject: Reply with quote

While there are tools that do better at this, I start at / and type du -hsx * and look where the size is >1g.

Focus on the filesystem that's full, in your case it's the / directory. The -x flag keeps the du command limited to one filesystem.

For /home/* remember that there are lots of hidden files and folders, you may need to pay better attention there.
Back to top
View user's profile Send private message
Jaglover
Watchman
Watchman


Joined: 29 May 2005
Posts: 7328
Location: Saint Amant, Acadiana

PostPosted: Thu Aug 03, 2017 6:34 pm    Post subject: Reply with quote

Logfiles that are not rotated, mysql automatic backups, distfiles (if you have local portage), to name a few.
_________________
Please learn how to denote units correctly!
Back to top
View user's profile Send private message
kilburna
Tux's lil' helper
Tux's lil' helper


Joined: 20 Sep 2004
Posts: 107

PostPosted: Thu Aug 03, 2017 6:55 pm    Post subject: Reply with quote

My problem is that I have deleted about 50G of disk space since I got the out of disk space. It seems that no matter what I delete the df still shows 100%

Code:

Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/md2       953239052 920260448         0 100% /
devtmpfs           10240         0     10240   0% /dev
tmpfs             814584       812    813772   1% /run
shm              4072912         0   4072912   0% /dev/shm
cgroup_root        10240         0     10240   0% /sys/fs/cgroup
/dev/md0          126867     34278     86039  29% /boot
/dev/sdc1       30013840    266144  29747696   1% /mnt/usbdrive


Here is /
Code:
/ # du -hsx *
7.4M    bin
32M     boot
0       dev
18M     etc
8.0K    home
0       lib
3.1M    lib32
17M     lib64
16K     lost+found
4.0K    media
28K     mnt
4.0K    opt
du: cannot access 'proc/31812/task/31812/fd/3': No such file or directory
du: cannot access 'proc/31812/task/31812/fdinfo/3': No such file or directory
du: cannot access 'proc/31812/fd/3': No such file or directory
du: cannot access 'proc/31812/fdinfo/3': No such file or directory
0       proc
59G     root
812K    run
6.6M    sbin
214M    stage3-amd64-20151001.tar.bz2
0       sys
12K     tmp
9.4G    usr
742G    var
Back to top
View user's profile Send private message
Section_8
Guru
Guru


Joined: 22 May 2004
Posts: 578
Location: Arlington, TX, US

PostPosted: Thu Aug 03, 2017 7:00 pm    Post subject: Reply with quote

Quote:
My problem is that I have deleted about 50G of disk space since I got the out of disk space. It seems that no matter what I delete the df still shows 100%

That means something still has the deleted files open - the disk space isn't released until they are closed.
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2562

PostPosted: Thu Aug 03, 2017 7:04 pm    Post subject: Reply with quote

Wow. You have almost a terabyte in /var.

Jaglover is right on the money. You have /var/log/* backups in there, portage temp space.

Look at tmpreaper to see about deleting old stuff.

cd /var and run the same command again: du -hsx * until you figure out where the space went.


Last edited by 1clue on Thu Aug 03, 2017 7:05 pm; edited 1 time in total
Back to top
View user's profile Send private message
kilburna
Tux's lil' helper
Tux's lil' helper


Joined: 20 Sep 2004
Posts: 107

PostPosted: Thu Aug 03, 2017 7:04 pm    Post subject: Reply with quote

How do I find what deleted files are open if I do not have lsof util on the server.
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2562

PostPosted: Thu Aug 03, 2017 7:06 pm    Post subject: Reply with quote

Did you run lsof as root? Or as a user? Edit: Never mind, it's in /usr/bin.
Back to top
View user's profile Send private message
kilburna
Tux's lil' helper
Tux's lil' helper


Joined: 20 Sep 2004
Posts: 107

PostPosted: Thu Aug 03, 2017 9:23 pm    Post subject: Reply with quote

I do not have lsof on the server, so I cannot run it. I also cannot install it as emerge fails with no disk space.
Back to top
View user's profile Send private message
Jaglover
Watchman
Watchman


Joined: 29 May 2005
Posts: 7328
Location: Saint Amant, Acadiana

PostPosted: Thu Aug 03, 2017 9:26 pm    Post subject: Reply with quote

Did you check out portage tempfiles as 1clue suggested?
_________________
Please learn how to denote units correctly!
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2562

PostPosted: Thu Aug 03, 2017 9:59 pm    Post subject: Reply with quote

Always focus on the most-full filesystem (in your case there's only one) and focus on the largest directory in that filesystem.

You're looking for big gains first, because right now you can't even install lsof. Always you're looking for biggest gains first.

So I would run du -hsx * in /var, and then again in whatever is biggest in that directory, and again in the biggest spot in that directory.

I used to use find to look for files greater than a certain amount, but that fails when you see a tmp directory with thousands of smaller files, like in /var/tmp, or with a ton of files in /var/log.

So while my approach is cumbersome and slower and very manual, it's better (for me at least) because each of these special-purpose directories might have something horribly wrong that won't be caught by something more automatic.

For me, I mount /tmp and /var/tmp in tmpfs, I have plenty of RAM and that keeps those files clean every time I reboot.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7200

PostPosted: Thu Aug 03, 2017 10:33 pm    Post subject: Reply with quote

if you are out of inodes you'll get disk full too, even you have free space, did you check your inodes?
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 14398

PostPosted: Fri Aug 04, 2017 1:43 am    Post subject: Reply with quote

His regular df output looks exhausted to me. He might also have an inode problem, but we know he has a block problem.

Given the size of /var and the assertion that this persisted across a reboot, my guess is there is a runaway process logging noise as fast as it can. Every time he frees up blocks, the runaway consumes them to write more noise to its log. Exploring inside /var, as suggested above, should lead the way.
Back to top
View user's profile Send private message
C5ace
Apprentice
Apprentice


Joined: 23 Dec 2013
Posts: 296
Location: Brisbane, Australia

PostPosted: Fri Aug 04, 2017 11:40 am    Post subject: Reply with quote

I would boot from Rescue System CD, mount the / (root partition) to /mnt/gentoo, try to find the files filling up /. Also check contents of /lost+fount. Delete all unwanted files, unmount /mnt/gentoo and run fsck -nfv on the / partition. If there are errors, run fsck -fv. See 'man fsck' for details.
mc (Midnight Commander) is your friend.

Problem may be caused by intermittent connector contacts, cables, memory contacts. Reseat all connectors and memory sticks.

Then boot your system. If the problem continues, backup the / partition. Re-format the / partition. Restore the last known good backup. If this does not fix the problem. install Gentoo from scratch. Use Openrc with USE='-systemd' in /etc/portage/make.conf. When everything works fine, bang the case several times to reconfirm that there are no dodgy contacts. Check the log files. If no errors, install step by step the server applications.
Back to top
View user's profile Send private message
ct85711
Veteran
Veteran


Joined: 27 Sep 2005
Posts: 1726

PostPosted: Fri Aug 04, 2017 1:33 pm    Post subject: Reply with quote

Quote:
Problem may be caused by intermittent connector contacts, cables, memory contacts. Reseat all connectors and memory sticks.

Then boot your system. If the problem continues, backup the / partition. Re-format the / partition. Restore the last known good backup. If this does not fix the problem. install Gentoo from scratch. Use Openrc with USE='-systemd' in /etc/portage/make.conf. When everything works fine, bang the case several times to reconfirm that there are no dodgy contacts. Check the log files. If no errors, install step by step the server applications.


This isn't a symptom of a bad connection issue and reinstall is NOT a solution to fix something. A reinstall is rarely ever the solution you should pick.

In this case, we know where he needs to look, in the /var. 2 prime spots to look in /var is going to be /var/log and maybe /var/tmp/portage. The later is the default that portage compiles packages at. Normally it should be empty (portage is correctly cleaning its self up) but it is also known to not do so. So anything in the /var/tmp/portage folder can be straight out deleted (assuming emerge is NOT running).

Quote:
Given the size of /var and the assertion that this persisted across a reboot, my guess is there is a runaway process logging noise as fast as it can. Every time he frees up blocks, the runaway consumes them to write more noise to its log. Exploring inside /var, as suggested above, should lead the way.


As Hu mentioned, this is more likely a case of some logs spamming a lot, filling the drive. So, more likely you will be able to spot the offending file easily enough by looking at ls -lh and look at file sizes... All we need to do after that is look at the log file and see what is being spammed a lot in the log so we can possibly disable that service initially so the system is running, and we can then correct or redirect that service to log somewhere else (like sending the useless lines to /dev/null)...
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6042

PostPosted: Fri Aug 04, 2017 4:15 pm    Post subject: Reply with quote

Removing that leftover stage3 tarball would help in the meantime. Assuming /var/log/ isn't filling up faster than that...
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2562

PostPosted: Fri Aug 04, 2017 5:04 pm    Post subject: Reply with quote

One more vote against a system restore or re-installation. This is a problem that can be solved.


  1. Find the huge file(s)
  2. If it's service-related then turn off that service. (e.g. if you have a ton of ssh entries then somebody's trying to brute-force your system. If you're on the console, turn off sshd.)
  3. Delete the huge file(s)
  4. Restart the service and try to determine why the file grew.
  5. Fix that problem.
  6. If your filesystem is still too full, repeat the large file discovery process and go after the largest file(s) again.
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10395
Location: Somewhere over Atlanta, Georgia

PostPosted: Fri Aug 04, 2017 5:09 pm    Post subject: Reply with quote

Ant P. wrote:
Removing that leftover stage3 tarball would help in the meantime. Assuming /var/log/ isn't filling up faster than that...
Also, removing /usr/portage/distfiles/* would be a good temporary fix as, unless you've actively running an emerge, none of those files will be open.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum