Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
harddrive usage eats up memory
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
kortec
n00b
n00b


Joined: 20 Dec 2004
Posts: 68

PostPosted: Wed Jan 23, 2008 8:54 pm    Post subject: harddrive usage eats up memory Reply with quote

I just finished installing Gentoo on a box, and it seems like everything is going just fine -- got X looking great, sound working, and all that junk -- but there's an odd lingering bug.

Whenever there's a relatively large disk read, the memory utilization goes through the roof, and everything slows to a crawl. This could be as simple as loading the firefox binary if it isn't in RAM already, downloading a file that's bigger than about 1 MB, starting acroread, editing id3 tags on mp3s -- whatever. As long as the hard drive is spinning, this happens. (When I say 'memory utilization goes through the roof', I mean that I have half a gig of ram in here, and after disk IO starts, only about 9MB is free according to 'free -m').

Any ideas what might be causing this? I did check to make sure that DMA support was enabled in the kernel, and it is.
Back to top
View user's profile Send private message
tobr
Guru
Guru


Joined: 29 May 2006
Posts: 330

PostPosted: Wed Jan 23, 2008 9:14 pm    Post subject: Reply with quote

That sounds normal.

a) Hard drive access is (relatively) slow. If there’s a large read things can slow down.
b) because of a) Linux caches hard disk in RAM (because empty RAM is useless). This will be freed if an application needs it. free shows the cache (surprisingly) under “cached”, which you need to subtract from the used RAM if you want to know what your apps use.
_________________
Please add [SOLVED] to your message title if your problem is solved.

Death to all blobs!
Back to top
View user's profile Send private message
gentoo_ram
Guru
Guru


Joined: 25 Oct 2007
Posts: 416
Location: San Diego, California USA

PostPosted: Wed Jan 23, 2008 11:24 pm    Post subject: Reply with quote

Yes, repeat after me:

"Free memory is wasted memory."

The kernel will use as much RAM as possible to cache disk accesses. This is a good thing. If the memory is needed by real programs, the kernel will free the disk cache. Don't worry about it unless you're running out of swap space too.
Back to top
View user's profile Send private message
kortec
n00b
n00b


Joined: 20 Dec 2004
Posts: 68

PostPosted: Wed Jan 23, 2008 11:48 pm    Post subject: Reply with quote

So lots of RAM usage is normal, and I do not pretend to understand the kernel's vm decisions, but the severe slow down isn't normal. Moving a 80 MB tar file from one directory to another shouldn't take 5 mins and make it impossible to move the mouse. The hardware isn't that old (athlon xp, 7200 RPM 250 GB hd, etc.). This isn't just in X either -- if I'm doing similar things just in a terminal, I can't switch between ttys or, if I background the process with a lot of disk usage, the tty takes too long to update from keyboard input to make anything possible.
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7347
Location: almost Mile High in the USA

PostPosted: Thu Jan 24, 2008 12:14 am    Post subject: Reply with quote

The problem is likely not memory usage, but rather DMA issues. Is DMA turned on for your hard drive?
Did you compile your disk controller's native driver into the kernel?

hdparm -t /dev/harddrive?
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
fenrisulfr
n00b
n00b


Joined: 23 Jan 2008
Posts: 1

PostPosted: Thu Jan 24, 2008 12:31 am    Post subject: Reply with quote

Is the drive on an addon controller card? SATA or IDE? What is the result of hdparm -t /dev/device? That choking on the drive is definitly not right. My little 233 laptop does just fine with read writes to the drive.
_________________
"It is not advisable to venture unsolicited opinions. You should spare yourself the embarrassing discovery of their exact value to your listener." -Francisco D' Anconia
Back to top
View user's profile Send private message
kortec
n00b
n00b


Joined: 20 Dec 2004
Posts: 68

PostPosted: Thu Jan 24, 2008 12:48 am    Post subject: Reply with quote

I get this from hdparm:

Code:

tilysi ~ # hdparm -tT /dev/hda

/dev/hda:
 Timing cached reads:   588 MB in  2.00 seconds = 293.51 MB/sec
 Timing buffered disk reads:    8 MB in  3.91 seconds =   2.05 MB/sec
tilysi ~ #


Giving hdparm no args to get an info dump yields

Code:


tilysi ~ # hdparm /dev/hda

/dev/hda:
 multcount     = 16 (on)
 IO_support    =  0 (default 16-bit)
 unmaskirq     =  0 (off)
 using_dma     =  0 (off)
 keepsettings  =  0 (off)
 readonly      =  0 (off)
 readahead     = 256 (on)
 geometry      = 30401/255/63, sectors = 488397168, start = 0
tilysi ~ #


I tried to turn DMA on, but got this
Code:

tilysi ~ # hdparm -d1 /dev/hda

/dev/hda:
 setting using_dma to 1 (on)
 HDIO_SET_DMA failed: Operation not permitted
 using_dma     =  0 (off)
tilysi ~ #

even though it's enabled in the kernel. Thoughts?
Back to top
View user's profile Send private message
kortec
n00b
n00b


Joined: 20 Dec 2004
Posts: 68

PostPosted: Thu Jan 24, 2008 6:05 am    Post subject: Reply with quote

Found the bug, I think. I recompiled my kernel with more specific/correct IDE chipset drivers, and got this output.

Code:

tilysi ~ # hdparm -tT /dev/hda

/dev/hda:
 Timing cached reads:   464 MB in  2.01 seconds = 231.42 MB/sec
 Timing buffered disk reads:  172 MB in  3.02 seconds =  56.98 MB/sec
tilysi ~ # hdparm /dev/hda

/dev/hda:
 multcount     = 16 (on)
 IO_support    =  1 (32-bit)
 unmaskirq     =  1 (on)
 using_dma     =  1 (on)
 keepsettings  =  0 (off)
 readonly      =  0 (off)
 readahead     = 256 (on)
 geometry      = 30401/255/63, sectors = 488397168, start = 0
tilysi ~ #


which is heartening, and makes me feel slightly dumb. Ah well.
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7347
Location: almost Mile High in the USA

PostPosted: Thu Jan 24, 2008 3:15 pm    Post subject: Reply with quote

glad you got it working.

The weirdest symptoms can mislead the proper solution eh?
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
kortec
n00b
n00b


Joined: 20 Dec 2004
Posts: 68

PostPosted: Sat Jan 26, 2008 8:45 pm    Post subject: Reply with quote

Seriously. The weirder thing is that I had tried to set the DMA on the drive earlier and got that very error and I barely noticed it. And then I had to talk through the really silly things I was noticing happening to get to any sort of a solution at all. Ah well.
Back to top
View user's profile Send private message
bunder
Bodhisattva
Bodhisattva


Joined: 10 Apr 2004
Posts: 5904

PostPosted: Sat Jan 26, 2008 9:20 pm    Post subject: Reply with quote

kortec wrote:
So lots of RAM usage is normal.


bingo.

Code:
$ free -m
             total       used       free     shared    buffers     cached
Mem:          1010        973         37          0         76        546
-/+ buffers/cache:        349        661 <--- this is the line you want to look at
Swap:         1019          0       1019

_________________
Neddyseagoon wrote:
The problem with leaving is that you can only do it once and it reduces your influence.

banned from #gentoo since sept 2017
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum