Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Linux Memory Management or 'Why is there no free RAM?'
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
log
Apprentice
Apprentice


Joined: 16 Jun 2003
Posts: 168
Location: Adelaide, Australia

PostPosted: Fri Jun 18, 2004 1:56 am    Post subject: Reply with quote

After I upgraded to 1GB and enabled high memory, I did not notice a drop in performance. Although the costomised software im using uses more ram than CPU...oh well, im happy :D
Back to top
View user's profile Send private message
sapphirecat
Guru
Guru


Joined: 15 Jan 2003
Posts: 376

PostPosted: Fri Jun 18, 2004 3:51 pm    Post subject: Reply with quote

log wrote:
After I upgraded to 1GB and enabled high memory, I did not notice a drop in performance. Although the costomised software im using uses more ram than CPU...oh well, im happy :D

Alright, then. I'll mark it "mostly theoretical"; AFAIK when Linux has to look up something in the page tables with high memory support, it has to go through three layers rather than two.
_________________
Former Gentoo user; switched to Kubuntu 7.04 when I got sick of waiting on gcc. Chance of thread necro if you reply now approaching 100%...
Back to top
View user's profile Send private message
i_hate_your_os
Tux's lil' helper
Tux's lil' helper


Joined: 29 Aug 2002
Posts: 128
Location: Manhattan Beach, CA

PostPosted: Fri Jun 18, 2004 6:09 pm    Post subject: Reply with quote

The .88G limitation was news to me. Great thread, thanks!
_________________
-IHYOS

"All laws which are repugnant to the constitution are null and void."
-Marbury vs. Madison, 5 US (2 Cranch) 137, 174, 176, (1803)
Back to top
View user's profile Send private message
nadamsieee
Guru
Guru


Joined: 30 May 2003
Posts: 340
Location: Atlanta, GA, USA

PostPosted: Fri Jun 18, 2004 6:49 pm    Post subject: Reply with quote

sapphirecat wrote:
Alright, then. I'll mark it "mostly theoretical"; AFAIK when Linux has to look up something in the page tables with high memory support, it has to go through three layers rather than two.


There is definately a real, live performance hit introduced by the extra layer. There is nothing theoretical about it. What is debatable is whether the hit is ever noticable by us low frequency humans. Just nit-picking. :wink:
_________________
nadams (at) ieee (dot) org
Back to top
View user's profile Send private message
tommy_fila
Guru
Guru


Joined: 19 Nov 2003
Posts: 450
Location: Phoenix, AZ

PostPosted: Sat Jun 19, 2004 3:43 pm    Post subject: Reply with quote

For everyone who is wondering why their swappiness settings are always changing to seemingly random numbers:

Direct quote from bk0

Quote:
gentoo-dev-sources (and probably other gentoo 2.6 kernels) contains the Con Kolivas autoregulated swappiness patch. This means that the kernel automatically adjusts the /proc/sys/vm/swappiness value as needed during runtime, so any changes you make will be clobbered next time it updates. A good explanation of this patch and how it works is here on KernelTrap.

I repeat: With gentoo-dev-sources it is neither necessary nor possible to permanently adjust the swappiness value. It's taken care of automatically, no need to worry.


This was taken from the following thread:

https://forums.gentoo.org/viewtopic.php?p=1258300#1258300
_________________
"What goes on in life, that goes for eternity."
Back to top
View user's profile Send private message
Chonhulio
n00b
n00b


Joined: 04 Mar 2003
Posts: 6
Location: Germany

PostPosted: Sat Jun 19, 2004 4:02 pm    Post subject: Reply with quote

Quote:
There is definately a real, live performance hit introduced by the extra layer. There is nothing theoretical about it. What is debatable is whether the hit is ever noticable by us low frequency humans. Just nit-picking.


IIRC 3-level page tables are only used when CONFIG_HIGHMEM64G is set in order to support RAM above 4GB (PAE mode).
The usual CONFIG_HIGHMEM4G just changes the adress space layout and adds additional mappings for RAM above 1GB.
So the performance penalty should only occur when actually allocating RAM above 1GB and should be very mild even then.

The major, yet mostly theoretical, downside of CONFIG_HIGHMEM4G is, that the process space per single process is reduced to 2GB. Even if you got 4GB of RAM, each process may only allocate 2GB, unless you enable CONFIG_HIGHMEM64G. However, this should be pretty irrelevant on workstations and most servers.
Additionally some kernel features and 3rd party patches may not work reliably (or at all) on HIGMEM Kernels. Examples are cryptoapi up to 2.6.4, suspend-to-disk up to 2.6.6 and some poorly maintained drivers, including some binary only crap.
Back to top
View user's profile Send private message
Clansman
Tux's lil' helper
Tux's lil' helper


Joined: 10 Jan 2004
Posts: 140

PostPosted: Sun Jun 20, 2004 12:21 pm    Post subject: Reply with quote

hi everyone.

excelent post by sapphirecat!

yet, i think few people that posted here got the original post's point according to the post's replies... overall, about the memory size and occupancy one can conclude:

Having low free memory reported from 'top' DOES NOT MEAN that your system is desperately and unrecoverably occupying all of your memory (or most of it), meaning that you must rush to the nearest and most expensive store to buy another Gb of ram.

The linux kernel has a memory management policy that cleverly tries to use up as much memory as it possibly can, to improve performance. This performance improvement can be very visible if that used memory becomes disk cache because as stated before, hard drive acesses are about 1000 times slower than RAM, so getting disk's data (cache) from ram is very fast. The RAM is the fastest and second biggest memory available outside the cpu.

That big memory cache forcedly used by the linux kernel to avoid wasted memory can be VERY QUICKLY shrinked up to allow user applications to use memory/more memory.

So, having 70Mb of free ram when you have only 1Gb is not a bad thing and does not mean you have to buy new memory to boost performance. You would probably find out that with your new 2Gb of ram, you'd end up having... 100Mb free...

[]
Back to top
View user's profile Send private message
sanity
n00b
n00b


Joined: 19 May 2002
Posts: 39
Location: Iowa

PostPosted: Mon Jun 21, 2004 3:23 am    Post subject: swappiness Reply with quote

I'm still not getting what auto-swappiness optimizes for, but I don't use it anyway.

On all my machines, I sort of take the opposite approach. I set swappiness to 100. This is especially useful on my desktop. Consider: My brother runs GNOME. I run Fluxbox. We run them on different X servers, so that we can swtich off, leaving programs open and running, without sharing settings / giving him access to my settings (PGP key, etc. etc.)

Obviously, when he's on, he uses a lot of RAM. Now suppose I want to play ut2004. You see the problem?

Of course, his stuff can be swapped out. It's not likely to happen, though, unless I'm actually running out of memory, or unless swappiness is quite high. And if he's hogging the RAM, I don't get to use it for buffer/cache, which can really help speed sometimes. (Oops! Firefox crashed! (buggy flash plugin) Oh, but it's cached -- 1-2 seconds and I have a browser again.)

This is also helpful against fragmentation. This may be getting just a bit obsessive, but I use reiser4, and if I'm downloading a 250 meg file and I have 500 megs of RAM, it may very well stay buffered until the download is done, and then be written entirely contiguously.

Basic formula goes like this: if you think have enough RAM, you might set swappiness=0. Like, if you've got 3 gigs of RAM on a desktop machine, and you leave it on all night, cron jobs won't take significantly longer, and they won't push your apps out into swap -- thus, you come back and everything's just as you left it, no waiting around.

If you do not have enough RAM, or if you think you might be wasting some RAM, you might set swappiness=100. Now, all the things you aren't using go to swap, out of the way for things you are using, such as ut2004. The downside? After leaving the computer for awhile, it's quite likely that a cron job or a running download or something has kicked your email, your browser, even your terminal emulator out to swap, and thus, it might take 5 seconds for your computer to come back to life, and another 1-2 seconds each time you bring up a window that you haven't seen in awhile.

My last OS was Windows 98. 5 seconds I can live with, to avoid buying enough RAM to run GNOME, two instances of Fluxbox, 3 X servers, and ut2004.
Back to top
View user's profile Send private message
sploo22
n00b
n00b


Joined: 21 Aug 2003
Posts: 20
Location: Cayman Brac, Cayman Islands

PostPosted: Sun Jun 27, 2004 3:05 pm    Post subject: Reply with quote

Along these same lines, is it possible to force a program and its assorted libraries to stay in the cache, so e.g. I can have Firefox ready to start up even if I haven't been using it recently? Preferably something more robust than "dd if=/usr/bin/firefox of=/dev/null" every 30 seconds. Does the kernel provide this level of control?
_________________
This signature will self-destruct in 10 seconds. Close browser window now to avoid permanent monitor damage.
Back to top
View user's profile Send private message
Clansman
Tux's lil' helper
Tux's lil' helper


Joined: 10 Jan 2004
Posts: 140

PostPosted: Sun Jun 27, 2004 8:10 pm    Post subject: Reply with quote

sploo22 wrote:
Along these same lines, is it possible to force a program and its assorted libraries to stay in the cache, so e.g. I can have Firefox ready to start up even if I haven't been using it recently? Preferably something more robust than "dd if=/usr/bin/firefox of=/dev/null" every 30 seconds. Does the kernel provide this level of control?


i seriously doubt it. and i don't think the kernel is made to act that way. it would be application-dependant and would require personalized configurations, which is not the case on any of today's kernel.

nevertheless it could become a usable idea, in some patch-package...

[]
Back to top
View user's profile Send private message
dmitrio
Tux's lil' helper
Tux's lil' helper


Joined: 10 Dec 2002
Posts: 115
Location: Pago Pago

PostPosted: Mon Jun 28, 2004 5:25 pm    Post subject: :. copied to gentoo-wiki.com Reply with quote

I have copied this, with permission of sapphirecat and bk0, to gentoo-wiki.com
http://gentoo-wiki.com/Linux_Memory_Management
If you see anything that should be added or changed, feel free to do so.

Thank you for a great HOWTO.
_________________

... Leaving ground, destination is unknown,
into the darkness and far away from home,
Will your dream come true and what will you find,
when fate is your guide ...
Back to top
View user's profile Send private message
solarium_rider
Tux's lil' helper
Tux's lil' helper


Joined: 23 Jun 2003
Posts: 88
Location: San Francisco

PostPosted: Tue Jun 29, 2004 5:54 am    Post subject: Reply with quote

I had a quick question, since nothing is really mentioned (explicetely)on whether it's good or not if the swap file is full. Currently, when i try to view a mpeg from the web or something (through mplayer) the whole system becomes laggy (mouse pointer skips bad.) The desktop also seems laggy when I'm running an emerge sync. I'm using vanilla 2.6.6 kernel. Also is it normal for kdeinit to use 30% of my ram (according to top).

Here's what my stuff looks like:

Code:

ccase@miles ccase $ free -m
             total       used       free     shared    buffers     cached
Mem:           503        495          7          0          4         34
-/+ buffers/cache:        456         46
Swap:          517        517          0
ccase@miles ccase $ uptime
 01:49:06 up 16 days, 13:40,  1 user,  load average: 0.13, 0.14, 0.17
ccase@miles ccase $ cat /proc/meminfo
MemTotal:       515520 kB
MemFree:          8100 kB
Buffers:          4468 kB
Cached:          35068 kB
SwapCached:       2092 kB
Active:         460344 kB
Inactive:        18756 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       515520 kB
LowFree:          8100 kB
SwapTotal:      530136 kB
SwapFree:           68 kB
Dirty:               8 kB
Writeback:           0 kB
Mapped:         462644 kB
Slab:            15784 kB
Committed_AS:   994508 kB
PageTables:       3592 kB
VmallocTotal:   516056 kB
VmallocUsed:     39604 kB
VmallocChunk:   475300 kB
ccase@miles ccase $ cat /proc/sys/vm/swappiness
60


Any recommendations on whether I should switch kernels or just change the swapiness? Do you think my lag issues are cpu or mem related? I was under the impression the 2.6 kernel was suppose to have better scheduler for a desktop system.

Thanks
Back to top
View user's profile Send private message
Clansman
Tux's lil' helper
Tux's lil' helper


Joined: 10 Jan 2004
Posts: 140

PostPosted: Tue Jun 29, 2004 7:31 am    Post subject: Reply with quote

what's your swapiness level? do you set it manually?

you could compile a 2.6.7 kernel and include con kolivas' autoregulated swapiness patch in it. i use it.

or, from kerneltrap.org kerneltrap.org/node/view/3366 andrew morton's patchset with nick piggins' addicion could also help you out.

Quote:
Nick's patchset, which applies against the 2.6.7-mm3 kernel, provides a different CPU scheduler and additional memory management work. For the interested, Nick advises, "if anyone is having swapping or interactivity problems, please try it out.


[]
Back to top
View user's profile Send private message
rich0
Developer
Developer


Joined: 15 Sep 2002
Posts: 160

PostPosted: Thu Jul 08, 2004 12:15 pm    Post subject: Reply with quote

solarium_rider wrote:
I had a quick question, since nothing is really mentioned (explicetely)on whether it's good or not if the swap file is full. Currently, when i try to view a mpeg from the web or something (through mplayer) the whole system becomes laggy (mouse pointer skips bad.) The desktop also seems laggy when I'm running an emerge sync. I'm using vanilla 2.6.6 kernel. Also is it normal for kdeinit to use 30% of my ram (according to top).

Here's what my stuff looks like:

<snip>

Any recommendations on whether I should switch kernels or just change the swapiness? Do you think my lag issues are cpu or mem related? I was under the impression the 2.6 kernel was suppose to have better scheduler for a desktop system.

Thanks


My guess is that swappiness has nothing to do with it - only 34MB of your RAM is used by cache. It looks to me like some processes is hogging a lot of RAM. And if kdeinit is using 30% of your RAM with this much RAM in the system, that would seem odd to me...
Back to top
View user's profile Send private message
sanity
n00b
n00b


Joined: 19 May 2002
Posts: 39
Location: Iowa

PostPosted: Fri Jul 09, 2004 3:49 am    Post subject: various replies Reply with quote

Quote:
my HD is NOT quiet and when the CPU is idle it goes cacheing some of the disk, and very annoying because for 15+ min it's being really noisy .


For 15+ min? How much swap space do you have? Something tells me that you need more RAM on that thing -- my hd is idle when my box is idle, for the most part, and I only have 512 megs.

Quote:
Everything I have read about swapspace in linux says that it is more effiecient to have a swap partition even if you have a lot or memory.


Depends. In theory, if you are not using tmpfs and you're on a tiny filesystem (like a floppy or a cd) with much more ram than "hard" space, then swap doesn't add anything.

Only real-world case I can think of is if you can predict that you will never need to have more memory than your total amount of RAM. But by "needing memory", I include disk cache, and more cache is usually better.

Only real-world scenario I can think of where someone should choose not to have swap is if they need the space and/or they are running a very active server, on which almost none of the programs running will ever be idle.
_________________
Every time you read a signature, God kills a kitten.
Back to top
View user's profile Send private message
Rainmaker
Veteran
Veteran


Joined: 12 Feb 2004
Posts: 1650
Location: /home/NL/ehv/

PostPosted: Sun Jul 11, 2004 5:56 am    Post subject: Reply with quote

Maybe add this bit:

Your system only reports 28 Megs when you actually have 32 Megs. This is because part of the memory is used for the kernel code, and another part of it is "reserved". Check the ouput of dmesg to see how much this is. The less you "built into" your kernel (i.e. use modules) the smaller your kernel gets.
_________________
If you can't dazzle them with brilliance, baffle them with bullshit.
Back to top
View user's profile Send private message
jkruer01
Apprentice
Apprentice


Joined: 15 Jul 2004
Posts: 214
Location: Louisville, KY

PostPosted: Wed Aug 11, 2004 1:26 pm    Post subject: Reply with quote

Can someone please explain to me what the benefits of swaping out the contents of cache? The purpose of cache is to store content on a higher speed medium so you can access it faster. If you move the cach to the harddrive then doesn't it defeat the purpose of the cache?
Back to top
View user's profile Send private message
rich0
Developer
Developer


Joined: 15 Sep 2002
Posts: 160

PostPosted: Wed Aug 11, 2004 2:34 pm    Post subject: Reply with quote

jkruer01 wrote:
Can someone please explain to me what the benefits of swaping out the contents of cache?...If you move the cach to the harddrive then doesn't it defeat the purpose of the cache?


You don't swap out the cache - I don't think any decent OS does that at all.

What the debate is about is swapping out programs in memory in order to make room for cache.

It is undisputed that if idle software needs more RAM than you have physical memory for, then swapping is necessary to load new more-active programs.

What is disputed is whether the system should go ahead and swap out programs that are idle when it does not need that memory to run other programs. By swapping this memory out, it frees up RAM to use as a cache. On the other hand, if you don't swap it out, then if one of those programs wakes up it won't have to be read in.

Neither side is advocating swapping out the cache itself. Linux simply uses unused memory as the cache/buffers.
Back to top
View user's profile Send private message
jkruer01
Apprentice
Apprentice


Joined: 15 Jul 2004
Posts: 214
Location: Louisville, KY

PostPosted: Wed Aug 11, 2004 5:38 pm    Post subject: Reply with quote

Your right, I'm sorry. I didn't take enough time to read every word. Now that I went back and read the original post more slowly it makes much more since to me.

Thanks!
Back to top
View user's profile Send private message
Stieltje
Apprentice
Apprentice


Joined: 03 Apr 2003
Posts: 158
Location: The land where the sky is black

PostPosted: Wed Aug 18, 2004 8:02 am    Post subject: Reply with quote

I guess that gkrellm does not report free mem the same way that
Code:
 free -m
does


Code:


# free -m
             total       used       free     shared    buffers     cached
Mem:          1518        790        728          0        144        415
-/+ buffers/cache:        229       1288
Swap:         4000          0       4000




gkrellm reports that I have 1289 MB free...


Anyway, nice post, I guess I won't have to bother about swappiness though I have the latest ck-sources installed (2.6.7-ck5)

/Stieltje
_________________
PCs nearly always come with Microsoft Windows as their operating systems, but the "OS," as it is called, can be changed to a non-Microsoft operating system with a little work.
Back to top
View user's profile Send private message
sapphirecat
Guru
Guru


Joined: 15 Jan 2003
Posts: 376

PostPosted: Sun Aug 22, 2004 10:32 pm    Post subject: Reply with quote

Stieltje wrote:
Code:
# free -m
             total       used       free     shared    buffers     cached
Mem:          1518        790        728          0        144        415
-/+ buffers/cache:        229       1288

gkrellm reports that I have 1289 MB free...

Must have been in a different thread that I mentioned this... gkrellm pretty much does the same as free with -/+ buffers/cache; the difference between 1288 and 1289 is probably rounding error. It's also possible to go into gkrellm's memory options and make it show three krells, splitting things into used, buffers, cache, and completely free.
_________________
Former Gentoo user; switched to Kubuntu 7.04 when I got sick of waiting on gcc. Chance of thread necro if you reply now approaching 100%...
Back to top
View user's profile Send private message
Nutterpc
Tux's lil' helper
Tux's lil' helper


Joined: 02 Feb 2004
Posts: 83

PostPosted: Tue Aug 24, 2004 5:26 am    Post subject: Reply with quote

after having this problem myself.....I always wondered what I could do to try to counteract it

part of it came by reading available values in sysctl -a

part of it came via research......well........most of it really :)

Now yes, I will admit I used to have issues with linux seemingly runnin off with all my RAM, and then going to swap, but with 1Gb DDR400 DualChannel.........it shouldn't need to :?

After nearly a weeks worth of research so far, I've come up whit some fairly drastic changes in sysctl.conf, which, on my machine......are producing very noticeable results

Programs take less time to load, they don't hog Proc time (which they used to beforehang 8O )
CPU just blips every now and then, and Disk doesn;t even move...unless I load a program

And for all those wondering what kernel I'm running, its ck-sources latest: 2.6.8-ck3

Nutterpc

PS: if anyone else running this kernel would like to try my sysctl.conf settings out to help me further finetune them, my email addy should be in my profile......just contact me on msn ;)
_________________
If it isn't broke, you ain't tweaked it right
Registered Linux User 353232
Back to top
View user's profile Send private message
smg
Veteran
Veteran


Joined: 13 Aug 2004
Posts: 1402
Location: /home/stephan

PostPosted: Tue Oct 26, 2004 10:41 pm    Post subject: Reply with quote

Mh, I'm just asking me, what's faster for my system, 2*256 DDR-Ram @ 400Mhz working as dual ram modules or making out of this 1GB. Can you recommend me one of this possibilities?

best regards 'ash'
_________________
GnuPG-Key-ID: 0xF8C275D4
Fingerprint: 5B6F 134A 189B A24D 342B 0961 8D4B 0230 F8C2 75D4
Code:
perl -WTe '($")=$/;print qq(@{[reverse('0'..'100')]}$/BOOM!$/);'
Back to top
View user's profile Send private message
Nutterpc
Tux's lil' helper
Tux's lil' helper


Joined: 02 Feb 2004
Posts: 83

PostPosted: Wed Oct 27, 2004 7:05 am    Post subject: Reply with quote

all depends really, there's quite a few factors that come into it

a) how much memory do u need
b) how much do u want to spend on memory
c) is the $$ needed to get the extra ram going to make it worthwhile for what *you* do?
_________________
If it isn't broke, you ain't tweaked it right
Registered Linux User 353232
Back to top
View user's profile Send private message
minaguib
n00b
n00b


Joined: 03 Nov 2003
Posts: 35
Location: Montréal

PostPosted: Wed Oct 27, 2004 7:55 pm    Post subject: "SWAP" column of `top` misleading Reply with quote

To add to the excellent post, I've noticed something peculiar about `top`s output that I thought I'd point out.

At the header near the top, `top` says that I'm using no swap:
Code:
Swap:   265064k total,       12k used,   265052k free,   112472k cached


(well, almost no swap - 12k is negligible)

However, the "SWAP" column in the process listing lists several processes with huge chunks of "swap":
Code:
  PID USER      PR  VIRT  RES  SHR %CPU %MEM    TIME+  SWAP CODE DATA COMMAND
25274 mina      15  132m  55m  32m  0.0 22.3   1:52.33  76m   68  99m thunderbird-bin
 1960 mina      15 95628  32m  31m  3.4 12.9   0:32.40  61m   68  61m firefox-bin
 6076 root      15 92496  24m  69m  2.1  9.6   4:38.54  66m 1680  18m X
 1277 mina      15 46084 7360  10m  0.2  2.9   0:03.57  37m  984  33m xmms
25060 mina      16 24216 8572  13m  0.6  3.3   0:47.93  15m  796 9320 gkrellm2
 6819 mina      15  9508 6480 5840  0.0  2.5   0:32.25 3028  808 2860 irssi


This, of course, is contradictory. If there's no swap usage, no process could possibly have swapped data.

My personal conclusion (corrections welcome) is that `top` uses a fairly primitive way to calculate swap per process:

Code:
Swap = Total Usage - Resident Usage


It does not take into account mapped memory from non-resident sources, such as mapped files on disk, or memory on a graphics card.

In my example above, it's most likely that the 66 megs of memory assigned to "X" as swap is files mapped to disk and the memory on my el-cheapo graphics card.

And so my conclusion is, the "SWAP" column in the process list is not accurate and, for the most part, should be ignored.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4  Next
Page 3 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum