Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Computer freeze when RAM near full
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
mr-simon
Guru
Guru


Joined: 22 Nov 2002
Posts: 364
Location: Leamington Spa, Warks, UK

PostPosted: Tue Jul 05, 2016 1:14 pm    Post subject: Computer freeze when RAM near full Reply with quote

Not sure if this belongs here or in desktop, but here goes.

When running close to full memory my machine becomes very unresponsive and will eventually freeze.

I've got 12G installed (3-channel, 4G each) but run multiple VMs under vagrant/virtualbox, along with a desktop environment and firefox, etc.

Swap is enabled, 16G. On my SSD.

I've noticed that it's treacherous running multiple VMs because as my RAM starts to fill up I reach a problem where suddenly everything slows to a crawl and will eventually freeze. It's primarily characterised by issues with moving my mouse/keyboard. My mouse will jump & freeze, with the freezes taking progressively longer. Similarly, when pressing capslock there will be a progressively longer delay before the capslock light toggles on and off.

If I'm not quick enough (the above process only takes about 60sec, tops) then my computer will be completely frozen, and even leaving it overnight to sort itself out doesn't bring about any changes.

I've also noticed the above symptoms when doing other RAM intensive tasks such as compiling something with webkit in it.

Killing the memory-intensive processes (nuking a VM, killing 'emerge') solves the problem and the computer quickly returns to normal.

Swap never gets very full during this process. There's nothing in dmesg that indicates that the kernel has anything to say on the matter.

Any suggestions, please?
_________________
"Pokey, are you drunk on love?"
"Yes. Also whiskey. But mostly love... and whiskey."
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7256
Location: almost Mile High in the USA

PostPosted: Tue Jul 05, 2016 2:09 pm    Post subject: Reply with quote

More RAM or shrink VM images, adding swap to each individual VM.

Most VMMs will disallow swapping of VM images on the host, it becomes too obvious to the guest OS.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
mr-simon
Guru
Guru


Joined: 22 Nov 2002
Posts: 364
Location: Leamington Spa, Warks, UK

PostPosted: Tue Jul 05, 2016 2:12 pm    Post subject: Reply with quote

Thanks, but as I mentioned it's not just VMs - any RAM intensive operation will do it.

When running VMs I'm leaving a minimum of 4G headroom for my host OS, so there still should be enough RAM to play with.
_________________
"Pokey, are you drunk on love?"
"Yes. Also whiskey. But mostly love... and whiskey."
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7256
Location: almost Mile High in the USA

PostPosted: Tue Jul 05, 2016 2:20 pm    Post subject: Reply with quote

Suggestions is still the same. Run 'top' and make sure the sum of all RSS of all VM images isn't adding up to a significant portion of your physical memory.

Merging C++ programs use a lot of ram, especially when building multithreaded. VMs will choke off swap making it hard for your PC to do its job.

So 12GB RAM, two 4GB VM images, is that all? How much ram is free in a quiescent state?

[EDIT]
Just my VM machine. I only have 8GB RAM on my machine, but minimize the VM containers (KVM/QEMU) and the machine runs OK. I had less than 8GB RAM before and kept on running into RAM issues as the VMs grew in size or ran multiple VMs. A LOT of headroom is needed I've found out, but it may be due to KVM/QEMU.
Code:
top - 08:38:24 up 28 days,  8:40,  2 users,  load average: 0.04, 0.04, 0.05
Tasks: 205 total,   1 running, 204 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us,  0.0 sy,  0.0 ni, 99.7 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8035232 total,   799000 free,  2208420 used,  5027812 buff/cache
KiB Swap: 16777212 total, 16205652 free,   571560 used.  5364876 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU     TIME+ P   SWAP COMMAND
 3448 qemu      20   0 4681696 1.146g   2816 S   0.7 413:30.34 1 248248 qemu-s+
 3431 qemu      20   0 2082292  54668   2796 S   0.0 105:34.01 3 241480 qemu-s+
 3160 mysql     20   0  653372  54404    860 S   0.0  11:49.91 0  59044 mysqld 
26347 user      20   0  461832  17228   6140 S   0.0   0:20.56 1      0 xfdesk+
21937 root      20   0  383336  13436   4384 S   0.0   5:23.39 1      0 X       


The two VMs are 1.0GB and 128MB in size. The 1.0GB VM is the much more active one, and has grew past 1GB oddly enough. Need to keep this in mind.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
Logicien
Veteran
Veteran


Joined: 16 Sep 2005
Posts: 1374
Location: Montréal

PostPosted: Tue Jul 05, 2016 3:38 pm    Post subject: Reply with quote

With VirtualBox it's a good idea to not give more Ram to the virtual machine than the quantity recommanded by VirtualBox.

You can use the sysctl command to tune the kernel variable vm.swappiness . The value of 0 limit the usage of the swap to the strict minimum, higher values allow swapping more.

Swappiness
_________________
Paul
Back to top
View user's profile Send private message
mr-simon
Guru
Guru


Joined: 22 Nov 2002
Posts: 364
Location: Leamington Spa, Warks, UK

PostPosted: Tue Jul 05, 2016 3:58 pm    Post subject: Reply with quote

Well, here's my world under normal load. Note there's a VMWare VM which has been given 6G to play with, and a VBox VM (headless, under vagrant) that has 2G. This is a normal workload for me.

Code:
top - 16:46:29 up 7 min,  1 user,  load average: 1.97, 2.56, 1.29
Tasks: 287 total,   1 running, 285 sleeping,   0 stopped,   1 zombie
%Cpu(s):  1.4 us,  3.7 sy,  2.4 ni, 92.2 id,  0.2 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 12276548 total,    86288 free,  4557708 used,  7632552 buff/cache
KiB Swap: 15999996 total, 15481452 free,   518544 used.  1205860 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND     
 4506 simon     20   0 8514004 6.040g 5.988g S   7.3 51.6  13:17.67 vmware-vmx 
 3691 simon     20   0 1383656 320584  70996 S   0.3  2.6   0:33.56 firefox     
 3471 simon     20   0 3508100 119268  57584 S   3.3  1.0   0:25.87 HipChat.bin
 6066 simon     20   0 3543400  89624  54496 S  21.9  0.7   2:50.88 VBoxHeadle+
 3211 simon     20   0 1730180 173916  46396 S   2.3  1.4   0:32.27 gnome-shell
 3564 simon     20   0 2180712 133372  42648 S  13.3  1.1   1:06.00 QtWebEngin+
 3207 root      20   0  191192  58964  28964 S   1.3  0.5   0:28.85 X           
 5044 simon     20   0  866820  29696  19456 S   0.3  0.2   0:00.72 gnome-term+
 2729 root      20   0  296448  21372  15376 S   0.0  0.2   0:00.07 apache2     
 5414 simon     20   0  648240  13876  10180 S   0.0  0.1   0:42.15 VBoxSVC     
 3755 simon     20   0  601524  53092   9976 S   0.0  0.4   0:03.17 vmware     
 3851 simon     20   0  343948  23040   9308 S   0.0  0.2   0:00.72 vmware-tray
 3563 simon     20   0 1416452  16012   8868 S   0.0  0.1   0:00.01 QtWebEngin+
 2618 mysql     20   0 3268956 932672   8336 S   0.0  7.6   0:00.53 mysqld     
 4471 simon     20   0  225804  15028   7984 S   0.0  0.1   0:00.31 vmware-uni+
 5409 simon     20   0  100448   9180   7240 S   0.0  0.1   0:00.57 VBoxXPCOMI+
 3473 simon     20   0  814916  39420   7112 S   0.3  0.3   0:02.37 mono       


VMware sits at 6G SHR and doesn't go anywhere. VBoxHeadless seems to want to be 'nicer' about how it uses its memory. I haven't ever seen it grow past 2G but I haven't really had a chance to diagnose it properly at the point of failure due to it crashing.

At the moment it's running fine, but I wouldn't dare run emerge whilst this is going on as I'm pretty sure it would lock as described.

My vm.swappiness is set to 60 currently (the default) - I'm not sure whether up or down would help... I'm thinking up? I don't have any frame of reference as to what a high number or low number should be. Nevertheless it would seem like if it were a choice between swap or freeze, the kernel should be choosing 'swap' irrespective.

[edit]Sorry, just noticed that there was a link to the docs on swappiness in the last post. It seems like 60 is a reasonable default for me but I'll try pushing it up to 90 to see what happens.[/edit]
_________________
"Pokey, are you drunk on love?"
"Yes. Also whiskey. But mostly love... and whiskey."
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7256
Location: almost Mile High in the USA

PostPosted: Tue Jul 05, 2016 4:19 pm    Post subject: Reply with quote

Yeah you do have a problem with your machine. Changing swappiness won't help.

Something is locking a lot of memory. You only have a bit over 1GB of memory free to work with, and this is NOT enough to do multithreaded C++ builds. Could you sort by resident memory size instead of shared memory? Then again the problem seems to be in the buffer utilization... what else is special about this machine? Special hardware? special (buggy?) filesystems? Kernel memory leaks (what version)? hmm...
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
mr-simon
Guru
Guru


Joined: 22 Nov 2002
Posts: 364
Location: Leamington Spa, Warks, UK

PostPosted: Tue Jul 05, 2016 4:38 pm    Post subject: Reply with quote

By RES:

Code:
top - 17:39:48 up  1:00,  1 user,  load average: 2.78, 3.99, 3.71
Tasks: 287 total,   1 running, 285 sleeping,   0 stopped,   1 zombie
%Cpu(s):  0.8 us, 24.0 sy,  0.0 ni, 75.0 id,  0.2 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 12276548 total,    71120 free,  4034680 used,  8170748 buff/cache
KiB Swap: 15999996 total, 14636568 free,  1363428 used.  1720276 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND     
 4506 simon     20   0 8514004 6.143g 6.088g S  87.7 52.5 129:56.83 vmware-vmx 
 3691 simon     20   0 1945416 526828  68924 S   3.0  4.3  11:54.18 firefox     
 8826 simon     20   0  709844 228064   9452 S   0.3  1.9   0:17.08 ruby       
 3211 simon     20   0 1740984 202480  45056 S   2.3  1.6   4:32.52 gnome-shell
 3564 simon     20   0 2277172 147944  42804 S   1.0  1.2   3:58.06 QtWebEngin+
 3471 simon     20   0 3509908 141992  59964 S   0.0  1.2   1:34.65 HipChat.bin
 3549 simon     20   0 4305724 112372  12180 S   1.3  0.9   0:04.88 dropbox     
 9537 simon     20   0 3545276  93736  58776 S 103.0  0.8  17:41.04 VBoxHeadle+
 2618 mysql     20   0 3268956  87800   7204 S   0.0  0.7   0:01.43 mysqld     
 3207 root      20   0  191328  57956  27956 S   1.0  0.5   2:59.87 X           
 3755 simon     20   0  601524  53272  10156 S   0.0  0.4   0:05.79 vmware     
 3414 simon     20   0  779396  41708   5152 S   0.0  0.3   0:00.09 evolution-+
 3473 simon     20   0  882720  41324   7432 S   0.3  0.3   0:10.08 mono       
 3426 simon     20   0  858564  40836   5236 S   0.0  0.3   0:00.07 evolution-+
 3438 simon     20   0  712476  40364   4888 S   0.0  0.3   0:00.06 evolution-+
 5044 simon     20   0  867012  26976  16428 S   0.3  0.2   0:05.06 gnome-term+
 3851 simon     20   0  343948  22528   8796 S   0.0  0.2   0:02.51 vmware-tray


I'm running stable gentoo-sources. Currently 4.4.6.

Mostly my partitions are ext4 but I do have an ntfs (fuse) partition mounted.

I don't think there's anything particularly unusual about my hardware. It's an Intel X58 with an NVidia graphics card.

Are there any particular tools that can help me figure out if my buffers are being used (im)properly?

I'm not sure that it's leaking... Looks like my 'avail Mem' has drifted slightly upwards since my last post and I've not really changed what I'm doing.

[edit]Only pasted a partial 'top' the first time and VBoxHeadless wasn't even on there[/edit]
_________________
"Pokey, are you drunk on love?"
"Yes. Also whiskey. But mostly love... and whiskey."
Back to top
View user's profile Send private message
Logicien
Veteran
Veteran


Joined: 16 Sep 2005
Posts: 1374
Location: Montréal

PostPosted: Tue Jul 05, 2016 11:29 pm    Post subject: Reply with quote

I think that swap from memory to hard disk and reverse is not a good idea in itself. It's just a workaround for the lack of Ram. Enough Ram mean no swap. I think that for programs execution, the processor is better to stay on the same bus. It's the reason why vm.swapinness=0 in all my machines.

I prefer the kernel to delete pages in Ram and read again the binary code to load it in memory again. In both cases, swap and not swap implies hard disk access but, less the code move from memory best it is in my opinion.

Your problem can effectively, like tell you eccerr0r, come from a process who is devouring memory and/or the processor time and/or a bus bandwitdth. Maybe you will have better results if you disable swap completely for at least, testing it.

The Linux kernel configuration may be a cause too.
_________________
Paul


Last edited by Logicien on Tue Jul 05, 2016 11:57 pm; edited 1 time in total
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7256
Location: almost Mile High in the USA

PostPosted: Tue Jul 05, 2016 11:54 pm    Post subject: Reply with quote

It looks like your machine has low uptime, so you have an easy way of testing this:
Try loading applications one by one and see which one jumps your buffer cache up and free memory down.

Perhaps check these numbers on a fresh boot without things loaded.

One possibility that I do not know much about is perhaps something is setting your cgroups to limit the amount of RAM you have. Perhaps a bad configuration somewhere. Again I don't know much about cgroups and how systemd uses them, but perhaps this will jog your memory on some config option that may have been set somewhere by mistake.

(and I'm assuming you're using systemd since it looks like you're using gnome? Or is that a bad assumption?)
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
gargoylle_ltk
n00b
n00b


Joined: 28 Jun 2004
Posts: 33

PostPosted: Wed Jul 06, 2016 1:02 am    Post subject: Reply with quote

Try turning swap off as well - I've seen machines under high memory pressure (running MongoDB databases) crawl to a halt when trying to swap but remaining quite responsive without it. The flipside is that the kernel WILL start to kill processes as the machine starts to run out of RAM.
Back to top
View user's profile Send private message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 527

PostPosted: Wed Jul 06, 2016 9:05 am    Post subject: Reply with quote

You're using VMware too? I've had trouble with very sluggish/freezing hosts systems while VMs were running, even below max memory usage. Turns out that hugepaged defrag was screwing things up big time.

you could try
Code:
echo never > /sys/kernel/mm/transparent_hugepage/defrag


More: http://comments.gmane.org/gmane.org.user-groups.linux.boston.discuss/50011
Back to top
View user's profile Send private message
duby2291
Guru
Guru


Joined: 17 Oct 2004
Posts: 583

PostPosted: Wed Jul 06, 2016 12:50 pm    Post subject: Reply with quote

Well, you are obviously swapping. Um, the only thing you can do is add more ram. A lot more ram. At 16GB, I'd increase it to 64GB. You can increase the swappiness value, which just spreads swaps out over a greater distance of time and can make it less terrible. But really the only solution is much more ram.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum