Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Gentoo freezes when it runs out of RAM
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
mikexx
n00b
n00b


Joined: 24 Aug 2018
Posts: 44

PostPosted: Wed Nov 14, 2018 3:58 pm    Post subject: Gentoo freezes when it runs out of RAM Reply with quote

Hi,

my laptop is very slow/freezes when it runs out of memory. The laptop has 16GB RAM and 16GB swap (SSD). However, as soon as ca 15 GB (RAM) are used the system freezes, for example moving the mouse pointer or enter the password in the lock screen takes more than one hour. I can observe that when chromium is emerged or when several VMs are open.
I am posting that in the kernel forum as I think it is because of a kernel configuration.
Any ideas?

Best
mike
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2549

PostPosted: Wed Nov 14, 2018 4:26 pm    Post subject: Reply with quote

What do the logs say? Is there anything in any logs concurrent with the frozen state? Or before?

As well, have you definitely linked this to an out-of-memory scenario in more than one instance?

I have a Gentoo box with the same memory, with some amount of swap configured (don't know off hand) but it does not have this problem.

I would suspect that, if it's actually swap-related, there would be some sort of log messages about swap. Or disk access. Or, if it's related to something else you should get messages about that. The fact that you still get extremely slow activity means it's not a kernel panic, so there should be a log message somewhere.

What comes to mind is an extreme thrashing situation. Or deadlocks without a means to break out.
Back to top
View user's profile Send private message
mikexx
n00b
n00b


Joined: 24 Aug 2018
Posts: 44

PostPosted: Wed Nov 14, 2018 7:18 pm    Post subject: Reply with quote

I didn't see anything suspicious last time. I will check it again.

Best
mike
Back to top
View user's profile Send private message
mikexx
n00b
n00b


Joined: 24 Aug 2018
Posts: 44

PostPosted: Thu Nov 15, 2018 6:01 am    Post subject: Reply with quote

I've checked the kernel logs again. Nothing has been logged about swap or something else what could explain the behavior.


Best
mike
Back to top
View user's profile Send private message
guitou
Guru
Guru


Joined: 02 Oct 2003
Posts: 399
Location: France

PostPosted: Thu Nov 15, 2018 12:53 pm    Post subject: Reply with quote

Hello.

Most probably a silly question, but did you double check that your swap part is active?

++
Gi)
Back to top
View user's profile Send private message
P.Kosunen
Guru
Guru


Joined: 21 Nov 2005
Posts: 309
Location: Finland

PostPosted: Fri Nov 16, 2018 3:24 pm    Post subject: Reply with quote

All systems freeze when swapping start. You can disable swap to not to freeze, but then some processes die when you run out of memory.
Back to top
View user's profile Send private message
mikexx
n00b
n00b


Joined: 24 Aug 2018
Posts: 44

PostPosted: Sun Nov 18, 2018 9:02 pm    Post subject: Reply with quote

@guitou
Swap is active.

@P.Kosunen
I am almost sure that I used systems in the past that did not freeze that way.


Best
mike
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2549

PostPosted: Tue Nov 20, 2018 7:08 pm    Post subject: Reply with quote

Re: freezing when swapping starts, it's usually transparent to the user. Operations that normally happen in RAM are stopped, and disk activity happens to page RAM out to the swap device.

If your system has a noticeable lag when swap starts there's a problem, or you have a really slow system in the first place, and a slow disk. If the system gets into a really bad place, swapping out huge amounts of RAM to disk and swapping other huge amounts back, then you can get noticeable lag. If the system is about to fail in spite of the swap, the lag can be severe.

That said, the very premise of swap is simply a last-ditch effort to keep a process working even though memory ran out. It's supposed to be a rare thing, not something that happens all the time. If you frequently have swap activity on your system you will get significant performance hits on long-running processes, and you will also have a higher probability of disk failure.
Back to top
View user's profile Send private message
PCmaniaK
n00b
n00b


Joined: 18 Aug 2018
Posts: 15

PostPosted: Tue Nov 20, 2018 10:36 pm    Post subject: Reply with quote

My two cents: Do not USE="jumbo-build" emerging chromium. It takes all the memory and asks for more.
Back to top
View user's profile Send private message
mikexx
n00b
n00b


Joined: 24 Aug 2018
Posts: 44

PostPosted: Wed Nov 21, 2018 4:54 pm    Post subject: Reply with quote

1clue wrote:
Re: freezing when swapping starts, it's usually transparent to the user. Operations that normally happen in RAM are stopped, and disk activity happens to page RAM out to the swap device.

If your system has a noticeable lag when swap starts there's a problem, or you have a really slow system in the first place, and a slow disk. If the system gets into a really bad place, swapping out huge amounts of RAM to disk and swapping other huge amounts back, then you can get noticeable lag. If the system is about to fail in spite of the swap, the lag can be severe.

That said, the very premise of swap is simply a last-ditch effort to keep a process working even though memory ran out. It's supposed to be a rare thing, not something that happens all the time. If you frequently have swap activity on your system you will get significant performance hits on long-running processes, and you will also have a higher probability of disk failure.


My system should not be very slow. It is a Thinkpad T480 with an i7, 16GB RAM and 512 GB SSD. Sure, it is normal that systems get slow as soon as swap starts, however, it should not take more than one hour to unlock the lock screen. It does not happen very often, only during system updates or if I have to start several virtual machines and because of other processes the system runs out of memory.

cheers
mike
Back to top
View user's profile Send private message
C5ace
Apprentice
Apprentice


Joined: 23 Dec 2013
Posts: 277
Location: Brisbane, Australia

PostPosted: Thu Nov 22, 2018 10:32 am    Post subject: Reply with quote

Run sys-process/iotop in a terminal and see which process reads and writes from and to disk.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5761

PostPosted: Thu Nov 22, 2018 3:12 pm    Post subject: Reply with quote

What disk scheduler are you using? (grep '' /sys/block/*/queue/scheduler)
Back to top
View user's profile Send private message
Goverp
l33t
l33t


Joined: 07 Mar 2007
Posts: 668

PostPosted: Fri Nov 23, 2018 9:35 am    Post subject: Reply with quote

I'd say you need to dissect /proc/meminfo. I've been reading up on things like HugePage support, which sounds great, but HugePages are unswappable. (Not saying this has anything to do with this particular situation, just that there be sharks in them there waters. I'm a complete novice here.) There are probably lots of other things that meminfo might be able to warn of.

One thing I found related to emerges - you'll find recommendations to use tmpfs for portage temporary files. And it's great for small apps. BUT if you use it for emerging big ones like Rust, webengine, LLVM, and chromium, it can be a disaster. You end up with the system having to swap out the temporary files to make space for the compilation, which wants to read the temp files and so swaps them back in. In essence, don't overcommit memory - tmpfs should be no more than say 50% of memory. I expect this gets even worse if you use "-pipe" as also recommended. Basically, for the big beasts, use package.env settings to switch OFF all the "go-faster" settings or your machine grinds to a halt.

Unfortunately this sort of problem becomes increasingly common - developers assume vast amounts of memory for cache, big data trees and so forth, and forget they are cohabiting with other apps doing the same. Many years ago I used an IBM mainframe with less than 1MB memory running a multi-user VSE/CICS system under VM, and we had "balanced systems guidelines" that reflected best practice, and explained how to identify memory hogs. These days I doubt even a keyboard poll routine would be rated "good".
_________________
Greybeard
Back to top
View user's profile Send private message
mikexx
n00b
n00b


Joined: 24 Aug 2018
Posts: 44

PostPosted: Mon Nov 26, 2018 5:36 pm    Post subject: Reply with quote

Ant P. wrote:
What disk scheduler are you using? (grep '' /sys/block/*/queue/scheduler)

Code:

/sys/block/dm-0/queue/scheduler:none
/sys/block/dm-1/queue/scheduler:none
/sys/block/dm-2/queue/scheduler:none
/sys/block/dm-3/queue/scheduler:none
/sys/block/dm-4/queue/scheduler:none
/sys/block/loop0/queue/scheduler:[mq-deadline] kyber none
/sys/block/loop1/queue/scheduler:[mq-deadline] kyber none
/sys/block/loop2/queue/scheduler:[mq-deadline] kyber none
/sys/block/loop3/queue/scheduler:[mq-deadline] kyber none
/sys/block/loop4/queue/scheduler:[mq-deadline] kyber none
/sys/block/loop5/queue/scheduler:[mq-deadline] kyber none
/sys/block/loop6/queue/scheduler:[mq-deadline] kyber none
/sys/block/loop7/queue/scheduler:[mq-deadline] kyber none
/sys/block/nvme0n1/queue/scheduler:[none] mq-deadline kyber     # <- my ssd (encrypted)
/sys/block/sda/queue/scheduler:noop deadline [cfq]
/sys/block/sdc/queue/scheduler:noop deadline [cfq]


@Goverp
Thanks. I will look at that.

Best
mike
Back to top
View user's profile Send private message
tholin
Apprentice
Apprentice


Joined: 04 Oct 2008
Posts: 168

PostPosted: Mon Nov 26, 2018 8:50 pm    Post subject: Reply with quote

When you say "my ssd (encrypted)" does that mean you have an encrypted swap partition with dm-crypt? If so that it likely the problem.

Swap on dm-crypt is problematic. I don't know for sure what the problem is but I suspect dm-crypt does memory allocations when data is encrypted. The problem with that is you are already low on memory when data is written to swap. If dm-crypt needs to do memory allocations before writing the data those allocations will force out even more data, which will result in even more memory allocations, which will force out even more data... [repeat]
The result is a long stall.

You can test if that is the case by reformatting the encrypted swap partition as a regular swap partition.

https://bugzilla.kernel.org/show_bug.cgi?id=200105
Here is a bug report I found while googling the problem a while back. There is no final conclusion in that report.
Back to top
View user's profile Send private message
mikexx
n00b
n00b


Joined: 24 Aug 2018
Posts: 44

PostPosted: Tue Nov 27, 2018 6:24 am    Post subject: Reply with quote

tholin wrote:
When you say "my ssd (encrypted)" does that mean you have an encrypted swap partition with dm-crypt? If so that it likely the problem.

Swap on dm-crypt is problematic. I don't know for sure what the problem is but I suspect dm-crypt does memory allocations when data is encrypted. The problem with that is you are already low on memory when data is written to swap. If dm-crypt needs to do memory allocations before writing the data those allocations will force out even more data, which will result in even more memory allocations, which will force out even more data... [repeat]
The result is a long stall.

You can test if that is the case by reformatting the encrypted swap partition as a regular swap partition.

https://bugzilla.kernel.org/show_bug.cgi?id=200105
Here is a bug report I found while googling the problem a while back. There is no final conclusion in that report.


When I wrote 'encrypted' I thought that could be the problem. :)
Yes swap is encrypted with dm-crypt.

Best
mike
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum