Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
System becomes unresponsive while compiling
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 531

PostPosted: Sat Oct 12, 2019 9:04 pm    Post subject: System becomes unresponsive while compiling Reply with quote

Greetings!

For a few months now I'm having problems with my system becoming slow and almost unresponsive while compiling big projects, such as Firefox.

This wasn't a problem in the past. Unfortunately I can't tell exactly when it started, nor which of the one million changes I've made since then could be responsible. I suspect it was a kernel update or kernel setting i inadvertendly changed - something CPU scheduling related? Or the result of CPU bug mitigations?

Factsheet:

- Ivy Bridge quadcore with hyperthreading (i7-3920XM)
- Kernel gentoo-4.19.75
- using CFS task scheduler (I think)
- SCHED_MC=y, CONFIG_SCHED_SMT=y
- Timer Frequency = 300Hz, CONFIG_PREEMPT_VOLUNTARY=y
- using deadline IO scheduler on an SSD
- /var/tmp/portage is entirely in tmpfs
- PORTAGE_NICENESS="15"
- PORTAGE_IONICE_COMMAND="ionice -c3 -p \${PID}"
- MAKEOPTS="-j8"
- plenty of spare RAM, no swapping taking place
- Load average during compilation reaches as high as 14

The problem is very evident in my running browser (Firefox), which always has some CPU background activity (I blame my dozens of open/loaded tabs) and spikes CPU usage when loading or interacting with sites. It becomes very slow. However it (nice=0) should always win over portage (nice=15).

It's frustating. Any ideas? Thanks!
Back to top
View user's profile Send private message
mike155
Veteran
Veteran


Joined: 17 Sep 2010
Posts: 1990
Location: Frankfurt, Germany

PostPosted: Sat Oct 12, 2019 9:14 pm    Post subject: Reply with quote

Is this a follow-up to https://forums.gentoo.org/viewtopic-p-8243856.html or something completely new?
Back to top
View user's profile Send private message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 531

PostPosted: Sat Oct 12, 2019 9:16 pm    Post subject: Reply with quote

mike155 wrote:
Is this a follow-up to https://forums.gentoo.org/viewtopic-p-8243856.html or something completely new?

I'm surprised someone remembers that thread :)

It's completely new. I haven't had the problem described in the old thread since upgrading to 32GB. Simply throwing memory at your problems helps!
Back to top
View user's profile Send private message
mike155
Veteran
Veteran


Joined: 17 Sep 2010
Posts: 1990
Location: Frankfurt, Germany

PostPosted: Sat Oct 12, 2019 10:15 pm    Post subject: Reply with quote

We had a couple of threads where users complained that their systems get unresponsive. That's why I looked back. If I rememeber correctly, we never found out what caused the issue...

The first thing you can do is: try to find out what actually triggers the problem. For example, is it the CPU load? What happens if you run 14 times
Code:
while [ 1 ]; do let i=1; done &

This will give you a load of 14. Does your system get unresponsive after a few minutes? Monitor the CPU core temperature (coretemp) using lm-sensors. Does it rise above 65°C?

If it's not the CPU load: add disk I/O or memory access to the test. If you know what causes the problem and if you can reproduce it, it will be easy to find a solution.


Last edited by mike155 on Sat Oct 12, 2019 10:37 pm; edited 1 time in total
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 4402
Location: Dallas area

PostPosted: Sat Oct 12, 2019 10:37 pm    Post subject: Re: System becomes unresponsive while compiling Reply with quote

haarp wrote:

- SCHED_MC=y, CONFIG_SCHED_SMT=y
- Timer Frequency = 300Hz, CONFIG_PREEMPT_VOLUNTARY=y


I turned off SCHED_MC and SCHED_SMT, as I found they didn't help on my system.
I also run the timer at 1000 and PREEMPT instead of PREEMPT_VOLUNTARY.
_________________
Asus m5a99fx, FX 8320 - nouveau, oss4, rx550 for qemu passthrough
Acer laptop E5-575, i3-7100u - i965, alsa
---both---
5.0.13 zen kernel, profile 17.1 (no-pie & modified) amd64-no-multilib
gcc 8.2.0, eudev, openrc, openbox, palemoon
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7368

PostPosted: Sun Oct 13, 2019 1:21 pm    Post subject: Re: System becomes unresponsive while compiling Reply with quote

Anon-E-moose wrote:
I also run the timer at 1000 and PREEMPT instead of PREEMPT_VOLUNTARY.

same, but with SCHED_MC=y, CONFIG_SCHED_SMT=y
Back to top
View user's profile Send private message
ulenrich
Veteran
Veteran


Joined: 10 Oct 2010
Posts: 1410

PostPosted: Sun Oct 13, 2019 2:16 pm    Post subject: Re: System becomes unresponsive while compiling Reply with quote

Anon-E-moose wrote:
haarp wrote:

- SCHED_MC=y, CONFIG_SCHED_SMT=y
- Timer Frequency = 300Hz, CONFIG_PREEMPT_VOLUNTARY=y


I turned off SCHED_MC and SCHED_SMT, as I found they didn't help on my system.
I also run the timer at 1000 and PREEMPT instead of PREEMPT_VOLUNTARY.

I used for a long time PREEMPT_VOLUNTARY and got hit of this unresponsiveness also. Yesterday I changed my .config and got a very responsive linux-5.3.6 now using:
Code:
# zcat /proc/config.gz |grep PREEMPT
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_PREEMPT_RCU=y
So my calculation to get better compiling performanc with VOLUNTARY got me hit in the ass lately. Seems PREEMPT_VOLUNTARY is not an option for anybody any more.
_________________
the thread ain't easily find an end
Back to top
View user's profile Send private message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 531

PostPosted: Mon Oct 28, 2019 3:32 pm    Post subject: Reply with quote

I've been experimenting with the MuQSS process scheduler lately (ck's successor to BFS) and setting my compilation jobs to SCHED_BATCH. this seems to work well and I maintain interactivity during heavy compiles.

For some weird reason tho, having heavy compile jobs running now seems to block new processes from spawning. e.g. opening a new terminal window either doesn't spawn any window at all, or I won't get a shell, until I stop/pause the compile job that is. Already running shells are unaffected.

Weirder and weirder...


Last edited by haarp on Tue Nov 12, 2019 1:57 pm; edited 1 time in total
Back to top
View user's profile Send private message
mike155
Veteran
Veteran


Joined: 17 Sep 2010
Posts: 1990
Location: Frankfurt, Germany

PostPosted: Mon Oct 28, 2019 4:10 pm    Post subject: Reply with quote

I really doubt that the 'unresponsiveness' is caused by CONFIG_PREEMPT_VOLUNTARY. It might well be that changing CONFIG_PREEMPT_XXXX helps, but something else is wrong.

Please post your kernel config using wgetpaste.
Back to top
View user's profile Send private message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 531

PostPosted: Mon Oct 28, 2019 6:51 pm    Post subject: Reply with quote

I agree.

mike155 wrote:
Please post your kernel config using wgetpaste.

here you go.
Back to top
View user's profile Send private message
mike155
Veteran
Veteran


Joined: 17 Sep 2010
Posts: 1990
Location: Frankfurt, Germany

PostPosted: Mon Oct 28, 2019 8:25 pm    Post subject: Reply with quote

Thanks for your kernel config.

My machine has an i5-3570K CPU and I run Linux kernel 4.19.75.

I compared your kernel config ('<') with mine ('>'). Below are the most important differences. There are more, but let's start with those.

  1. Code:
    < # Linux/x86 4.19.2-ck Kernel Configuration
    ---
    > # Linux/x86 4.19.72 Kernel Configuration

    4.19.2-ck? Please switch to the vanilla kernel, at least until the problem is solved.

  2. Code:
    < # CONFIG_NO_HZ is not set
    < # CONFIG_PREEMPT_VOLUNTARY is not set
    < CONFIG_PREEMPT=y
    < CONFIG_PREEMPT_COUNT=y
    ---
    > CONFIG_NO_HZ=y
    > CONFIG_PREEMPT_VOLUNTARY=y
    > # CONFIG_PREEMPT is not set

    You may want to try CONFIG_NO_HZ=y

  3. Code:
    > CONFIG_SCHED_AUTOGROUP=y

    Automatic process group scheduling is NOT enabled on your machine. You may want to try that. It's known to make a difference.

  4. Code:
    < CONFIG_ARCH_HIBERNATION_HEADER=y
    < CONFIG_SUSPEND=y
    < CONFIG_SUSPEND_FREEZER=y
    < CONFIG_HIBERNATE_CALLBACKS=y
    < CONFIG_HIBERNATION=y
    < CONFIG_PM_STD_PARTITION=""
    < CONFIG_PM_SLEEP=y
    < CONFIG_PM_SLEEP_SMP=y
    < # CONFIG_PM_AUTOSLEEP is not set
    < CONFIG_PM_WAKELOCKS=y
    < CONFIG_PM_WAKELOCKS_LIMIT=100
    < CONFIG_PM_WAKELOCKS_GC=y
    ---
    > # CONFIG_SUSPEND is not set

    You could try to disable Hibernation and Suspend. I doubt that it will make a difference, but it's worth a try...
Back to top
View user's profile Send private message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 531

PostPosted: Mon Oct 28, 2019 9:10 pm    Post subject: Reply with quote

Thanks for your efforts!

mike155 wrote:

4.19.2-ck? Please switch to the vanilla kernel, at least until the problem is solved.

I was using vanilla kernels at the start of this thread. I only switched to -ck a few days ago to see if its improved process scheduler would help. See my reply earlier today. -ck uses the same kernel config with only minor differences due to enabling of MuQSS.

Quote:
You may want to try CONFIG_NO_HZ=y

This is basically enabled through CONFIG_NO_HZ_IDLE=y:
Code:
CONFIG_NO_HZ
This is the old config entry that enables dynticks idle.
We keep it around for a little while to enforce backward
compatibility with older config files.


Quote:
Automatic process group scheduling is NOT enabled on your machine. You may want to try that. It's known to make a difference.

It was enabled on the vanilla kernel which showed the problems described in the start post. It's disabled now because it's incompatible with MuQSS.


Quote:
You could try to disable Hibernation and Suspend. I doubt that it will make a difference, but it's worth a try...

I don't see how these could hurt. They're also features I need every day, so disabling them is not an option.
Back to top
View user's profile Send private message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 531

PostPosted: Tue Nov 12, 2019 1:59 pm    Post subject: Reply with quote

The problem I had in
haarp wrote:
I've been experimenting with the MuQSS process scheduler lately (ck's successor to BFS) and setting my compilation jobs to SCHED_BATCH. this seems to work well and I maintain interactivity during heavy compiles.

For some weird reason tho, having heavy compile jobs running now seems to block new processes from spawning. e.g. opening a new terminal window either doesn't spawn any window at all, or I won't get a shell, until I stop/pause the compile job that is. Already running shells are unaffected.

has been solved by switching scheduler class SCHED_IDLEPRIO. I don't know why or how SCHED_BATCH has such an effect.

I'd mark this thread as solved, but it still feels like I merely worked around a more serious problem. I wonder if anyone has some insight?


Last edited by haarp on Tue Nov 12, 2019 6:51 pm; edited 1 time in total
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 44945
Location: 56N 3W

PostPosted: Tue Nov 12, 2019 4:45 pm    Post subject: Reply with quote

haarp,


Code:
- MAKEOPTS="-j8"
- plenty of spare RAM, no swapping taking place

...

32G RAM ..


By default you have 16G max of tmpfs and MAKEOPTS="-j8" can consume 16G of RAM on large C++ projects.
tmpfs is flexible. It only takes an much RAM as it needs, up to the maximum, so if your builds wanted more that 16G, its probably OK.

It does not follow that because swap is not being used that there is no swapping happening.
swap space is used only for RAM that does not have a permanent home on disk. Dynamically allocated RAM.
The kernel has other ways to swap too.
It can drop read only data and reload it when required. That includes caches and pages of executable code too.
So the kernel can drop and reload bits of Firefox while you use it. You get lots of page faults and context switches
which do nothing for program execution but take a long time.
The kernel can flush 'dirty' RAM to disk and drop the content.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 3546
Location: Illinois, USA

PostPosted: Tue Nov 12, 2019 5:06 pm    Post subject: Reply with quote

NeddySeagoon wrote:
By default you have 16G max of tmpfs and MAKEOPTS="-j8" can consume 16G of RAM on large C++ projects.
tmpfs is flexible. It only takes an much RAM as it needs, up to the maximum, so if your builds wanted more that 16G, its probably OK.

How to increase that maxium for those of us with 32->64 GB RAM?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 44945
Location: 56N 3W

PostPosted: Tue Nov 12, 2019 5:17 pm    Post subject: Reply with quote

Tony0945,

The default is 50% RAM. Give the size option in fstab. Half my 16G is no longer enough to build things.
Code:
# build in RAM but 50% RAM is now too small
# and the motherboard is maxed out.
/dev/shm                 /var/tmp/portage      tmpfs noatime,nodev,nosuid,size=9G   0 0

size=n% works too.

For completeness, changing the size when something is building works.
The content of tmpfs is not lost.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 3546
Location: Illinois, USA

PostPosted: Tue Nov 12, 2019 6:25 pm    Post subject: Reply with quote

Awesome, Neddy. Thanks!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum