Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Kernel not able to keep time and latency issues.
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
silent_Walker
Tux's lil' helper
Tux's lil' helper


Joined: 25 Jul 2012
Posts: 100
Location: forums.gentoo.org

PostPosted: Fri Sep 04, 2015 6:50 pm    Post subject: Kernel not able to keep time and latency issues. Reply with quote

I compiled a kernel, 4.0.9. The tree can be found at :

https://github.com/volkswagen/mycustomkernel

The problem that I am having is that after about 15 minutes ( could be more or less ) the time becomes off, it seems that the system thinks that 1 minute in real life is 1 second. The mobo. battery is a new battery. Also when the computer forgets what a second is, I get really bad latency, it takes about 30 seconds for a menue to pop up while I am hovering over it. I can force the menue open by highlighting it and hiting the enter key. Keystrokes also begin to lag behind. Since latency is horrible multitasking is impossible. But what is weird is during this time throughput is high. As reported by time ( maybe not the best tool) but the sys line under time is like .20 when the system can keep time but when the system can not keep time and latency goes down sys is like .07 or .01.

I did edit cfq-iosched.c I also editied fair.c radeon.h ( I get way better fps about 700 increase according to glx gears 1200 after the edit about 500 before the edit.) sysctl.c workqueue.c prio.h sched.h Kernel hz is at 300. I am using HPET.

I tried this kernel under Gentoo as well as other distros and other hardware and get the same result thats how I know that it is a kernel problem. The hardware I am running is a bit old. Intel p4 HT with 512mb 500GB sata I hdd .

Any help is really helpful!
_________________
^.^


Last edited by silent_Walker on Mon Sep 07, 2015 3:36 pm; edited 6 times in total
Back to top
View user's profile Send private message
Roman_Gruber
Advocate
Advocate


Joined: 03 Oct 2006
Posts: 3806
Location: Austro Bavaria

PostPosted: Fri Sep 04, 2015 6:55 pm    Post subject: Reply with quote

check those rtc /(real time clock) and watchdog things in the kernel and build those as modules.
Therefore you need to rebuild one more the kernel and boot up that new kernel.
Back to top
View user's profile Send private message
silent_Walker
Tux's lil' helper
Tux's lil' helper


Joined: 25 Jul 2012
Posts: 100
Location: forums.gentoo.org

PostPosted: Fri Sep 04, 2015 7:05 pm    Post subject: Reply with quote

tw04l124 wrote:
check those rtc /(real time clock) and watchdog things in the kernel and build those as modules.
Therefore you need to rebuild one more the kernel and boot up that new kernel.

Thank You, I will try that, I always build the kernel without watchdog.

EDIT: I updated git to reflect watchdog changes
_________________
^.^


Last edited by silent_Walker on Fri Sep 04, 2015 7:45 pm; edited 1 time in total
Back to top
View user's profile Send private message
Buffoon
Veteran
Veteran


Joined: 17 Jun 2015
Posts: 1074
Location: EU or US

PostPosted: Fri Sep 04, 2015 7:13 pm    Post subject: Reply with quote

Is there anything in dmesg that can shed light on this?
Back to top
View user's profile Send private message
silent_Walker
Tux's lil' helper
Tux's lil' helper


Joined: 25 Jul 2012
Posts: 100
Location: forums.gentoo.org

PostPosted: Fri Sep 04, 2015 7:42 pm    Post subject: Reply with quote

Buffoon wrote:
Is there anything in dmesg that can shed light on this?


The only thing that stands out to me is:

Code:
>>>WARNING<<< Wrong ufstype may corrupt your filesystem, default is ufstype=old
[  177.873750] hfs: can't find a HFS filesystem on dev sda2
[  346.660488] workqueue: max_active 8900 requested for writeback is out of range, clamping between 1 and 512
[  347.783922] tee (16172): drop_caches: 1
[  348.325122] tee (16200): drop_caches: 1


Would that be able to bring the system to a crawl? Or is this normal?
_________________
^.^
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Fri Sep 04, 2015 7:53 pm    Post subject: Reply with quote

@silent_Walker:

don't bother doing the diff

I'm uploading the branch & diff as of now


there only seem to be a picture file and the kernel configs you posted


so it's not a kernel source issue but most likely a kernel config

or more precisely a P4 processor related issue (could also be a regression)



Why 4.0.9 btw ?

Did you try out older or newer kernels (like 4.1.6 or even 4.2) ?


There has been some changes to older core 2 (and also p4 ?) processors recently which could be related


https://github.com/kernelOfTruth/linux/commits/b27e12262751b9d47e70fcd61b0cff8923c0ede0


related to the error message in dmesg:

http://www.mailbrowse.com/linux-kernel/990785.html [PATCH 28/35] workqueue: increase max_active of keventd and kill current_is_keventd()
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
silent_Walker
Tux's lil' helper
Tux's lil' helper


Joined: 25 Jul 2012
Posts: 100
Location: forums.gentoo.org

PostPosted: Fri Sep 04, 2015 8:19 pm    Post subject: Reply with quote

kernelOfTruth wrote:
@silent_Walker:




Why 4.0.9 btw ?

Did you try out older or newer kernels (like 4.1.6 or even 4.2) ?


I started to build this kernel a few weeks ago now and at that time that was the latest release.

Quote:
don't bother doing the diff

I'm uploading the branch & diff as of now


Your to fast man, you beat me to it.

Quote:




https://github.com/kernelOfTruth/linux/commits/b27e12262751b9d47e70fcd61b0cff8923c0ede0


related to the error message in dmesg:

http://www.mailbrowse.com/linux-kernel/990785.html [PATCH 28/35] workqueue: increase max_active of keventd and kill current_is_keventd()


So what was in dmesg was not from me changing values in cfq-iocshed, then. I will apply that patch and see what happens in terms of performance.
_________________
^.^
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Fri Sep 04, 2015 8:24 pm    Post subject: Reply with quote

no no no,

it's already included in the source (2010),

the changes you did to the scheduler and CFQ don't seem to be there (on github) ?


What exactly to do you want to get with those changes of the scheduler ?

better responsiveness - low latency ?


you know of BFQ ?

and BFS ?


if you're striving for low latency try those out:

http://algo.ing.unimo.it/people/paolo/disk_sched/sources.php

http://ck-hack.blogspot.co.at/
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Fri Sep 04, 2015 8:26 pm    Post subject: Re: Kernel not able to keep time and latency issues. Reply with quote

silent_Walker wrote:
I compiled a kernel, 4.0.9. The tree can be found at :

[snip]

I did edit cfq-iosched.c I also editied fair.c Kernel hz is at 300. I am using HPET.

I tried this kernel under Gentoo as well as other distros and other hardware and get the same result thats how I know that it is a kernel problem. The hardware I am running is a bit old. Intel p4 HT with 512mb 500GB sata I hdd .

Any help is really helpful!

Quote:
The machine I am mostly testing this on is a p4 HT 512 meg ram and sata I disk really old, but I also tested this on a AMD Turion and the results are the same.





that's really weird :?




I suggest trying with a new kernel source and then applying either CFQ or CFS


you could start anew with the kernel config of SysRescueCD or e.g. Pappy's Kernel Seeds (https://forums-web2.gentoo.org/viewtopic-t-942572-postdays-0-postorder-asc-start-0.html?sid=95423dcba19b11bc8ecae688c0575971)

then slim down the kernel (binary) from there


Unfortunately I don't have any i686 kernel config available right now

I tended to have them for an HP Microserver but it seems I deleted or lost them some time ago :?


anyway - I'm "off" for some time


hope this helps
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
silent_Walker
Tux's lil' helper
Tux's lil' helper


Joined: 25 Jul 2012
Posts: 100
Location: forums.gentoo.org

PostPosted: Fri Sep 04, 2015 8:37 pm    Post subject: Reply with quote

kernelOfTruth wrote:
no no no,

it's already included in the source (2010),



Oops, did not look at the year

Quote:


the changes you did to the scheduler and CFQ don't seem to be there (on github) ?


They are there look under linux-4.0.9/block/ and then look for cfq-iosched.c, I just edited values I did not add anything or remove anything.

Quote:

What exactly to do you want to get with those changes of the scheduler ?

better responsiveness - low latency ?


I just wanted to see if that I could increase throughput while decreasing latency a bit or decrease throughput and get the latency really low. I also did not want any delay, as soon as one task finishes I want the next one to start right then, not a little after.


Quote:

you know of BFQ ?

and BFS ?


I have heard of those, but where is the fun in using some one else's ideas.

Basically what I am doing is instead of having a sysctl.conf I am hard coding the values that I want the kernel to use.
_________________
^.^


Last edited by silent_Walker on Mon Sep 07, 2015 3:03 pm; edited 1 time in total
Back to top
View user's profile Send private message
silent_Walker
Tux's lil' helper
Tux's lil' helper


Joined: 25 Jul 2012
Posts: 100
Location: forums.gentoo.org

PostPosted: Fri Sep 04, 2015 11:33 pm    Post subject: Reply with quote

After leaving the machine on for 30 minutes with a webpage open I checked dmesg

and I get:

Code:

 1920.836649] CPU0: Core temperature/speed normal
[ 1920.836652] CPU1: Core temperature/speed normal
[ 1949.996079] mce: [Hardware Error]: Machine check events logged
[ 2266.291725] [sched_delayed] sched: RT throttling activated
[ 2359.055686] CPU0: Core temperature above threshold, cpu clock throttled (total events = 1413)
[ 2359.056145] CPU1: Core temperature/speed normal
[ 2359.056147] CPU0: Core temperature/speed normal
[ 2399.996085] mce: [Hardware Error]: Machine check events logged
[ 2845.148090] CPU1: Core temperature above threshold, cpu clock throttled (total events = 1428)
[ 2845.148098] CPU0: Core temperature above threshold, cpu clock throttled (total events = 1428)
[ 2845.148657] CPU1: Core temperature/speed normal
[ 2845.148659] CPU0: Core temperature/speed normal
[ 2849.996060] mce: [Hardware Error]: Machine check events logged
[ 2965.646424] psmouse serio1: Wheel Mouse at isa0060/serio1/input0 lost synchronization, throwing 1 bytes away.
[ 7192.197732] CPU1: Core temperature above threshold, cpu clock throttled (total events = 10229)
[ 7192.197740] CPU0: Core temperature above threshold, cpu clock throttled (total events = 10229)
[ 7192.198232] CPU1: Core temperature/speed normal
[ 7192.198234] CPU0: Core temperature/speed normal
[ 7505.684125] CPU1: Core temperature above threshold, cpu clock throttled (total events = 11741)
[ 7505.684135] CPU0: Core temperature above threshold, cpu clock throttled (total events = 11741)
[ 7505.684517] CPU1: Core temperature/speed normal
[ 7505.684518] CPU0: Core temperature/speed normal
[ 7810.376090] CPU1: Core temperature above threshold, cpu clock throttled (total events = 12469)
[ 7810.376095] CPU0: Core temperature above threshold, cpu clock throttled (total events = 12469)
[ 7810.376672] CPU1: Core temperature/speed normal
[ 7810.376671] CPU0: Core temperature/speed normal
[ 8113.106502] CPU1: Core temperature above threshold, cpu clock throttled (total events = 16218)
[ 8113.106515] CPU0: Core temperature above threshold, cpu clock throttled (total events = 16218)
[ 8113.107078] CPU0: Core temperature/speed normal
[ 8113.107080] CPU1: Core temperature/speed normal
[ 8413.692102] CPU0: Core temperature above threshold, cpu clock throttled (total events = 20604)
[ 8413.692114] CPU1: Core temperature above threshold, cpu clock throttled (total events = 20604)
[ 8413.692561] CPU1: Core temperature/speed normal
[ 8413.692561] CPU0: Core temperature/speed normal
[ 8910.520094] CPU0: Core temperature above threshold, cpu clock throttled (total events = 24043)
[ 8910.520104] CPU1: Core temperature above threshold, cpu clock throttled (total events = 24043)
[ 8910.520637] CPU1: Core temperature/speed normal
[ 8910.520639] CPU0: Core temperature/speed normal
[ 9452.168097] CPU0: Core temperature above threshold, cpu clock throttled (total events = 27060)
[ 9452.168107] CPU1: Core temperature above threshold, cpu clock throttled (total events = 27060)
[ 9452.168674] CPU1: Core temperature/speed normal
[ 9452.168677] CPU0: Core temperature/speed normal
[ 9752.389374] CPU1: Core temperature above threshold, cpu clock throttled (total events = 28735)
[ 9752.389381] CPU0: Core temperature above threshold, cpu clock throttled (total events = 28735)
[ 9752.389954] CPU0: Core temperature/speed normal
[ 9752.389956] CPU1: Core temperature/speed normal
[10065.584118] CPU0: Core temperature above threshold, cpu clock throttled (total events = 32286)
[10065.584126] CPU1: Core temperature above threshold, cpu clock throttled (total events = 32286)
[10065.584617] CPU0: Core temperature/speed normal
[10065.584619] CPU1: Core temperature/speed normal
[10366.447957] CPU1: Core temperature above threshold, cpu clock throttled (total events = 36493)
[10366.447964] CPU0: Core temperature above threshold, cpu clock throttled (total events = 36493)
[10366.448454] CPU1: Core temperature/speed normal
[10366.448454] CPU0: Core temperature/speed normal
[10668.685251] CPU1: Core temperature above threshold, cpu clock throttled (total events = 37678)
[10668.685258] CPU0: Core temperature above threshold, cpu clock throttled (total events = 37678)
[10668.685715] CPU0: Core temperature/speed normal
[10668.685717] CPU1: Core temperature/speed normal
[11035.508090] CPU1: Core temperature above threshold, cpu clock throttled (total events = 39158)
[11035.508098] CPU0: Core temperature above threshold, cpu clock throttled (total events = 39158)
[11035.508555] CPU0: Core temperature/speed normal
[11035.508557] CPU1: Core temperature/speed normal
[11341.756081] CPU1: Core temperature above threshold, cpu clock throttled (total events = 39257)
[11341.756088] CPU0: Core temperature above threshold, cpu clock throttled (total events = 39257)


But this can not be the case I am putting my finger on the cooler and it is cold, I do not burn my finger at all. So how would I turn off cpu throtling even if the cpu gets "too hot" which it is not because my finger is on the cooler and did not get burned. Why is my computer lying to me?
_________________
^.^


Last edited by silent_Walker on Wed Sep 09, 2015 3:10 pm; edited 1 time in total
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sat Sep 05, 2015 12:04 am    Post subject: Reply with quote

Quote:
mce: [Hardware Error]: Machine check events logged


well, there's something happening


install mcelog

Code:
emerge mcelog



after that see what the program says about that hardware error


also what do lm_sensors say ?

Code:
emerge lm_sensors



configure it with sensors-detect


then fire it up

Code:
/etc/init.d/lm_sensors start



Code:
sensors

_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7197

PostPosted: Sat Sep 05, 2015 2:03 pm    Post subject: Reply with quote

kernelOfTruth wrote:
Quote:
mce: [Hardware Error]: Machine check events logged


well, there's something happening

Overheat are report also thru mce ; so that should be that. He should really look at temperature trouble.
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sat Sep 05, 2015 11:00 pm    Post subject: Reply with quote

@krinn:

roger that - forgot about that connection,

thanks :)


@silent_Walker:

first make sure you have a stable base system (or kernel base) then you can tweak to your hearts liking

(also make sure to have a backup of that stable system then)
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
silent_Walker
Tux's lil' helper
Tux's lil' helper


Joined: 25 Jul 2012
Posts: 100
Location: forums.gentoo.org

PostPosted: Mon Sep 07, 2015 2:58 pm    Post subject: Reply with quote

krinn wrote:
kernelOfTruth wrote:
Quote:
mce: [Hardware Error]: Machine check events logged


well, there's something happening

Overheat are report also thru mce ; so that should be that. He should really look at temperature trouble.


My thermal paste was dry ( that was the stock paste from 2004), so most likely that is why the cooler was not getting warm since the heat was not transferring well.

However I still get horrible latency. So I do not think that it is a heat issue. I built the kernel again with 1000 hz and right after boot into X the clock did not work and task switching and latency were poor. I heard of the divider command via grub and set it for 25, so I was running on 40hz, it did boot, but because of the low hz it was way slower then 1000 with the horrible latency. I tried setting it to divider=10 for 100hz but every time the computer tried to mount the HDD it would freeze and the scroll and caps lock will just flash. I will try to build the kernel without pre-emption and see if that would fix it.

If not I will rebuild the kernel but only add one of my modified files, till I get the bad performance then I know which file caused the bad latency.

I also built the kernel with debugging and everything came back with an OK or PASSED . Maybe the hardware I am testing on is just to old to handle the edits I made.
_________________
^.^


Last edited by silent_Walker on Mon Sep 07, 2015 3:17 pm; edited 4 times in total
Back to top
View user's profile Send private message
silent_Walker
Tux's lil' helper
Tux's lil' helper


Joined: 25 Jul 2012
Posts: 100
Location: forums.gentoo.org

PostPosted: Mon Sep 07, 2015 3:05 pm    Post subject: Reply with quote

kernelOfTruth wrote:

you know of BFQ ?

and BFS ?


if you're striving for low latency try those out:

http://algo.ing.unimo.it/people/paolo/disk_sched/sources.php

http://ck-hack.blogspot.co.at/


Do you know of any other schedulers that are out there that work with the mainline kernel? Besides deadline BFS CFS.

How well do BFS CFQ work together.
_________________
^.^
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Mon Sep 07, 2015 4:22 pm    Post subject: Reply with quote

silent_Walker wrote:


Do you know of any other schedulers that are out there that work with the mainline kernel? Besides deadline BFS CFS.



I can't consciously list them,

in the past years I've come across a few which were in the proving stage - meaning:

they work for one kernel releases and are a technical previews (mainly developed to achieve a thesis, master degree - at universities)

but aren't maintained.

If you're up to it do some googling and see what comes up, then go from there

silent_Walker wrote:


How well do BFS CFQ work together.


they're fine,

you don't necessarily need BFQ but the latency is somewhat lower than CFQ with high i/o

also with BFS you have SMTNICE which works with SMT capable processors,

CFS doesn't really consider it (well)




Consider using


Code:
schedtool
chrt


tools to raise priority of tasks and/or IRQs - this can cut down latency,

also use a preemptive kernel instead of voluntary preemption or non preemptive,


there's also the RCU settings (boosting) that you can tinker with


also boot up the kernel with appending


threadirqs


that - simply said - the IRQs, etc. get "split up" to have a finer granular tuning working surface (more knobs to tweak)
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
Buffoon
Veteran
Veteran


Joined: 17 Jun 2015
Posts: 1074
Location: EU or US

PostPosted: Mon Sep 07, 2015 4:42 pm    Post subject: Reply with quote

Gentoo is a Linux you build for yourself.
I would test this hardware with something that is built by somebody else, just to see if it is your Gentoo that can't handle the hardware or the hardware itself is expiring on you.

There are times when CMOS reset can help to get rid of weird issues, my 2¢.
Back to top
View user's profile Send private message
silent_Walker
Tux's lil' helper
Tux's lil' helper


Joined: 25 Jul 2012
Posts: 100
Location: forums.gentoo.org

PostPosted: Wed Sep 09, 2015 2:09 pm    Post subject: Reply with quote

Buffoon wrote:
Gentoo is a Linux you build for yourself.
I would test this hardware with something that is built by somebody else, just to see if it is your Gentoo that can't handle the hardware or the hardware itself is expiring on you.

There are times when CMOS reset can help to get rid of weird issues, my 2¢.

Thanks.

I took the CMOS battery out and left it out for 30 minutes but still get the same problems. I will try maybe slitaz, as it is supposed to be great on older hardware.

At first I thought I knew what the problem was, I was looking thru the files that I edited and saw that I changed ( under sysctl.c)

Code:
 #ifdef CONFIG_PRINTK
 static int ten_thousand = 10000;


to

Code:
 #ifdef CONFIG_PRINTK
static int ten_thousand = 1000;


But apparently the value ranges from 0 to 10,000 and PRINTK does something else then what I thought it did. Before I did my research on PRINTK I thought that 10,000 was the perfect value for latency and throughput. I was going to put it back to 10k and seen if that would fix it ( the problem) and then go higher for better latency at the risk of lower throughput. Unfortunately what I thought what would happen wont happen.

I was looking at this webpage describing CFS, so I think that now may values may be out of sync. under fair.c and sysctl.c

my sysctl.c has these values:
Code:

static int min_sched_granularity_ns = 10000;      /* 10 usecs */
static int max_sched_granularity_ns = 80000;   /* 80 usecond */
static int min_wakeup_granularity_ns;         /* 0 usecs */
static int max_wakeup_granularity_ns = 25000;   /* 25 usecond */


from the default of:
Code:

263 static int min_sched_granularity_ns = 100000;           /* 100 usecs */
264 static int max_sched_granularity_ns = NSEC_PER_SEC;     /* 1 second */
265 static int min_wakeup_granularity_ns;                   /* 0 usecs */
266 static int max_wakeup_granularity_ns = NSEC_PER_SEC;    /* 1 second */


and my fair.c has:

Code:

unsigned int sysctl_sched_latency =200000ULL; ( do I need the space after the "=" sign? )
unsigned int normalized_sysctl_sched_latency = 75500000ULL;  ( woah,  I am an idiot, I set this number way to high.  I put it to 75.5 seconds.  Following "stock" this should be the same as sysctl_sched_latency. so 200000 or .2 seconds)
unsigned int sysctl_sched_min_granularity = 150000ULL;
unsigned int normalized_sysctl_sched_min_granularity = 200000ULL;
static unsigned int sched_nr_latency = 20;
unsigned int sysctl_sched_wakeup_granularity = 0UL;
unsigned int normalized_sysctl_sched_wakeup_granularity = 80000UL;
const_debug unsigned int sysctl_sched_migration_cost = 695UL;  ( 0.000695 seconds or .695 ms )
nsigned int sysctl_sched_cfs_bandwidth_slice = 166000UL


from the default of :
Code:

unsigned int sysctl_sched_latency = 6000000ULL;
unsigned int normalized_sysctl_sched_latency = 6000000ULL;
unsigned int sysctl_sched_min_granularity = 750000ULL;
unsigned int normalized_sysctl_sched_min_granularity = 750000ULL;
static unsigned int sched_nr_latency = 8;
unsigned int sysctl_sched_wakeup_granularity = 1000000UL;
unsigned int normalized_sysctl_sched_wakeup_granularity = 1000000UL;
const_debug unsigned int sysctl_sched_migration_cost = 500000UL; ( 0.5 seconds)
unsigned int sysctl_sched_cfs_bandwidth_slice = 5000UL;


So I am thinking that maybe the time-slice is off and that tasks are not being allowed to run.

I am probably wrong but the sysctl_sched_min_granularity decides the minimum amount of time it runs before being pre-empted I changed the value from 75 milliseconds to 15, so things are getting pre-empted faster. So if they are getting pre-empted faster that means the cpu is taking on more things at once and can not cope and is killing off some tasks like the clock because it has lower priority. I have no idea.

But when running at 1000hz more things go thru the CFS right? And things get pre-empted even faster thats why right after boot on 1000hz the system crawls, while on 300hz it takes time to reach that point where latency "drops dead".

Throughput does increase tho opening thunar when latency goes down time reads: Maybe this is from normalized_sysctl_sched_latency = 75500000ULL?

Code:

real   0m1.255s
user   0m0.050s
sys   0m0.008s

And here is the output from when latency is good

real   0m2.025s
user   0m0.414s
sys   0m0.054s


So it opens 1.2 seconds faster. So for a headless machine this would be awesome, not so much for a desktop tho.

EDIT: Found an old email from 2007 http://lkml.iu.edu/hypermail/linux/kernel/0711.0/1207.html

Quote:
kernel_sched_nr_latency should always be set to kernel_sched_latency/kernel_sched_min_granularity. (it's not a free tunable)

mine is not. It should be set from 20 to 13.3333333, I will just put that to 13. ( or should I round up?)

EDIT2: Found something from Fedora, maybe this applies to me as well.

Quote:
Description of problem:
Big virtualization hosts can experience high contention at the run queue lock, please see bug 960816 for more details.

Setting kernel.sched_migration_cost tunable to 5000000 (10X its default value) eliminates the contention.


Mine is waayyy below .5 seconds so increase the number to 5000000 ( 5 seconds ) may help or at least restoring it to the stock of .5s

I also read somewhere else that

Code:
/proc/sys/kernel/sched_autogroup_enabled


should be set to 0 for performance I have mine on 1. I do not think that that is causing the problem but it may be a way to get around the problem. I will recompile the kernel with
Code:

static unsigned int sched_nr_latency = 20;

to

static unsigned int sched_nr_latency = 13;

and

normalized_sysctl_sched_latency = 75500000 (75 seconds)

to

normalized_sysctl_sched_latency = 200000 (0.2 seconds)


Then if that does not work I will increase the value of

Code:
 
sysctl_sched_migration_cost = 695 (0.000695 seconds)


to stock of
Code:


sysctl_sched_migration_cost = 500000 (5ms)

_________________
^.^
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Wed Sep 09, 2015 3:48 pm    Post subject: Reply with quote

Keep in mind that


lower latency could equal lower performance & throughput



Take a look at the following settings:


http://www.overclock.net/t/1519964/experiment-overclocking-the-linux-os-through-the-sysclt-proc-sys-sys-block-and-other-settings#post_23094293
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
silent_Walker
Tux's lil' helper
Tux's lil' helper


Joined: 25 Jul 2012
Posts: 100
Location: forums.gentoo.org

PostPosted: Wed Sep 09, 2015 4:01 pm    Post subject: Reply with quote

kernelOfTruth wrote:
Keep in mind that


lower latency could equal lower performance & throughput


So having normalized_sysctl_sched_latency = 75500000ULL or 75 seconds is why throughput is high. Because higher latency could equal higher performance and throughput.

Quote:

#Given value and default was 950000
#At it's default of 950000 RTs can take 95% of CPU usage/sec
#Setting this to -1 disables RT bandwidth enforcement and isn't recommended unless you're sure you'll avoid blocks/locks
#999999 worked best so far
echo -1 | sudo tee /proc/sys/kernel/sched_rt_runtime_us


Is disabling this really a safe option.
_________________
^.^
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Wed Sep 09, 2015 4:32 pm    Post subject: Reply with quote

silent_Walker wrote:

Quote:

#Given value and default was 950000
#At it's default of 950000 RTs can take 95% of CPU usage/sec
#Setting this to -1 disables RT bandwidth enforcement and isn't recommended unless you're sure you'll avoid blocks/locks
#999999 worked best so far
echo -1 | sudo tee /proc/sys/kernel/sched_rt_runtime_us


Is disabling this really a safe option.


I'd keep it around 95 - 97% since there were some lockups with having it disabled
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6024

PostPosted: Mon Sep 14, 2015 8:08 pm    Post subject: Reply with quote

Have you benchmarked any of these changes you've made using tools like latencytop and perf?

I'd love to know what kind of a performance boost you're seeing with edits like these.
Back to top
View user's profile Send private message
silent_Walker
Tux's lil' helper
Tux's lil' helper


Joined: 25 Jul 2012
Posts: 100
Location: forums.gentoo.org

PostPosted: Tue Sep 15, 2015 2:39 pm    Post subject: Reply with quote

Ant P. wrote:
Have you benchmarked any of these changes you've made using tools like latencytop and perf?

I'd love to know what kind of a performance boost you're seeing with edits like these.


I have not been able to run latency top and perf yet. All of my kernels on my p4 get to a kernel panic, about 2 minutes into boot. And my laptop charger is barley working ( thats why I am on this ancient p4)

Code:
 Kernel panic - not syncing: CPU context corruption


Unfortunately thats all it says, and the computer locks up.

But as far as my edits to radeon. They have been some what good. With my p4 machine I have an old RV280SE, that was not able to play 1080p fluidly, I was only getting anywhere from 6-9 ( 9 for a non action scene and 6 for an action scene) fps with about 1 fps drop every 5 seconds just about, and with the new 4k resolution, videos would play at 1 fps. When I edited, the radeon.h

I could get 1080p videos to play at about 25 fps but get about 3 dropped frames a second on a non action scene, like some one walking around or talking to another person, but with an action scene I would get 11 fps with about 10 dropped frames a second, so it would be laggy. I tried to find a reason for this and I edited the radeon clocks file, this does not overclock the card as far as I know. But dropped fps went down about 2%.

Also when I did not edit the radeon clocks file, and ran glx gears, the fps would start at 1200 ( up from the 600 I usually get) but it would keep climbing, if I wanted to I could of gotten a million fps, if I left it on for an hour, after editing the clocks file, the result is more constantly around 1200 fps. That is with no other windows open, and I am running Openbox not something with a lot of eye candy to take away from glxgears.

I am about to edit the radeon.h one more time, because I want to see if I can lower the dropped fps more by increasing the timeout. If you look the stock is 100 ms timeout I changed that to 3ms timeout. So I will change it to 60ms timeout and will run glx gears and youtube stats for nerds to see whether or not fps is increased or decreased by the timeout and also see if dropped fps goes down.

This is with flash and not html5 as the default viewer for youtube as my hardware does not fully support features of html5.

Without and with the edits I can not play full screen videos via youtube at all, its all lag. This is why I have not run any games on it yet, as I think that the game would fail horribly performance wise on this machine, although once I can get around the kernel panic I will try nexuiz in a window mode and in full screen and report the fps count.

Also if you are wondering about the "y" that is in there at line 5 on radeon_clocks.c I added that for no reason.

Of course some may say these results are not 100% true, because glxgears is not a true benchmark tool but that is what I had available at that time. The same for youtube's stats for nerds.
_________________
^.^


Last edited by silent_Walker on Tue Sep 15, 2015 6:01 pm; edited 3 times in total
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Tue Sep 15, 2015 3:08 pm    Post subject: Reply with quote

Ant P. wrote:
Have you benchmarked any of these changes you've made using tools like latencytop and perf?

I'd love to know what kind of a performance boost you're seeing with edits like these.


Ant P., thanks for the hint,


I'm out :o
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum