Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Linux Performance Tuning Questions
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
njcwotx
Guru
Guru


Joined: 25 Feb 2005
Posts: 564
Location: Texas

PostPosted: Fri Apr 11, 2008 4:15 pm    Post subject: Linux Performance Tuning Questions Reply with quote

Greets everyone! I have some linux performance tuning questions. This is not Gentoo, but a Red Hat EL5 system, but I actually value the posters opinions here more. So, I posted in unsupported software to generate a little discussion on some topics that have come up recently with some of the output with top, vmstat, iostat and mpstat.

The server runs a front end application and database progress/QAD is the software. 8CPU's and 6GB Ram on an HP DL580 G3.

For some time my dba's and app guys have been saying users are reporting slowness and immediately ask for more RAM and CPU's without any evidence. I look at our stats and dont see the need. Here is my output. I want to know if there is anything wrong here or if I am missing something. I have particular interest in the wa (waiting on I/O) value as that looks of interest to me. Often I can see values but wonder what is considered out of normal parameters.

some specific starter questions:
1. How much disk cache is really useful/needed? I have seen anywhere from 2.5Gb-5Gb Disk Cache. As I have read and researched, free RAM in Linux is always low since Linux does not want to have unused wasted RAM and so turns it into disk cache. If this is so, does 2.5Gb of Disk Cache lead me to believe this is not a RAM issue?

2. How high of a iowait time is too high? Sometimes I see single digits, others I see double digits in "wa" using the vmstat command or "%iowait" using mpstat. I am wondering if my problem may be one of Disk I/O load than RAM.

3. What is load average mean in Top?




Code:

Last login: Fri Apr 11 10:25:17 2008 from
[user@host]# free -m
             total       used       free     shared    buffers     cached
Mem:          6085       5878        207          0         15       2476
-/+ buffers/cache:       3386       2699
Swap:         4094         68       4026
[user@host]# vmstat 5 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  2  69844 215236  19424 2520872    0    0    33     5    2    0  6  1 89  4  0
 1  2  69844 214148  19456 2521656    0    0  1618   270 1623 2716  3  1 83 13  0
 0  2  69844 213680  19560 2522592    0    0  1632   143 1576 2714  3  1 84 13  0
 0  2  69844 212920  19700 2516416    0    0  1733   330 1609 2771  5  1 77 16  0
 1  1  69844 208748  19652 2520528    0    0  2140   212 1634 2850  5  1 80 14  0
 1  4  69844 212228  19780 2517104    0    0  3324   513 1683 2933 10  1 73 16  0
 2  2  69844 214088  20084 2514616    0    0  2365   667 1810 3258 10  1 69 20  0
 1  2  69844 211140  20220 2515396    0    0  3235   900 2121 3447 17  2 66 15  0
 0  4  69844 212988  20272 2514256    0    0  2077   694 1707 2990 13  1 72 13  0
 1  4  69844 212388  20404 2513248    0    0  2892   383 1742 3128 11  1 73 14  0
[user@host]# iostat
Linux 2.6.18-53.1.6.el5PAE    04/11/2008

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.27    0.00    1.26    3.58    0.00   88.89

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
cciss/c0d0       16.94       292.85       334.80  856805770  979511400
cciss/c1d0       18.21        46.84       317.35  137041700  928475648
cciss/c1d1        8.19      1216.78       574.54 3559950266 1680935800
cciss/c1d2       95.37       441.11       321.96 1290544980  941966352

[user@host]# mpstat
Linux 2.6.18-53.1.6.el5PAE    04/11/2008

11:05:52 AM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
11:05:52 AM  all    6.27    0.00    1.20    3.58    0.01    0.06    0.00   88.89   1262.28
[user@host]# mpstat 5 10
Linux 2.6.18-53.1.6.el5PAE   04/11/2008

11:05:58 AM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
11:06:03 AM  all    5.35    0.00    0.47    1.47    0.02    0.05    0.00   92.63   1262.73
11:06:08 AM  all    4.22    0.00    0.87    3.70    0.00    0.07    0.00   91.13   1331.14
11:06:13 AM  all    1.25    0.00    0.47    1.88    0.03    0.03    0.00   96.35   1243.09
11:06:18 AM  all    2.35    0.00    0.42    3.57    0.02    0.02    0.00   93.60   1292.23
11:06:23 AM  all    1.42    0.00    0.47    1.85    0.00    0.00    0.00   96.25   1184.37
11:06:28 AM  all    7.05    0.00    0.77    5.57    0.02    0.07    0.00   86.50   1316.83
11:06:33 AM  all    1.75    0.00    0.52    2.80    0.02    0.05    0.00   94.85   1307.36
11:06:38 AM  all    3.52    0.00    0.55    3.42    0.00    0.05    0.00   92.45   1434.07
11:06:43 AM  all    2.90    0.00    0.82    1.65    0.02    0.05    0.00   94.55   1292.42
11:06:48 AM  all    3.88    0.00    0.70    3.30    0.03    0.03    0.00   92.08   1289.98
Average:     all    3.37    0.00    0.61    2.92    0.02    0.04    0.00   93.04   1295.44
[user@host]# top
top - 11:08:06 up 33 days, 20:48, 282 users,  load average: 1.60, 2.41, 2.73
Tasks: 1493 total,   1 running, 1491 sleeping,   0 stopped,   1 zombie
Cpu(s): 11.6%us,  2.2%sy,  0.0%ni, 64.5%id, 21.5%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   6231760k total,  6021848k used,   209912k free,    24136k buffers
Swap:  4192924k total,    69844k used,  4123080k free,  2511324k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
20886 jmp       17   0  753m  85m  79m D   30  1.4   0:02.90 _progres
10745 100pac    16   0  562m 121m 110m D   15  2.0   0:38.40 _progres
10855 root      15   0 14884 8024 2220 S   14  0.1  93:43.48 cupsd
29984 root      16   0  3256 1848  792 R    8  0.0   0:00.69 top
26854 400new    17   0  563m 340m 329m D    8  5.6   0:18.76 _progres
21065 510map    16   0  558m  53m  47m D    7  0.9   0:01.65 _progres
16607 lkf       16   0  759m 310m 298m S    6  5.1   0:34.10 _progres
10297 root      16   0  1820  616  520 S    4  0.0 435:35.70 pam_timestamp_c
 3328 410cjs    16   0  562m 276m 266m S    3  4.5   0:12.39 _progres
15710 cmb       15   0  755m  38m  30m S    2  0.6   0:01.86 _progres
28529 2345ash   16   0  250m  96m  90m S    2  1.6   2:42.67 _progres
13023 sls       16   0  759m 566m 554m S    2  9.3   2:17.40 _progres
13481 250rls    16   0  563m 309m 297m S    2  5.1   0:19.67 _progres
23281 500smm    16   0  561m  76m  66m S    2  1.3   0:06.50 _progres
 7544 500bcv    16   0  562m  74m  63m S    1  1.2   0:09.88 _progres
27477 root      22   0  236m  16m 7784 S    1  0.3   3:42.11 java
29148 460tep    16   0  562m  51m  40m S    1  0.8   0:05.34 _progres
29996 100pac    16   0  6592 1792 1232 S    1  0.0   0:00.03 lp
24166 450smg    16   0  563m 276m 264m S    1  4.5   0:27.09 _progres
  312 root      10  -5     0    0    0 S    0  0.0  55:37.26 kswapd0
 1801 root      15   0  439m  80m  80m S    0  1.3   0:00.16 _mprshut
10710 100pac    15   0  9612 1472  960 S    0  0.0   0:02.93 sshd
12987 sls       16   0  9608 1460  956 S    0  0.0   0:06.58 sshd
15647 lmp       16   0  754m 210m 202m S    0  3.5   0:20.89 _progres
15668 cmb       15   0  9608 1452  956 S    0  0.0   0:00.10 sshd
16572 lkf       15   0  9612 1464  956 S    0  0.0   0:02.25 sshd
17854 ibarash   15   0  9608 1452  956 S    0  0.0   0:01.08 sshd
20314 tem       15   0  9608 1452  956 S    0  0.0   0:02.32 sshd
20349 tem       16   0  758m 558m 547m S    0  9.2   1:13.48 _progres
    1 root      15   0  2040  644  556 S    0  0.0   0:07.32 init
    2 root      RT   0     0    0    0 S    0  0.0   0:24.29 migration/0
    3 root      34  19     0    0    0 S    0  0.0   0:00.24 ksoftirqd/0
    4 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/0
    5 root      RT   0     0    0    0 S    0  0.0   0:01.88 migration/1
    6 root      34  19     0    0    0 S    0  0.0   8:10.20 ksoftirqd/1
    7 root      RT   0     0    0    0 S    0  0.0   0:00.02 watchdog/1
    8 root      RT   0     0    0    0 S    0  0.0   0:02.71 migration/2


_________________
Drinking from the fountain of knowldege.
Sometimes sipping.
Sometimes gulping.
Always thirsting.
Back to top
View user's profile Send private message
micmac
l33t
l33t


Joined: 28 Nov 2003
Posts: 992

PostPosted: Fri Apr 11, 2008 4:41 pm    Post subject: Reply with quote

Which IO scheduler is used (if your kernel supports that)? Maybe try deadline if available, it's supposed to do a decent jobs when using databases. Also, is the system 32 bit or 64 bit? 64 bit should make a difference with DBs and a big box like yours.
Back to top
View user's profile Send private message
njcwotx
Guru
Guru


Joined: 25 Feb 2005
Posts: 564
Location: Texas

PostPosted: Fri Apr 11, 2008 4:45 pm    Post subject: Reply with quote

its a 32bit system. I am not sure what you mean by which ioscheduler, how can I tell and what options are there?

I also have another new question...when I run top, I have some users max cpu at 100% but the Total CPU is 10% or something. Is there a way to divide out individual cpu usage values? Maybe the threads are not getting balanced across CPU's well enough, perhaps this is where the ioscheduler comes into play.
_________________
Drinking from the fountain of knowldege.
Sometimes sipping.
Sometimes gulping.
Always thirsting.
Back to top
View user's profile Send private message
njcwotx
Guru
Guru


Joined: 25 Feb 2005
Posts: 564
Location: Texas

PostPosted: Fri Apr 11, 2008 5:20 pm    Post subject: Reply with quote

Ok, your question got me to look this up and I learned something new. Thanks!

I am using the cfq scheduler, but deadline is an option, in fact noop and anticipatory are too. I will keep this out as an option to try if we get to tweaking.
_________________
Drinking from the fountain of knowldege.
Sometimes sipping.
Sometimes gulping.
Always thirsting.
Back to top
View user's profile Send private message
Neo2
Apprentice
Apprentice


Joined: 25 Sep 2006
Posts: 224
Location: Italy

PostPosted: Fri Apr 11, 2008 5:44 pm    Post subject: Reply with quote

I don't know how fedora kernel packages work, anyway, seems like your problem is related to non-optimized kernel. Usually binary distros build up generic kernels that include of all the possible hardware that the user might need, hence big kernel images and potentially slow execution (not to mention possible hardware conflicts). If I were you I'd dig for some easy way to compile your own kernel on that system.
You would then manually tune things like the disk scheduler, the specific CPU model your kernel is built for, which memory allocator to use and which drivers you really need to include for your system to work properly. You could install the kernel sources with your package manager and then use the gentoo linux kernel guide as a reference in choosing the right options. This will result in a lighter, faster and tuned kernel.
That's just a suggestion. It should provide the performance boosts you're asking for :)
_________________
Neo2
Unofficial minimal liveCD for x86/amd64 w/reiser4+truecrypt
Back to top
View user's profile Send private message
micmac
l33t
l33t


Joined: 28 Nov 2003
Posts: 992

PostPosted: Fri Apr 11, 2008 8:13 pm    Post subject: Reply with quote

You can change the ioscheduler on the fly, no need to reboot to test that. "deadline" should be good for what your box is doing.
Back to top
View user's profile Send private message
njcwotx
Guru
Guru


Joined: 25 Feb 2005
Posts: 564
Location: Texas

PostPosted: Fri Apr 11, 2008 10:37 pm    Post subject: Reply with quote

Thanks.

Any ideas on how much disk cache is optimal? Does it do any good to get mega gigs of disk cache or is it a simple way to lower your cost/benefit ratio?
_________________
Drinking from the fountain of knowldege.
Sometimes sipping.
Sometimes gulping.
Always thirsting.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43009
Location: 56N 3W

PostPosted: Fri Apr 11, 2008 11:02 pm    Post subject: Reply with quote

njcwotx,

An 8 way 64 bit capable box with 6GB of RAM and a 32 bit bit install.

You can only address 4Gb of RAM directly with a 32 bit install, the rest is paged like expanded (not extended) RAM in days gone by.
This uses the PAE to do the best it can. That may explain the large amount of free RAM and 70MB used in swap.
You are running out of space with the per process 3G/1G memory split.

I suspect you need to move to a 64 bit OS so you can get back to a linear address space. This won't help if your application is only available in 32 bit form.
You would get to use the top half of all your cores too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
SeaTiger
l33t
l33t


Joined: 22 Nov 2007
Posts: 603
Location: Toronto, Ontario, Canada

PostPosted: Sun Apr 13, 2008 3:21 pm    Post subject: Reply with quote

njcwotx:

Is the "free", "top", "iostat" and "mpstat" taken during the time user complain or after?

It will be more useful to have a "snapshot" when user feel the "slowness".

I am seeing 282 users, are they all (or 'many') using _progress ?? If yes, maybe memory is an issue. My guess is the 70M swap was created during the peek load (and the swap maybe larger than 70M during that time), which is long before you run "top".

Regarding swap memory, according to my experience, whenever the machine start to use swap, it is going to be slow. Basically, you want to minimize swap usage.

Can you check if your kernel support PAE?
Try the following:
Code:
cd /boot
grep PAE config
grep 64 config
Back to top
View user's profile Send private message
njcwotx
Guru
Guru


Joined: 25 Feb 2005
Posts: 564
Location: Texas

PostPosted: Mon Apr 14, 2008 2:30 pm    Post subject: Reply with quote

The stats above are done from a few days later, but I did check during the 'slowness'. At the time I saw no paging, and that 70M of swap was there all week. Right now its 17M (there was a reboot saturday) , I think when your swap file is low and not regularly accessed a small amount of swap should not be an issue. Im fairly certain we have no paging at this time.

However, I have been really watching the cached value. It fluctuates quite a bit, I have seen it go from almost 5Gb of RAM to 2.5Gb of ram over the course of a day, then the next morning after everyone logged out and went home the numbers go back up. It sounds reasonable. As far as the slowness, I dont think its an all-day-long affair. I think its intermittent and possibly tied to when someone does a large report. I am waiting for a re-occurance to re-snap the values. I am predicting some kind of disk I/O bottleneck and will be looking at the 'waiting on I/O' values and iostats as well as the available cache. I am wondering if my cached RAM is fairly low at the time it gets slow.


Code:
# Linux kernel version: 2.6.18-53.1.6.el5PAE
CONFIG_X86_PAE=y
CONFIG_X86_CMPXCHG64=y
CONFIG_HIGHMEM64G=y
CONFIG_RESOURCES_64BIT=y
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_BLK_DEV_CMD640=y
CONFIG_BLK_DEV_CMD640_ENHANCED=y
CONFIG_BLK_DEV_CMD64X=y
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
CONFIG_I2O_EXT_ADAPTEC_DMA64=y
# CONFIG_R3964 is not set
CONFIG_AGP_AMD64=y
CONFIG_VIDEO_UPD64031A=m
CONFIG_VIDEO_UPD64083=m
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
# CONFIG_SND_MTS64 is not set
CONFIG_NLS_CODEPAGE_864=m

_________________
Drinking from the fountain of knowldege.
Sometimes sipping.
Sometimes gulping.
Always thirsting.
Back to top
View user's profile Send private message
SeaTiger
l33t
l33t


Joined: 22 Nov 2007
Posts: 603
Location: Toronto, Ontario, Canada

PostPosted: Tue Apr 15, 2008 3:10 am    Post subject: Reply with quote

njcwotx wrote:
The stats above are done from a few days later, but I did check during the 'slowness'. At the time I saw no paging, and that 70M of swap was there all week. Right now its 17M (there was a reboot saturday) , I think when your swap file is low and not regularly accessed a small amount of swap should not be an issue. Im fairly certain we have no paging at this time.

I agree with that, linux swap seems to be very 'lazy'.

njcwotx wrote:
However, I have been really watching the cached value. It fluctuates quite a bit, I have seen it go from almost 5Gb of RAM to 2.5Gb of ram over the course of a day, then the next morning after everyone logged out and went home the numbers go back up. It sounds reasonable. As far as the slowness, I dont think its an all-day-long affair. I think its intermittent and possibly tied to when someone does a large report. I am waiting for a re-occurance to re-snap the values. I am predicting some kind of disk I/O bottleneck and will be looking at the 'waiting on I/O' values and iostats as well as the available cache. I am wondering if my cached RAM is fairly low at the time it gets slow.

When doing "free", you will want to look that "-/+ buffers/cache: " under the "used" column, if "Mem" + "cache" under used is approaching your memory limit like 5G, then it is approaching limit.

Besides "I/O wait %", You may want to check the system load with "top" or "uptime".

Code:
# Linux kernel version: 2.6.18-53.1.6.el5PAE
CONFIG_X86_PAE=y
CONFIG_X86_CMPXCHG64=y
CONFIG_HIGHMEM64G=y
CONFIG_RESOURCES_64BIT=y
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_BLK_DEV_CMD640=y
CONFIG_BLK_DEV_CMD640_ENHANCED=y
CONFIG_BLK_DEV_CMD64X=y
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
CONFIG_I2O_EXT_ADAPTEC_DMA64=y
# CONFIG_R3964 is not set
CONFIG_AGP_AMD64=y
CONFIG_VIDEO_UPD64031A=m
CONFIG_VIDEO_UPD64083=m
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
# CONFIG_SND_MTS64 is not set
CONFIG_NLS_CODEPAGE_864=m

You kernel has PAE support means it can address all your memory and allocate them to process. However each process will still be limit by 32bit address space. If any of the process virtual size approach 2.5G, it may be problem.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum