Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Keep running out of memory (solved)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2  
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
Jaglover
Watchman
Watchman


Joined: 29 May 2005
Posts: 7038
Location: Saint Amant, Acadiana

PostPosted: Thu Mar 30, 2017 3:06 pm    Post subject: Reply with quote

Turning COMPACTION on will not remove the cause of original issue ...
_________________
Please learn how to denote units correctly!
Back to top
View user's profile Send private message
Bigun
Advocate
Advocate


Joined: 21 Sep 2003
Posts: 2153

PostPosted: Sat Apr 01, 2017 2:19 pm    Post subject: Reply with quote

So it's happening again:

Code:
 # ~/ps_mem.py  -S
 Private  +   Shared  =  RAM used   Swap used   Program

100.0 KiB +  76.0 KiB = 176.0 KiB         0.0 KiB     0.0 KiB   init
188.0 KiB +  80.5 KiB = 268.5 KiB         0.0 KiB     0.0 KiB   rpc.idmapd
168.0 KiB + 110.5 KiB = 278.5 KiB         0.0 KiB     0.0 KiB   cron
196.0 KiB +  88.0 KiB = 284.0 KiB         0.0 KiB     0.0 KiB   rpcbind
492.0 KiB +  74.0 KiB = 566.0 KiB         0.0 KiB     0.0 KiB   rpc.mountd
376.0 KiB + 211.0 KiB = 587.0 KiB         0.0 KiB     0.0 KiB   su
592.0 KiB + 102.0 KiB = 694.0 KiB         0.0 KiB     0.0 KiB   rpc.statd
728.0 KiB +  70.0 KiB = 798.0 KiB         0.0 KiB     0.0 KiB   smartd
  1.1 MiB + 147.0 KiB =   1.3 MiB         0.0 KiB     0.0 KiB   systemd-udevd
784.0 KiB + 578.0 KiB =   1.3 MiB         0.0 KiB     0.0 KiB   agetty (6)
664.0 KiB +   1.3 MiB =   1.9 MiB         0.0 KiB     0.0 KiB   bash (2)
  1.4 MiB + 636.0 KiB =   2.0 MiB         0.0 KiB     0.0 KiB   syslog-ng (2)
  1.9 MiB +   2.4 MiB =   4.2 MiB         0.0 KiB     0.0 KiB   sshd (3)
-------------------------------------------------------------
                         14.3 MiB         0.0 KiB     0.0 KiB
=============================================================


Code:
top - 10:19:06 up 2 days, 40 min,  1 user,  load average: 0.34, 0.36, 0.36
Tasks: 158 total,   1 running, 157 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.4 us,  1.8 sy,  1.3 ni, 89.0 id,  4.4 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem :  8230380 total,    47512 free,   718668 used,  7464200 buff/cache
KiB Swap:   979836 total,   979624 free,      212 used.  7215344 avail Mem


Code:
 # ps -A
  PID TTY          TIME CMD
    1 ?        00:00:01 init
    2 ?        00:00:00 kthreadd
    3 ?        00:00:18 ksoftirqd/0
    7 ?        00:00:14 rcu_sched
    8 ?        00:00:00 rcu_bh
    9 ?        00:00:00 migration/0
   10 ?        00:00:00 lru-add-drain
   11 ?        00:00:00 cpuhp/0
   12 ?        00:00:00 cpuhp/1
   13 ?        00:00:00 migration/1
   14 ?        00:00:00 ksoftirqd/1
   17 ?        00:00:00 kdevtmpfs
   18 ?        00:00:00 netns
  384 ?        00:00:00 oom_reaper
  385 ?        00:00:00 writeback
  387 ?        00:00:00 kcompactd0
  388 ?        00:00:00 crypto
  389 ?        00:00:00 bioset
  391 ?        00:00:00 kblockd
  566 ?        00:00:00 ata_sff
  586 ?        00:00:00 md
  595 ?        00:00:00 cfg80211
  689 ?        00:00:00 rpciod
  690 ?        00:00:00 xprtiod
  723 ?        00:05:36 kswapd0
  724 ?        00:00:00 vmstat
  803 ?        00:00:00 nfsiod
  812 ?        00:00:00 cifsiod
  822 ?        00:00:00 bioset
  830 ?        00:00:00 xfsalloc
  831 ?        00:00:00 xfs_mru_cache
  898 ?        00:00:00 acpi_thermal_pm
  916 ?        00:00:00 i915/signal:0
  917 ?        00:00:00 i915/signal:1
  918 ?        00:00:00 i915/signal:2
  935 ?        00:00:00 bioset
  936 ?        00:00:00 bioset
  937 ?        00:00:00 bioset
  938 ?        00:00:00 bioset
  939 ?        00:00:00 bioset
  940 ?        00:00:00 bioset
  941 ?        00:00:00 bioset
  942 ?        00:00:00 bioset
  943 ?        00:00:00 bioset
  944 ?        00:00:00 bioset
  945 ?        00:00:00 bioset
  946 ?        00:00:00 bioset
  947 ?        00:00:00 bioset
  948 ?        00:00:00 bioset
  949 ?        00:00:00 bioset
  950 ?        00:00:00 bioset
  985 ?        00:00:00 bioset
  988 ?        00:00:00 bioset
  991 ?        00:00:00 bioset
  994 ?        00:00:00 bioset
  997 ?        00:00:00 bioset
 1000 ?        00:00:00 bioset
 1003 ?        00:00:00 bioset
 1007 ?        00:00:00 bioset
 1009 ?        00:00:00 iscsi_eh
 1042 ?        00:00:00 scsi_eh_0
 1043 ?        00:00:00 scsi_tmf_0
 1046 ?        00:00:00 scsi_eh_1
 1047 ?        00:00:00 scsi_tmf_1
 1050 ?        00:00:00 scsi_eh_2
 1051 ?        00:00:00 scsi_tmf_2
 1052 ?        00:00:00 scsi_eh_3
 1055 ?        00:00:00 scsi_tmf_3
 1058 ?        00:00:00 scsi_eh_4
 1059 ?        00:00:00 scsi_tmf_4
 1062 ?        00:00:00 scsi_eh_5
 1063 ?        00:00:00 scsi_tmf_5
 1066 ?        00:00:00 scsi_eh_6
 1068 ?        00:00:00 scsi_tmf_6
 1071 ?        00:00:00 scsi_eh_7
 1073 ?        00:00:00 scsi_tmf_7
 1112 ?        00:00:09 kworker/1:0H
 1147 ?        00:00:00 raid5wq
 1188 ?        00:00:00 bioset
 1196 ?        00:00:00 bioset
 1205 ?        00:00:00 bioset
 1213 ?        00:00:00 bioset
 1233 ?        00:00:00 bioset
 1246 ?        00:00:00 bioset
 1258 ?        00:00:00 bioset
 1262 ?        00:00:00 bioset
 1266 ?        00:00:00 bioset
 1267 ?        00:01:40 md126_raid1
 1269 ?        00:00:00 bioset
 1273 ?        00:00:00 bioset
 1274 ?        00:00:01 md125_raid1
 1276 ?        00:00:00 bioset
 1280 ?        00:00:00 bioset
 1281 ?        00:00:00 md124_raid1
 1284 ?        00:00:00 xfs-buf/md126
 1285 ?        00:00:00 xfs-data/md126
 1286 ?        00:00:00 xfs-conv/md126
 1287 ?        00:00:00 xfs-cil/md126
 1288 ?        00:00:00 xfs-reclaim/md1
 1289 ?        00:00:00 xfs-log/md126
 1290 ?        00:00:00 xfs-eofblocks/m
 1291 ?        00:00:27 xfsaild/md126
 1695 ?        00:00:00 systemd-udevd
 1860 ?        00:00:00 bioset
 1870 ?        00:00:00 bioset
 1871 ?        00:55:37 md127_raid5
 2062 ?        00:00:00 xfs-buf/md127
 2063 ?        00:00:00 xfs-data/md127
 2064 ?        00:00:00 xfs-conv/md127
 2065 ?        00:00:00 xfs-cil/md127
 2066 ?        00:00:00 xfs-reclaim/md1
 2067 ?        00:00:00 xfs-log/md127
 2068 ?        00:00:00 xfs-eofblocks/m
 2069 ?        00:00:03 xfsaild/md127
 2771 ?        00:00:00 syslog-ng
 2772 ?        00:00:00 syslog-ng
 2829 ?        00:00:00 rpcbind
 2855 ?        00:00:00 rpc.statd
 2905 ?        00:00:00 rpc.idmapd
 2936 ?        00:00:00 rpc.mountd
 2940 ?        00:00:00 nfsd4_callbacks
 2941 ?        00:00:00 lockd
 2943 ?        00:00:00 nfsd
 2944 ?        00:00:00 nfsd
 2945 ?        00:00:00 nfsd
 2946 ?        00:00:00 nfsd
 2947 ?        00:00:00 nfsd
 2948 ?        00:00:00 nfsd
 2949 ?        00:00:00 nfsd
 2950 ?        00:00:00 nfsd
 3153 ?        00:00:00 smartd
 3183 ?        00:00:00 sshd
 3208 ?        00:00:00 cron
 3239 tty1     00:00:00 agetty
 3240 tty2     00:00:00 agetty
 3241 tty3     00:00:00 agetty
 3242 tty4     00:00:00 agetty
 3243 tty5     00:00:00 agetty
 3244 tty6     00:00:00 agetty
 3483 ?        00:00:45 cifsd
 8959 ?        00:00:10 kworker/0:2
14053 ?        00:00:04 kworker/1:2
14067 ?        00:00:00 kworker/1:1H
14357 ?        00:00:00 kworker/u4:1
14960 ?        00:00:00 kworker/u4:2
29191 ?        00:00:00 kworker/0:1
29838 ?        00:00:00 kworker/0:0H
29860 ?        00:00:00 kworker/1:1
29861 ?        00:00:00 kworker/0:2H
29895 ?        00:00:00 kworker/1:0
29897 ?        00:00:00 kworker/u4:0
29898 ?        00:00:00 kworker/0:1H
29905 ?        00:00:00 sshd
29911 ?        00:00:00 sshd
29916 pts/0    00:00:00 bash
29970 ?        00:00:00 kworker/0:0
29976 pts/0    00:00:00 su
29979 pts/0    00:00:00 bash
29982 pts/0    00:00:00 ps


Code:
 # tail -n 100 /var/log/messages
Apr  1 10:21:54 projector kernel: [ 3153]     0  3153     1098      128       7       3        0             0 smartd
Apr  1 10:21:54 projector kernel: [ 3183]     0  3183     1355      580       6       3        0         -1000 sshd
Apr  1 10:21:54 projector kernel: [ 3208]     0  3208      600      452       5       3        0             0 cron
Apr  1 10:21:54 projector kernel: [ 3239]     0  3239     1096      378       6       3        0             0 agetty
Apr  1 10:21:54 projector kernel: [ 3240]     0  3240     1096      351       6       3        0             0 agetty
Apr  1 10:21:54 projector kernel: [ 3241]     0  3241     1096      370       6       3        0             0 agetty
Apr  1 10:21:54 projector kernel: [ 3242]     0  3242     1096      359       6       3        0             0 agetty
Apr  1 10:21:54 projector kernel: [ 3243]     0  3243     1096      351       6       3        0             0 agetty
Apr  1 10:21:54 projector kernel: [ 3244]     0  3244     1096      349       5       3        0             0 agetty
Apr  1 10:21:54 projector kernel: [29905]     0 29905     2086     1172       7       3        0             0 sshd
Apr  1 10:21:54 projector kernel: [29911]  1000 29911    10412     1017      11       3        0             0 sshd
Apr  1 10:21:54 projector kernel: [29916]  1000 29916      925      741       4       3        0             0 bash
Apr  1 10:21:54 projector kernel: [29976]  1000 29976      755      578       5       3        0             0 su
Apr  1 10:21:54 projector kernel: [29979]     0 29979      925      761       6       3        0             0 bash
Apr  1 10:21:54 projector kernel: [29984]     0 29984      563      184       5       3        0             0 dmesg
Apr  1 10:21:54 projector kernel: Out of memory: Kill process 29905 (sshd) score 0 or sacrifice child
Apr  1 10:21:54 projector kernel: Killed process 29911 (sshd) total-vm:41648kB, anon-rss:736kB, file-rss:3332kB, shmem-rss:0kB
Apr  1 10:21:54 projector kernel: dmesg invoked oom-killer: gfp_mask=0x24040c0(GFP_KERNEL|__GFP_COMP), nodemask=0, order=2, oom_score_adj=0
Apr  1 10:21:54 projector kernel: dmesg cpuset=/ mems_allowed=0
Apr  1 10:21:54 projector kernel: CPU: 0 PID: 29984 Comm: dmesg Not tainted 4.9.6-gentoo-r1 #2
Apr  1 10:21:54 projector kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./H61M-DS2H, BIOS F5 04/02/2012
Apr  1 10:21:54 projector kernel:  d5e75ca4 c12f0831 d5e75d7c c18ff2c4 d5e75ccc c10e86d5 00000000 d4c95900
Apr  1 10:21:54 projector kernel:  00200206 d5e75ccc c12f49e2 f0f40900 c18ff2c4 00000000 d5e75cf0 c10b571b
Apr  1 10:21:54 projector kernel:  c10b556d 00000000 00000000 d5e75d7c d5e75d7c c19f2700 c19f29d4 d5e75d0c
Apr  1 10:21:54 projector kernel: Call Trace:
Apr  1 10:21:54 projector kernel:  [<c12f0831>] dump_stack+0x47/0x5b
Apr  1 10:21:54 projector kernel:  [<c10e86d5>] dump_header.isra.13+0x5d/0x168
Apr  1 10:21:54 projector kernel:  [<c12f49e2>] ? ___ratelimit+0xa1/0xab
Apr  1 10:21:54 projector kernel:  [<c10b571b>] oom_kill_process+0x66/0x2fe
Apr  1 10:21:54 projector kernel:  [<c10b556d>] ? oom_badness+0xc5/0xfc
Apr  1 10:21:54 projector kernel:  [<c10b5d08>] out_of_memory+0x254/0x28b
Apr  1 10:21:54 projector kernel:  [<c10b89d3>] __alloc_pages_nodemask+0x91f/0x9c7
Apr  1 10:21:54 projector kernel:  [<c12a9600>] ? avc_has_perm_noaudit+0x1/0x83
Apr  1 10:21:54 projector kernel:  [<c10c8a25>] kmalloc_order+0x16/0x28
Apr  1 10:21:54 projector kernel:  [<c106b16b>] devkmsg_open+0x39/0xc6
Apr  1 10:21:54 projector kernel:  [<c1345a5b>] memory_open+0x48/0x4c
Apr  1 10:21:54 projector kernel:  [<c10ee281>] chrdev_open+0x10c/0x12a
Apr  1 10:21:54 projector kernel:  [<c10e8e24>] do_dentry_open+0x193/0x272
Apr  1 10:21:54 projector kernel:  [<c10ee175>] ? cdev_put+0x1a/0x1a
Apr  1 10:21:54 projector kernel:  [<c10e9be2>] vfs_open+0x45/0x4e
Apr  1 10:21:54 projector kernel:  [<c10f634b>] path_openat+0xae7/0xcb2
Apr  1 10:21:54 projector kernel:  [<c10f6547>] do_filp_open+0x31/0x77
Apr  1 10:21:54 projector kernel:  [<c10ffc3b>] ? __alloc_fd+0x72/0x10f
Apr  1 10:21:54 projector kernel:  [<c10e9ee1>] do_sys_open+0x12d/0x1a6
Apr  1 10:21:54 projector kernel:  [<c10e9f72>] SyS_open+0x18/0x1a
Apr  1 10:21:54 projector kernel:  [<c1001023>] do_fast_syscall_32+0x8b/0xf6
Apr  1 10:21:54 projector kernel:  [<c16cf1ab>] sysenter_past_esp+0x40/0x6a
Apr  1 10:21:54 projector kernel: Mem-Info:
Apr  1 10:21:54 projector kernel: active_anon:354 inactive_anon:1532 isolated_anon:0\x0a active_file:773116 inactive_file:1064369 isolated_file:0\x0a unevictable:0 dirty:0 writeback:0 unstable:0\x0a slab_reclaimable:28383 slab_unreclaimable:13352\x0a mapped:1283 shmem:234 pagetables:134 bounce:0\x0a free:12096 free_pcp:528 free_cma:0
Apr  1 10:21:54 projector kernel: Node 0 active_anon:1416kB inactive_anon:6128kB active_file:3092464kB inactive_file:4257476kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:5132kB dirty:0kB writeback:0kB shmem:936kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Apr  1 10:21:54 projector kernel: DMA free:3212kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15976kB managed:15360kB mlocked:0kB slab_reclaimable:92kB slab_unreclaimable:1648kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Apr  1 10:21:54 projector kernel: lowmem_reserve[]: 0 793 8021 8021
Apr  1 10:21:54 projector kernel: Normal free:5148kB min:3572kB low:4464kB high:5356kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:20kB unevictable:0kB writepending:0kB present:894968kB managed:813356kB mlocked:0kB slab_reclaimable:113440kB slab_unreclaimable:51760kB kernel_stack:1264kB pagetables:0kB bounce:0kB free_pcp:852kB local_pcp:144kB free_cma:0kB
Apr  1 10:21:54 projector kernel: lowmem_reserve[]: 0 0 57825 57825
Apr  1 10:21:54 projector kernel: HighMem free:40024kB min:512kB low:8644kB high:16776kB active_anon:1416kB inactive_anon:6128kB active_file:3092464kB inactive_file:4257456kB unevictable:0kB writepending:0kB present:7401664kB managed:7401664kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:536kB bounce:0kB free_pcp:1260kB local_pcp:648kB free_cma:0kB
Apr  1 10:21:54 projector kernel: lowmem_reserve[]: 0 0 0 0
Apr  1 10:21:54 projector kernel: DMA: 11*4kB (UE) 6*8kB (UE) 3*16kB (E) 6*32kB (UE) 1*64kB (U) 4*128kB (UE) 3*256kB (UE) 1*512kB (U) 1*1024kB (E) 0*2048kB 0*4096kB = 3212kB
Apr  1 10:21:54 projector kernel: Normal: 897*4kB (ME) 195*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5148kB
Apr  1 10:21:54 projector kernel: HighMem: 2744*4kB (UM) 2351*8kB (UM) 69*16kB (UM) 19*32kB (UM) 4*64kB (UM) 3*128kB (UM) 3*256kB (UM) 2*512kB (U) 4*1024kB (M) 1*2048kB (M) 0*4096kB = 40072kB
Apr  1 10:21:54 projector kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr  1 10:21:54 projector kernel: 1837719 total pagecache pages
Apr  1 10:21:54 projector kernel: 0 pages in swap cache
Apr  1 10:21:54 projector kernel: Swap cache stats: add 462, delete 462, find 9/9
Apr  1 10:21:54 projector kernel: Free swap  = 979624kB
Apr  1 10:21:54 projector kernel: Total swap = 979836kB
Apr  1 10:21:54 projector kernel: 2078152 pages RAM
Apr  1 10:21:54 projector kernel: 1850416 pages HighMem/MovableOnly
Apr  1 10:21:54 projector kernel: 20557 pages reserved
Apr  1 10:21:54 projector kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Apr  1 10:21:54 projector kernel: [ 1695]     0  1695     2740      730       6       3        0         -1000 systemd-udevd
Apr  1 10:21:54 projector kernel: [ 2771]     0  2771     1703      103       7       3        0             0 syslog-ng
Apr  1 10:21:54 projector kernel: [ 2772]     0  2772     6470      731      10       3        0             0 syslog-ng
Apr  1 10:21:54 projector kernel: [ 2829]     0  2829      613       39       5       3        0             0 rpcbind
Apr  1 10:21:54 projector kernel: [ 2855]     0  2855      730      454       5       3        0             0 rpc.statd
Apr  1 10:21:54 projector kernel: [ 2905]     0  2905      692       45       5       3        0             0 rpc.idmapd
Apr  1 10:21:54 projector kernel: [ 2936]     0  2936      817      123       5       3        0             0 rpc.mountd
Apr  1 10:21:54 projector kernel: [ 3153]     0  3153     1098      128       7       3        0             0 smartd
Apr  1 10:21:54 projector kernel: [ 3183]     0  3183     1355      580       6       3        0         -1000 sshd
Apr  1 10:21:54 projector kernel: [ 3208]     0  3208      600      452       5       3        0             0 cron
Apr  1 10:21:54 projector kernel: [ 3239]     0  3239     1096      378       6       3        0             0 agetty
Apr  1 10:21:54 projector kernel: [ 3240]     0  3240     1096      351       6       3        0             0 agetty
Apr  1 10:21:54 projector kernel: [ 3241]     0  3241     1096      370       6       3        0             0 agetty
Apr  1 10:21:54 projector kernel: [ 3242]     0  3242     1096      359       6       3        0             0 agetty
Apr  1 10:21:54 projector kernel: [ 3243]     0  3243     1096      351       6       3        0             0 agetty
Apr  1 10:21:54 projector kernel: [ 3244]     0  3244     1096      349       5       3        0             0 agetty
Apr  1 10:21:54 projector kernel: [29916]  1000 29916      925      741       4       3        0             0 bash
Apr  1 10:21:54 projector kernel: [29976]  1000 29976      755      578       5       3        0             0 su
Apr  1 10:21:54 projector kernel: [29979]     0 29979      925      761       6       3        0             0 bash
Apr  1 10:21:54 projector kernel: [29984]     0 29984      563      184       5       3        0             0 dmesg
Apr  1 10:21:54 projector kernel: Out of memory: Kill process 29916 (bash) score 0 or sacrifice child
Apr  1 10:21:54 projector kernel: Killed process 29976 (su) total-vm:3020kB, anon-rss:304kB, file-rss:2008kB, shmem-rss:0kB
Apr  1 10:21:54 projector kernel: oom_reaper: reaped process 29984 (dmesg), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr  1 10:22:21 projector sshd[29988]: SSH: Server;Ltype: Version;Remote: 192.168.0.104-15575;Protocol: 2.0;Client: PuTTY_Release_0.66
Apr  1 10:22:21 projector sshd[29988]: SSH: Server;Ltype: Kex;Remote: 192.168.0.104-15575;Enc: aes256-ctr;MAC: hmac-sha2-256;Comp: none [preauth]
Apr  1 10:22:26 projector sshd[29988]: Accepted keyboard-interactive/pam for bigun from 192.168.0.104 port 15575 ssh2
Apr  1 10:22:26 projector sshd[29988]: pam_unix(sshd:session): session opened for user bigun by (uid=0)
Apr  1 10:22:26 projector sshd[29994]: SSH: Server;Ltype: Kex;Remote: 192.168.0.104-15575;Enc: aes256-ctr;MAC: hmac-sha2-256;Comp: none
Apr  1 10:23:45 projector su[30003]: Successful su for root by bigun
Apr  1 10:23:45 projector su[30003]: + /dev/pts/0 bigun:root
Apr  1 10:23:45 projector su[30003]: pam_unix(su:session): session opened for user root by bigun(uid=1000)

_________________
"It's ok, they might have guns but we have flowers." - Perpetual Victim
Back to top
View user's profile Send private message
cboldt
l33t
l33t


Joined: 24 Aug 2005
Posts: 829

PostPosted: Sat Apr 01, 2017 4:10 pm    Post subject: Reply with quote

One purported fix (found at unix.stackexchange.com ... I think linked above)

Code:
sync && echo 1 > /proc/sys/vm/drop_caches


Here is another shot in the dark ...

Code:
root@hypoid-2 [3] 26 /root # zgrep SL[AU]B /proc/config.gz
CONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_SLAB_FREELIST_RANDOM is not set
CONFIG_SLABINFO=y
# CONFIG_DEBUG_SLAB is not set


I'm not getting close to full RAM use, so no swapping going on here - just saying, I'm not pushing up against the RAM limit, so don't know if those settings "work" to make swap happen, or not. The mention of SLAB/SLUB is in https://lkml.org/lkml/2016/12/12/49, you may have to follow some of the links around.

Edit to add a couple more remarks and references.

The preference to resolve this is SLUB, not SLAB. In other words, the settings on my system would not tend to resolve oom-killer acting before the kernel resorts to swap.

The "sync && echo 1 > /proc/sys/vm/drop_caches" has to be in a cronjob, not a one-shot. See https://bugzilla.redhat.com/show_bug.cgi?id=1373339
Back to top
View user's profile Send private message
tholin
Apprentice
Apprentice


Joined: 04 Oct 2008
Posts: 167

PostPosted: Sat Apr 01, 2017 7:36 pm    Post subject: Reply with quote

Bigun wrote:
So it's happening again:

You have plenty of available (easily freeable) ram so the OOM condition is probably a bug. You are not locking huge amounts of page cache right?
There have been a lot of problems with memory management and early OOM with recent kernels. You use 4.9.6 which is an old 4.9 version. Try 4.9.20. Here are some patches for fixing OOM conditions that went in 4.9.7. No idea if that's your problem though.

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=96e5cec10e7a75c931f8993633b3a5cedc99144e
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=ade7afe9dca6b13919f88abd38eefe32f22eaeb3
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=d1656c5aef4d72f03a7833d07a378c8f604b8307

In general I would advice against running any kernel that hasn't been "stable" for at least 6 months. Unless you really need some of the changes in 4.9 try downgrading to 4.4.59.


cboldt wrote:
sync && echo 1 > /proc/sys/vm/drop_caches

That's a horribly ugly workaround. Better to find what cause the problem than doing things like that.
Back to top
View user's profile Send private message
Roman_Gruber
Advocate
Advocate


Joined: 03 Oct 2006
Posts: 3806
Location: Austro Bavaria

PostPosted: Sat Apr 01, 2017 9:21 pm    Post subject: Reply with quote

4.4 branch is also buggy. I think it depends on the platform and the use case. I had another topic htis week i responded where the 4.4. branch was the culprit for the hole topic. I update my kernel on a weekly basis. And i stay on 4.9 becasue 4.10 is not supported by the binary nvidia drivers afaik.
Back to top
View user's profile Send private message
Bigun
Advocate
Advocate


Joined: 21 Sep 2003
Posts: 2153

PostPosted: Sun Apr 02, 2017 9:37 am    Post subject: Reply with quote

tholin wrote:
Bigun wrote:
So it's happening again:
...You are not locking huge amounts of page cache right?


I'm not sure what you mean by locking, but here is my current cache usage:

Code:
 $ free
              total        used        free      shared  buff/cache   available
Mem:        8230380      425608       21528         776     7783244     7508572
Swap:        979836           0      979836

_________________
"It's ok, they might have guns but we have flowers." - Perpetual Victim
Back to top
View user's profile Send private message
Bigun
Advocate
Advocate


Joined: 21 Sep 2003
Posts: 2153

PostPosted: Sun Apr 02, 2017 9:40 am    Post subject: Reply with quote

Now this is interesting.

Code:
[ 1774.540540] kworker/dying (1072) used greatest stack depth: 5424 bytes left
[54758.518469] kworker/dying (3320) used greatest stack depth: 5416 bytes left
[66456.307781] kworker/dying (30573) used greatest stack depth: 5296 bytes left

_________________
"It's ok, they might have guns but we have flowers." - Perpetual Victim
Back to top
View user's profile Send private message
Bigun
Advocate
Advocate


Joined: 21 Sep 2003
Posts: 2153

PostPosted: Sun Apr 02, 2017 9:49 am    Post subject: Reply with quote

Roman_Gruber wrote:
4.4 branch is also buggy. I think it depends on the platform and the use case. I had another topic htis week i responded where the 4.4. branch was the culprit for the hole topic. I update my kernel on a weekly basis. And i stay on 4.9 becasue 4.10 is not supported by the binary nvidia drivers afaik.


I usually update to a newer kernel when it gets marked as stable anyway, upgrading now. We'll see if this works.

cboldt wrote:
One purported fix (found at unix.stackexchange.com ... I think linked above)

Code:
sync && echo 1 > /proc/sys/vm/drop_caches


...



Going to try this next. I want to pinpoint why it's happening.
_________________
"It's ok, they might have guns but we have flowers." - Perpetual Victim
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 42847
Location: 56N 3W

PostPosted: Sun Apr 02, 2017 10:10 am    Post subject: Reply with quote

Bigun,

Code:
sync && echo 1 > /proc/sys/vm/drop_caches
if that works, you have a kernel bug.
The kernel is supposed to do that for itself by way of normal operation.

sync flushes dirty buffers to the disc. The filesystem ensures that there is nothing more than a few seconds old in dirty buffers anyway.
drop_caches, flushes clean buffers, which the kernel is supposed to do when it needs RAM.
However, drop_caches, drops all caches, which will have a performance impact. The kernel mechanism only drops what's needed.

As has been said, its a horrible hack.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Bigun
Advocate
Advocate


Joined: 21 Sep 2003
Posts: 2153

PostPosted: Thu Apr 06, 2017 1:57 pm    Post subject: Reply with quote

I'm starting to think it was a kernel bug. After upgrading to 4.9.16, I've noticed caching doesn't completely consume my RAM, and the SWAP is actually being used. Four days uptime, when I get to 14, I'll consider the issue resolved, but I'm starting to guess that was the issue.
_________________
"It's ok, they might have guns but we have flowers." - Perpetual Victim
Back to top
View user's profile Send private message
Bigun
Advocate
Advocate


Joined: 21 Sep 2003
Posts: 2153

PostPosted: Mon Apr 17, 2017 4:11 pm    Post subject: Reply with quote

Consider it solved. Upgrading to the new kernel version seems to have fixed the issue.
_________________
"It's ok, they might have guns but we have flowers." - Perpetual Victim
Back to top
View user's profile Send private message
Carnildo
Guru
Guru


Joined: 17 Jun 2004
Posts: 593

PostPosted: Mon Apr 17, 2017 10:56 pm    Post subject: Reply with quote

Bigun wrote:
I'm starting to think it was a kernel bug. After upgrading to 4.9.16, I've noticed caching doesn't completely consume my RAM, and the SWAP is actually being used. Four days uptime, when I get to 14, I'll consider the issue resolved, but I'm starting to guess that was the issue.


It almost certainly was: https://bugzilla.kernel.org/show_bug.cgi?id=190351 and https://bugzilla.redhat.com/show_bug.cgi?id=1401012
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Goto page Previous  1, 2
Page 2 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum