Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
(Kernel?) memory leak [SOLVED]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
jhoos
n00b
n00b


Joined: 30 May 2004
Posts: 42

PostPosted: Sun Jun 19, 2016 7:32 pm    Post subject: (Kernel?) memory leak [SOLVED] Reply with quote

Hi,

i have a memory problem on a fresh gentoo install. After installing and booting something starts to eat up my ram and after a while the kernel crashes and the system halts.

When I look at /proc/meminfo I can see that SUnreclaim increases all the time at approximately 2-8 thousand kb per second which takes about 10 to 20 minutes until no memory is left.

It is neither the buffers that takes the ram nor any visible process. The largest process is only about 4 MB:

Code:
 ps aux  | awk '{print $6/1024 " MB\t\t" $11}'  | sort -nr
4.51953 MB              sshd:
3.75781 MB              bash
3.59766 MB              bash
3.53906 MB              -bash
3.52344 MB              -bash
3.28906 MB              sshd:
...


Meminfo shows this and one can see at SUnreclaim that the crash will happen soon.

Code:
 cat /proc/meminfo                                                                                                                       Sun Jun 19 19:26:11 2016

MemTotal:        4041184 kB
MemFree:           31080 kB
MemAvailable:     483988 kB
Buffers:              60 kB
Cached:           429312 kB
SwapCached:            0 kB
Active:           218264 kB
Inactive:         219124 kB
Active(anon):       3684 kB
Inactive(anon):     4820 kB
Active(file):     214580 kB
Inactive(file):   214304 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4399100 kB
SwapFree:        4399100 kB
Dirty:              4800 kB
Writeback:             0 kB
AnonPages:          8144 kB
Mapped:             7380 kB
Shmem:               360 kB
Slab:            3525980 kB
SReclaimable:      54108 kB
SUnreclaim:      3471872 kB
KernelStack:        2784 kB
PageTables:         1476 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6419692 kB
Committed_AS:      15292 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       63040 kB
DirectMap2M:     4130816 kB


Code:
 emerge --info
Portage 2.2.28 (python 2.7.10-final-0, default/linux/amd64/13.0, gcc-4.9.3, glibc-2.22-r4, 4.4.6-gentoo x86_64)
=================================================================
System uname: Linux-4.4.6-gentoo-x86_64-Intel-R-_Atom-TM-_CPU_D510_@_1.66GHz-with-gentoo-2.2
KiB Mem:     4041184 total,   1276836 free
KiB Swap:    4399100 total,   4399100 free
Timestamp of repository gentoo: Sun, 19 Jun 2016 00:45:01 +0000
sh bash 4.3_p42-r1
ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1
app-shells/bash:          4.3_p42-r1::gentoo
dev-lang/perl:            5.20.2::gentoo
dev-lang/python:          2.7.10-r1::gentoo, 3.4.3-r1::gentoo
dev-util/pkgconfig:       0.28-r2::gentoo
sys-apps/baselayout:      2.2::gentoo
sys-apps/openrc:          0.19.1::gentoo
sys-apps/sandbox:         2.10-r1::gentoo
sys-devel/autoconf:       2.69::gentoo
sys-devel/automake:       1.14.1::gentoo, 1.15::gentoo
sys-devel/binutils:       2.25.1-r1::gentoo
sys-devel/gcc:            4.9.3::gentoo
sys-devel/gcc-config:     1.7.3::gentoo
sys-devel/libtool:        2.4.6::gentoo
sys-devel/make:           4.1-r1::gentoo
sys-kernel/linux-headers: 4.3::gentoo (virtual/os-headers)
sys-libs/glibc:           2.22-r4::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="de_DE.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j1"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="acl amd64 berkdb bindist bzip2 cli cracklib crypt cxx dri fortran gdbm iconv ipv6 mmx mmxext modules multilib ncurses nls nptl openmp pam pcre readline seccomp session sse sse2 ssl tcpd unicode xattr zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4" RUBY_TARGETS="ruby20 ruby21" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON


Here is my make.conf:

Code:
cat /etc/portage/make.conf
# These settings were set by the catalyst build script that automatically
# built this stage.
# Please consult /usr/share/portage/config/make.conf.example for a more
# detailed example.
CFLAGS="-march=native -O2 -pipe"
CXXFLAGS="${CFLAGS}"
MAKEOPTS="-j1"
# WARNING: Changing your CHOST is not something that should be done lightly.
# Please consult http://www.gentoo.org/doc/en/change-chost.xml before changing.
CHOST="x86_64-pc-linux-gnu"
# These are the USE flags that were used in addition to what is provided by the
# profile used for building.
USE="bindist mmx sse sse2"
PORTDIR="/usr/portage"
DISTDIR="${PORTDIR}/distfiles"
PKGDIR="${PORTDIR}/packages"


Kernel was compiled during install with genkernel.

How can I find out what causes this? I suppose it to be the kernel or a kernel module. But how do I find out which one and why?


Last edited by jhoos on Thu Jun 23, 2016 12:36 pm; edited 1 time in total
Back to top
View user's profile Send private message
fedeliallalinea
Bodhisattva
Bodhisattva


Joined: 08 Mar 2003
Posts: 22219
Location: here

PostPosted: Mon Jun 20, 2016 10:37 am    Post subject: Re: (Kernel?) memory leak Reply with quote

jhoos wrote:
How can I find out what causes this? I suppose it to be the kernel or a kernel module. But how do I find out which one and why?

slabtop command can help?
_________________
Questions are guaranteed in life; Answers aren't.
Back to top
View user's profile Send private message
jhoos
n00b
n00b


Joined: 30 May 2004
Posts: 42

PostPosted: Mon Jun 20, 2016 11:14 am    Post subject: Reply with quote

Hi,

slabtop points to kmalloc-64:

Code:
Objekte aktiv / gesamt (% benutzt) : 62003361 / 62030053 (100,0%)
 Slabs aktiv / gesamt (% benutzt)   : 986142 / 986143 (100,0%)
 Caches aktiv / gesamt (% benutzt)  : 63 / 138 (45,7%)
 GrM-CM-6M-C~_e aktiv / gesamt (% benutzt) : 3872772,21K / 3885016,04K (99,7%)
 Minimum / Average / Maximum Object : 0,02K / 0,06K / 4096,00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
60774966 60774963  29%    0,06K 964682       63   3858728K kmalloc-64
1064168 1064104  99%    0,03K   8582      124     34328K kmalloc-32
 89992  89945  99%    0,07K   1607       56      6428K ext4_io_end
 39312  21384  54%    0,56K   5616        7     22464K radix_tree_node
 18348  18234  99%    0,12K    556       33      2224K kernfs_node_cache
  6944   6887  99%    0,12K    224       31       896K kmalloc-node
  4368   1577  36%    0,19K    208       21       832K dentry
  3780   3627  95%    0,14K    135       28       540K btrfs_path
  3108    672  21%    0,27K    222       14       888K btrfs_extent_buffer
  2904   2904 100%    0,50K    363        8      1452K kmalloc-512
  2583   2583 100%    0,19K    123       21       492K kmalloc-192
  1960   1870  95%    0,07K     35       56       140K Acpi-Operand
  1850    863  46%    0,08K     37       50       148K btrfs_extent_state
  1817   1769  97%    0,17K     79       23       316K vm_area_struct
  1811   1811 100%    4,00K   1811        1      7244K kmalloc-4096
  1616   1263  78%    0,25K    101       16       404K kmalloc-256
  1089   1051  96%    0,04K     11       99        44K Acpi-Namespace
   980    814  83%    0,95K    245        4       980K btrfs_inode
   966    938  97%    0,59K    161        6       644K shmem_inode_cache
   961    868  90%    0,12K     31       31       124K kmalloc-96
   952    824  86%    0,07K     17       56        68K anon_vma
Back to top
View user's profile Send private message
fedeliallalinea
Bodhisattva
Bodhisattva


Joined: 08 Mar 2003
Posts: 22219
Location: here

PostPosted: Mon Jun 20, 2016 3:18 pm    Post subject: Reply with quote

Try with a newer kernel first.
For deepen analysis would need a tool like this or this but my knowledge ends here.
_________________
Questions are guaranteed in life; Answers aren't.
Back to top
View user's profile Send private message
jhoos
n00b
n00b


Joined: 30 May 2004
Posts: 42

PostPosted: Tue Jun 21, 2016 6:05 pm    Post subject: Reply with quote

Switched kernel from 4.4.6 to 4.6.2 with same result. My self compiled kernel eats my ram. Now playing with kedr. Got it installed and running but yet missing a clear strategy on how to make use of it to locate the cause of my problem. To be continued...
Back to top
View user's profile Send private message
jhoos
n00b
n00b


Joined: 30 May 2004
Posts: 42

PostPosted: Wed Jun 22, 2016 8:05 am    Post subject: Reply with quote

Put away kedr for now as it seems to be more for testing a single modul for memory leaks instead of finding which module causes it.

What I did now is to recomplile the kernel with CONFIG_DEBUG_SLAB and CONFIG_DEBUG_SLAB_LEAK enabled which creates a file /proc/slab_allocators that shows who is consuming the slab objects:

Code:
cat /proc/slab_allocators|sort -g -k2 -r                                                                                                                                                                                                                                                                                                 

btrfs_free_space: 90999 __load_free_space_cache+0x32c/0x700
btrfs_extent_buffer: 57353 __alloc_extent_buffer+0x2a/0xe0
radix_tree_node: 38393 radix_tree_node_alloc+0x5f/0xb0
kernfs_node_cache: 18935 __kernfs_new_node+0x41/0xc0
selinux_inode_security: 17631 selinux_inode_alloc_security+0x3a/0xa0
dentry: 17630 __d_alloc+0x25/0x180
inode_cache: 14536 alloc_inode+0x51/0x90
btrfs_extent_map: 3582 alloc_extent_map+0x1a/0x60
ftrace_event_field: 3156 __trace_define_field+0x36/0xa0
Acpi-Operand: 1858 acpi_ut_allocate_object_desc_dbg+0x3c/0x68
trace_event_file: 1535 trace_create_new_event+0x20/0x70
btrfs_inode: 1390 btrfs_alloc_inode+0x1f/0x210
Acpi-Namespace: 1029 acpi_ns_create_node+0x37/0x46
shmem_inode_cache: 986 shmem_alloc_inode+0x1a/0x30
btrfs_extent_state: 948 alloc_extent_state+0x21/0xd0
proc_inode_cache: 627 proc_alloc_inode+0x1d/0xb0
anon_vma_chain: 527 anon_vma_clone+0x68/0x1b0
vm_area_struct: 509 mmap_region+0x281/0x580
vm_area_struct: 509 copy_process.part.42+0xb59/0x1990
anon_vma: 415 anon_vma_prepare+0xf2/0x150
selinux_file_security: 348 selinux_file_alloc_security+0x37/0x60
anon_vma_chain: 336 anon_vma_prepare+0x48/0x150
vm_area_struct: 328 __split_vma.isra.43+0x5c/0x1b0
anon_vma: 303 anon_vma_fork+0x65/0x120
idr_layer_cache: 280 ida_pre_get+0x64/0xe0
anon_vma_chain: 276 anon_vma_fork+0x97/0x120
task_delay_info: 197 __delayacct_tsk_init+0x1e/0x40
radix_tree_node: 84 __radix_tree_preload+0x35/0x90
vm_area_struct: 58 do_brk+0x23d/0x2e0
vm_area_struct: 40 __install_special_mapping+0x3c/0x110
idr_layer_cache: 36 idr_layer_alloc+0x2b/0x80
file_lock_ctx: 34 locks_get_lock_context+0x9b/0x110
xfs_ioend: 32 mempool_alloc_slab+0x15/0x20
jfs_mp: 32 mempool_alloc_slab+0x15/0x20
idr_layer_cache: 32 idr_preload+0x63/0xa0
cfq_queue: 31 cfq_get_queue+0x184/0x410
request_queue: 30 blk_alloc_queue_node+0x28/0x2a0
inotify_inode_mark: 30 SyS_inotify_add_watch+0x20a/0x310
cfq_io_cq: 27 ioc_create_icq+0x32/0x150
btrfs_free_space: 27 __btrfs_add_free_space+0x31/0x360
avc_node: 26 avc_alloc_node+0x28/0x140
blkdev_requests: 24 alloc_request_struct+0x17/0x20
buffer_head: 22 alloc_buffer_head+0x22/0x70
vm_area_struct: 20 do_execveat_common.isra.39+0x25e/0x6b0
btrfs_delayed_node: 20 btrfs_get_or_create_delayed_node+0x66/0x1a0
blkdev_ioc: 19 create_task_io_context+0x21/0x100
eventpoll_pwq: 13 ep_ptable_queue_proc+0x2d/0xa0
radix_tree_node: 12 radix_tree_node_alloc+0x3c/0xb0
ip_fib_alias: 10 fib_table_insert+0x180/0x460


The elements that constantly grow is btrfs_extent_buffer and radix_tree_node. After some while both values drop a lot (e.g. btrfs_extent_buffer from approx. 60.000 to 6.000) but no memory is freed.

Code:
btrfs_free_space: 90999 __load_free_space_cache+0x32c/0x700
kernfs_node_cache: 18934 __kernfs_new_node+0x41/0xc0
selinux_inode_security: 15572 selinux_inode_alloc_security+0x3a/0xa0
dentry: 15571 __d_alloc+0x25/0x180
inode_cache: 12484 alloc_inode+0x51/0x90
radix_tree_node: 12236 radix_tree_node_alloc+0x5f/0xb0
btrfs_extent_buffer: 6813 __alloc_extent_buffer+0x2a/0xe0
btrfs_extent_map: 3582 alloc_extent_map+0x1a/0x60


So I suspect either btrfs_extent_buffer or radix_tree_node to be the cause. I'll now try a complete reinstall but this time on an ext4 partition and omit the btrfs support in the kernel to see how the system behaves without btrfs support.
Back to top
View user's profile Send private message
jhoos
n00b
n00b


Joined: 30 May 2004
Posts: 42

PostPosted: Wed Jun 22, 2016 2:08 pm    Post subject: Reply with quote

Think I can confirm it to btrfs related. Install on ext4 and without btrfs support compiled into the kernel shows system is up and running fine. Free mem stays up.
Back to top
View user's profile Send private message
fedeliallalinea
Bodhisattva
Bodhisattva


Joined: 08 Mar 2003
Posts: 22219
Location: here

PostPosted: Wed Jun 22, 2016 2:29 pm    Post subject: Reply with quote

jhoos wrote:
Think I can confirm it to btrfs related. Install on ext4 and without btrfs support compiled into the kernel shows system is up and running fine. Free mem stays up.

Good to know.

In this article suggest to add this command in crontab for free memory
Code:
echo 2 > /proc/sys/vm/drop_caches

_________________
Questions are guaranteed in life; Answers aren't.
Back to top
View user's profile Send private message
jhoos
n00b
n00b


Joined: 30 May 2004
Posts: 42

PostPosted: Thu Jun 23, 2016 12:35 pm    Post subject: Reply with quote

I analysed this a little bit further:

I can compile btrfs into the kernel and can create and mount new btrfs partitions without problems. The problem arises only if i try to mount my main and existing btrfs data volume. As soon as i mount that volume my memory runs down. I assume this volume to be somehow defective and the btrfs kernel to have a memory leak in case it shall work on this defective volume.

This gives me some doubts about the stability of BTRFS but i'll give it another try and start with a whole new btrfs partion and restore my data from backup.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum