Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
rlimit hit despite ignore_rlimit_data
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
Sargastic
n00b
n00b


Joined: 28 Aug 2011
Posts: 35

PostPosted: Sat Jan 25, 2020 8:29 pm    Post subject: rlimit hit despite ignore_rlimit_data Reply with quote

Hello,

I am running a very memory-intensive program (psteal.py, from plaso, if it may be useful).

On a 32Gb-Ram computer, after several days of work, the program dies with

Code:
kernel: mmap: psteal.py (26061): VmData 4295028736 exceed data ulimit 4294967296. Update limits or use boot option ignore_rlimit_data.


So I modified grub and boot options, which are now

Code:
linux   /vmlinuz-4.19.97-gentoo-LTS root=/dev/mapper/root ro keymap=fr crypt_root=UUID=<UUID> root=/dev/mapper/root ignore_rlimit_data


After a reboot, ulimit seems to tell that there are no limits anymore :

Code:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127973
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 127973
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


Despite this, the program still dies, with

Code:
kernel: mmap: psteal.py (8553): VmData 4295192576 exceed data ulimit 4294967296. Update limits.


The « use ignore_rlimit_data boot option » has disappeared, but the problem still remains. I may have misunderstood something somewhere.

Any idea as to how to proceed ?

Tia.
Back to top
View user's profile Send private message
Ionen
Guru
Guru


Joined: 06 Dec 2018
Posts: 538

PostPosted: Sun Jan 26, 2020 2:11 am    Post subject: Reply with quote

I'm not overly familiar with how rlimit works but "unlimited" means as high as it can go rather than unlimited, so it's simply the max value of unsigned integers. However it should be using 64bit ints (allowing for 16 exa bytes ram usage instead of 4GB) unless it's a 32bit python interpreter or something (is it? alternatively, maybe it's self-limiting for security and changing the limits, can't say I know how that script or python in general does things)

For ignore_rlimit_data, you can check/change the value from "/sys/module/kernel/parameters/ignore_rlimit_data", not that it's something I ever touched nor should be needed. If set it should still show the warning (minus the notice to use ignore_rlimit_data), but allow expansion of vm space anyway (if it's possible).
Back to top
View user's profile Send private message
mike155
Veteran
Veteran


Joined: 17 Sep 2010
Posts: 1982
Location: Frankfurt, Germany

PostPosted: Sun Jan 26, 2020 2:46 am    Post subject: Reply with quote

@Sargastic: I can't tell you why your program fails - but I can explain some of your observations.

The only place where I found the 'exceed data ulimit' message is function 'may_expand_vm()' in '/usr/src/linux/mm/mmap.c'.
Code:
/*
 * Return true if the calling process may expand its vm space by the passed
 * number of pages
 */
bool may_expand_vm(struct mm_struct *mm, vm_flags_t flags, unsigned long npages)
{
  if (mm->total_vm + npages > rlimit(RLIMIT_AS) >> PAGE_SHIFT)
      return false;

  if (is_data_mapping(flags) &&
    mm->data_vm + npages > rlimit(RLIMIT_DATA) >> PAGE_SHIFT) {
      /* Workaround for Valgrind */
      if (rlimit(RLIMIT_DATA) == 0 &&
        mm->data_vm + npages <= rlimit_max(RLIMIT_DATA) >> PAGE_SHIFT)
          return true;

      pr_warn_once("%s (%d): VmData %lu exceed data ulimit %lu. Update limits%s.\n",
             current->comm, current->pid,
             (mm->data_vm + npages) << PAGE_SHIFT,
             rlimit(RLIMIT_DATA),
             ignore_rlimit_data ? "" : " or use boot option ignore_rlimit_data");

      if (!ignore_rlimit_data)
          return false;
  }

  return true;
}

First of all, it's a warning message, not an error message.

The code shows why 'or use boot option ignore_rlimit_data' wasn't printed after you added 'ignore_rlimit_data' to the kernel command line. 'ignore_rlimit_data' was accepted by the kernel, and for that reason, the message wasn't printed.

So it seems that adding 'ignore_rlimit_data' to the kernel command line doesn't solve the problem.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum