Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[PARTIALLY SOLVED] Geronimo throws "Too many open files" ...
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
c00l.wave
Apprentice
Apprentice


Joined: 24 Aug 2003
Posts: 245

PostPosted: Thu Apr 01, 2010 7:36 pm    Post subject: [PARTIALLY SOLVED] Geronimo throws "Too many open files Reply with quote

I try to run Apache Geronimo 2.2 J2EE 5 Jetty 7 on a amd64 machine (kernel 2.6.30-gentoo-r5) under Sun JDK 1.6.0_17. If I browse the deployed sites for a few minutes (it doesn't matter if it's a custom deployed webapp or just the server console on a clean installation), Geronimo will reproducibly slow down, crash and start logging "Too many open files" exceptions at a furious speed.

I already tried to increase the max file limit from the default 1024 to 4096 without success. ulimit -n shows the new limit but I still get these exceptions although lsof only lists ~1500 lines for Geronimo's PID (maybe I'm still hitting the 1024 limit, but why?).

Errors I get in the log:

Code:

2010-04-01 20:11:11,403 WARN [org.apache.geronimo.j2ee.deployment.EARConfigBuilder] - Unable to delete 1 files while recursively deleting directory /opt/geronimo-2.2/repository/default/hudson/1270145469792/hudson-1270145469792.war
The first file that could not be deleted was:
 /opt/geronimo-2.2/repository/default/hudson/1270145469792/hudson-1270145469792.war
2010-04-01 20:11:12,235 INFO [org.apache.commons.httpclient.HttpMethodDirector] - I/O exception (java.net.SocketException) caught when processing request: Too many open files
2010-04-01 20:11:12,235 INFO [org.apache.commons.httpclient.HttpMethodDirector] - Retrying request
...
2010-04-01 20:11:12,288 WARN [org.eclipse.jetty.util.log] - EXCEPTION
java.io.IOException: Too many open files
 at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
 at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
 at org.eclipse.jetty.server.nio.SelectChannelConnector$1.acceptChannel(SelectChannelConnector.java:73)
 at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564)
 at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:180)
 at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:121)
 at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:745)
 at org.apache.geronimo.pool.ThreadPool$1.run(ThreadPool.java:214)
 at org.apache.geronimo.pool.ThreadPool$ContextClassLoaderRunnable.run(ThreadPool.java:344)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:619)


(repeated that exception infinitely until I killed Geronimo a few minutes later when I noticed I had it for 6 GB)

Shortly after a reboot (because I thought I may have forgotten to reboot after I changed the security limits), while clicking around in Nexus webapp:

Code:

2010-04-01 21:04:43,280 ERROR [org.sonatype.nexus.timeline.Timeline:default] - Could not search timeline index!
java.io.FileNotFoundException: /mnt/data/geronimo/home/sonatype-work/nexus/timeline/_1vi.cfs (Too many open files)


Geronimo is started by calling its startup.sh script by start-stop-daemon with --chuid geronimo, usually having shell set to /sbin/nologin. If I check ulimit -n it seems correct:

Code:

myserver log # usermod -s /bin/bash geronimo
myserver log # su geronimo
geronimo@myserver /opt/geronimo-2.2/var/log $ ulimit -n
4096
geronimo@myserver /opt/geronimo-2.2/var/log $ exit
myserver log # cat /proc/sys/fs/file-max
756768


I added the following lines to /etc/security/limits.conf and rebooted (I did nothing else; maybe I have to do something more than that?):
Code:

geronimo        soft    nofile 4096
geronimo        hard    nofile 65535


Usually that should have solved the problem? Has somebody else had the same problem? Any ideas what I should try next?

Edit: I just saw /proc/.../limits:
Code:

myserver log # cat /proc/15966/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            ms
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             73728                73728                processes
Max open files            1024                 1024                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       73728                73728                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us


It's still 1024, not my configured and (by ulimit) reported 4096 file limit. Why didn't it apply?
_________________
nohup nice -n -20 cp /dev/urandom /dev/null &


Last edited by c00l.wave on Thu Apr 01, 2010 8:34 pm; edited 1 time in total
Back to top
View user's profile Send private message
c00l.wave
Apprentice
Apprentice


Joined: 24 Aug 2003
Posts: 245

PostPosted: Thu Apr 01, 2010 8:32 pm    Post subject: Reply with quote

Okay, I should have seen that limits table earlier. I was able to find an old bug report mentioning that start-stop-daemon is leaving the limits of its surrounding shell unchanged (at 1024 or whatever root is set to). A workaround is to call ulimit -n before running start-stop-daemon.

I see two options for my init-script:

1) save old ulimits -n value, set the new one, call start-stop-daemon, restore old value (or is that unnecessary?)
2) use su -c instead of start-stop-daemon (in case that really helps)

For option 2, I would have to set a real shell for my user 'geronimo' instead of using /sbin/nologin. Are there any other options I forgot or is that what I'm left with? Grepping the server's init scripts I could only see asterisk changing ulimit parameters (without restoring the original values?). I don't like either option but would go with the first one unless there's a different way not involving setting a full shell for that user.
_________________
nohup nice -n -20 cp /dev/urandom /dev/null &
Back to top
View user's profile Send private message
root84
n00b
n00b


Joined: 09 May 2010
Posts: 1

PostPosted: Sun May 09, 2010 2:17 am    Post subject: Reply with quote

I just had a similar problem with Tomcat 6. I deployed Hudson on my Tomcat server and started a rather huge Maven2 build. Eventually Hudson would die with the following error message:

Code:
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to create assembly: Failed to retrieve OS environment variables. Reason: Cannot run program "env": java.io.IOException: error=24, Too many open files
[INFO] ------------------------------------------------------------------------


So I increased the nofiles value in /etc/security/limits.conf, but Tomcat seemed to ignore it. Finally my solution was to edit the /etc/init.d/tomcat-6 init script and insert "eval" before every "start-stop-daemon" command. That seemed to do the trick for me.

I already wrote a few other init-scripts for Java applications some time ago, and my observation was that start-stop-daemon never worked properly without using eval...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum