Optimizing Tomcat's JVM Options for Railo 4.1

Tue, Jul 02, 2013 at 12:15AM

Before you change your JVM options, please consider that many developers don't think this is a good idea unless you know what you are doing or are willing to take some risks.  Make sure you test and do further research before applying changes you don't understand.

I'm using Railo and Tomcat, 64-bit version.  My OS is Centos 6 Linux.  Update 11/11/2013: I'm now using Ubuntu 13.04 with the same settings.  Make sure you read the detailed notes below as some of these settings don't work automatically without other configuration details.


Old configuration:

JAVA_OPTS="-Xms1324m -Xmx1324m -XX:PermSize=340M -XX:MaxPermSize=340M -XX:ParallelGCThreads=8 -XX:+UseParNewGC -javaagent:/opt/railo/lib/railo-inst.jar";

New configuration:

JAVA_OPTS="-Xms256m -Xmx512m -XX:MaxPermSize=128m "; # memory settings

CATALINA_OPTS="-server -Dsun.io.useCanonCaches=false -Djava.library.path=/usr/local/apr/lib -Xms1324m -Xmx1324m -XX:PermSize=340M -XX:MaxPermSize=340M -XX:NewSize=340M -XX:MaxNewSize=340M  -XX:GCTimeRatio=5 -XX:ThreadPriorityPolicy=42 -XX:ParallelGCThreads=4 -XX:+UseParNewGC -XX:MaxGCPauseMillis=50 -XX:+DisableExplicitGC -XX:MaxHeapFreeRatio=70 -XX:MinHeapFreeRatio=40 -XX:+UseStringCache -XX:+OptimizeStringConcat -XX:+UseTLAB -XX:+ScavengeBeforeFullGC -XX:CompileThreshold=1500 -XX:+TieredCompilation -XX:+UseBiasedLocking -Xverify:none -XX:+UseThreadPriorities -XX:+UseLargePages  -XX:LargePageSizeInBytes=2m -XX:+UseFastAccessorMethods -XX:+UseCompressedOops -javaagent:/opt/railo/lib/railo-inst.jar";

If you haven't manually compiled the Tomcat APR library, then remove  -Djava.library.path=/usr/local/apr/lib until you have done that.

This configuration is quite a bit different from the JVM defaults.

I think "-XX:MaxHeapFreeRatio=70 -XX:MinHeapFreeRatio=40" forces more heap to be available after each full GC, so there are less full garbage collections happening now.
I think if I used the CMS garbage collector, it would avoid pausing too, but the useParNewGC one has been more stable & faster under load for me.

What I set out to do was to minimize the time spent in garbage collection since I have some heavy use of memory sometimes when deploying new code and running scheduled tasks.  When Railo runs out of heap space, it can take up to 5 times longer to finish the request due to all the pausing with the default JVM settings.  This configuration greatly reduces the "stop the world" full garbage collection from running as long, and emphasizes cleaning up the young generation more frequently in parallel instead.

Some of the options I don't fully understand, but other people have researched them, and I reviewed each one in the documentation and forums.  They seem worth trying out for a while since my quick load tests and production server results so far seem better.

Some of the options are more performance oriented.  Start-up performance should be improved.  The app should "warm up" faster since it will compile earlier after startup then a default server.  There are also various compiler tweaks to enable some performance optimizations.  I don't have time to verify every options is faster or slower, but the main goal was to reduce garbage collection pauses, and I have.

My app (including this web site) is executing everything a bit faster now.  My home page used to take around 10 milliseconds, but now it is taking 3 milliseconds to process the CFML.

My app only needs around 350m heap and 64m permgen to start-up and run all its features, but over time the memory can expand to over 1gb during development.  I think this happens because of all compilation of new classes and shared memory references.  A server that isn't having new code uploaded to it has lower memory requirement for my app and is more stable.  I've been trying to optimize my app for low memory configurations so it could run on cheaper cloud / vps servers.  Depending on how many sites are added and what features are used, it should be able to run on any server with a minimum of 1gb main memory assuming the web server and database are on the same server with minimal memory usage as well.

Hope this helps someone else trying to tune Railo!

Update: more info about UseLargePages option:

I just found out that using -XX:+UseLargePages on Windows or Linux requires extra steps according to this article:

I was getting this error when restarting tomcat:
Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve shared memory
I just had to add this line to /etc/sysctl.conf to allow around 600mb of memory reserved for huge pages
On a system with more "extra" ram, you can raise this.  It does warn that if you make it too high, it will eventually switch back to regular pages or cause memory issues with other software.
To apply that change without rebooting, you can do:
echo 300 > /proc/sys/vm/nr_hugepages
The article explains how this can help performance of memory-intense apps. Instead of default 4k memory pages, it will jump up to around 2mb on modern cpus for TLBs, which, I think the layman terms would be: It makes one of the CPU/memory caching layers work faster.  It also forces the memory to be reserved for your specific app (i.e. Railo) and never swap those pages to the swap disk.

Bookmark & Share