Optimizing Tomcat's JVM Options for Railo 4.1

Before you change your JVM options, please consider that many developers don't think this is a good idea unless you know what you are doing or are willing to take some risks.  Make sure you test and do further research before applying changes you don't understand.

I'm using Railo and Tomcat, 64-bit version.  My OS is Centos 6 Linux.  Update 11/11/2013: I'm now using Ubuntu 13.04 with the same settings.  Make sure you read the detailed notes below as some of these settings don't work automatically without other configuration details.

/opt/railo/tomcat/bin/setenv.sh

Old configuration:

JAVA_OPTS="-Xms1324m -Xmx1324m -XX:PermSize=340M -XX:MaxPermSize=340M -XX:ParallelGCThreads=8 -XX:+UseParNewGC -javaagent:/opt/railo/lib/railo-inst.jar";

New configuration:

JAVA_OPTS="-Xms256m -Xmx512m -XX:MaxPermSize=128m "; # memory settings

CATALINA_OPTS="-server -Dsun.io.useCanonCaches=false -Djava.library.path=/usr/local/apr/lib -Xms1324m -Xmx1324m -XX:PermSize=340M -XX:MaxPermSize=340M -XX:NewSize=340M -XX:MaxNewSize=340M  -XX:GCTimeRatio=5 -XX:ThreadPriorityPolicy=42 -XX:ParallelGCThreads=4 -XX:+UseParNewGC -XX:MaxGCPauseMillis=50 -XX:+DisableExplicitGC -XX:MaxHeapFreeRatio=70 -XX:MinHeapFreeRatio=40 -XX:+UseStringCache -XX:+OptimizeStringConcat -XX:+UseTLAB -XX:+ScavengeBeforeFullGC -XX:CompileThreshold=1500 -XX:+TieredCompilation -XX:+UseBiasedLocking -Xverify:none -XX:+UseThreadPriorities -XX:+UseLargePages  -XX:LargePageSizeInBytes=2m -XX:+UseFastAccessorMethods -XX:+UseCompressedOops -javaagent:/opt/railo/lib/railo-inst.jar";

If you haven't manually compiled the Tomcat APR library, then remove  -Djava.library.path=/usr/local/apr/lib until you have done that.

This configuration is quite a bit different from the JVM defaults.

What I set out to do was to minimize the time spent in garbage collection since I have some heavy use of memory sometimes when deploying new code and running scheduled tasks.  When Railo runs out of heap space, it can take up to 5 times longer to finish the request due to all the pausing with the default JVM settings.  This configuration greatly reduces the "stop the world" full garbage collection from running as long, and emphasizes cleaning up the young generation more frequently in parallel instead.

Some of the options I don't fully understand, but other people have researched them, and I reviewed each one in the documentation and forums.  They seem worth trying out for a while since my quick load tests and production server results so far seem better.

Some of the options are more performance oriented.  Start-up performance should be improved.  The app should "warm up" faster since it will compile earlier after startup then a default server.  There are also various compiler tweaks to enable some performance optimizations.  I don't have time to verify every options is faster or slower, but the main goal was to reduce garbage collection pauses, and I have.

My app (including this web site) is executing everything a bit faster now.  My home page used to take around 10 milliseconds, but now it is taking 3 milliseconds to process the CFML.

My app only needs around 350m heap and 64m permgen to start-up and run all its features, but over time the memory can expand to over 1gb during development.  I think this happens because of all compilation of new classes and shared memory references.  A server that isn't having new code uploaded to it has lower memory requirement for my app and is more stable.  I've been trying to optimize my app for low memory configurations so it could run on cheaper cloud / vps servers.  Depending on how many sites are added and what features are used, it should be able to run on any server with a minimum of 1gb main memory assuming the web server and database are on the same server with minimal memory usage as well.

Hope this helps someone else trying to tune Railo!

Update: more info about UseLargePages option:

I just found out that using -XX:+UseLargePages on Windows or Linux requires extra steps according to this article: