Incredible Railo application start-up performance boost using ObjectLoad/ObjectSave to persist shared memory scopes

  Follow me: Follow Bruce Kirkpatrick by email subscription Bruce Kirkpatrick on Twitter Bruce Kirkpatrick on Facebook
Mon, May 20, 2013 at 12:35AM

This article describes in technical detail how I made my app start-up 10 times faster and persist/restore memory based session data using Railo 4.

Storing scope data on disk

Today, I was able to write a simple shell script that calls a script via a secure URL that dumps the Railo application and session scope shared memory to disk using ObjectSave() with an exclusive lock wrapped around it.

The script then restarts the Railo process using "service railo_ctl restart" like normal.

Once Railo has finished restarting, the script calls a URL of one of the web sites to get the Railo app started if no other request has already been made.

Restoring the dumped scope data

On first load of onApplicationStart(), I now detect the dumped object files and reload them using ObjectLoad(). If the dumps aren't found, fail to load or my manual cache clearing flag is set, the objects are re-built the normal way. To ensure the dumps are binary compatible, I put the value of "server.railo.version" in the filename, so it will automatically ignore the dumped objects when Railo is updated to a new version.  I also delete the persisted objects on disk once they are loaded so that a Railo hang/crash or system poweroff won't cause the app to load an out of date version on the next start-up.  This means if I forget to use my custom script to restart Railo, there won't be any unexpected side-effects. 

High performance results

A single ObjectLoad calls is vastly superior in performance compared to rebuilding objects from the multiple database queries, disk operations and CPU calculations on the fly. ObjectLoad is also faster because hard drives are faster with sequential I/O so I put all my data into a few big objects rather then many small ones.

I found that it loads all my 100+ production web sites in just a few seconds instead of the 15 to 25 seconds it usually takes during start-up once CFML begins processing. The start-up performance improvement is INCREDIBLE. The actual batch process script is able to dump, restart railo and finish a successful web site request in just 15 seconds now.  Most of that time is actually Tomcat/Railo start-up time.  My app start-up time is around 4 seconds. Because all the cached data for all the web sites is fully loaded at that point, all 100+ sites load quickly for anyone visiting. Before this optimization, there used to be a delay for a few seconds on the first request.

Railo built-in scope persistence someday maybe?

I know Mura and many other CFML apps have very slow start-up times because so many objects are put into the application scope during startup. Persisting native shared memory when restarting Railo would make them start-up so much faster. It seems like Railo should have the ability to dump and reload its shared memory scopes like this as an optional built in feature someday. I can't think of any problems with doing this. In my app, dumping all session and application memory only takes up 16mb on disk, which is pretty small. Even if it was significantly more, the performance would still be very good.

Handling OS and Railo restarts via bash scripting

I also made a replacement init.d bash script for railo_ctl that calls the original after performing the memory scope dump.  I renamed the original to railo_ctl_original and left it disabled.

If the OS kernel is updated and you need to reboot the OS, the railo_ctl script isn't enough.  You also need to put wrapper functions on the "reboot" and "poweroff" commands so that the memory dump is performed before system services start shutting down.  I added these by modifying /root/.bashrc with source /root/wrapper-functions.lib and defining the wrapper functions in the bash script.  

This will ensure users can continue to use the app preserving their session data when it comes back up a minute or two later.  

Reduced downtime and minimal impact for users when performing maintenance

It may be better to have a cluster of Railo servers and use some kind of session replication to avoid downtime entirely, but I like to save money and complexity by using a single server for my sites.  Because I have reduced the deployment and start-up time so much, any maintenance related downtime has been greatly reduced.

Getting access to session scope data for all users

It took a little extra work to figure out how to persist sessions to disk. I had to dig in some of Railo's internals to find the session data.

In https://github.com/getrailo/railo/blob/master/railo-java/railo-core/src/railo/runtime/type/scope/ScopeContext.java
I found these two methods:
public Struct getAllSessionScopes(PageContext pc)
public Struct getAllApplicationScopes()

That led me to want to figure out how to access those functions so I could persist the session memory storage to disk manually. I dug through the getpagecontext() metadata by dumping various function results and found that there is a way to grab all session and application scopes at once:

<cfscript>
c=getpagecontext().getCFMLFactory().getScopeContext();
writedump(c.getAllApplicationScopes());
writedump(c.getAllSessionScopes(getpagecontext()));
</cfscript>

However, I found that getAllSessionScopes() actually returns a copy of the session data, so it is seemingly impossible to globally modify the session data for other users in Railo 4.0.

This didn't stop me though. I decided to dumped the copy of the session data anyway and load it into the application scope (instead of the session scope) during start-up. Then when a user comes along with a matching session.sessionid, I copy the session data from the application scope into their session scope and remove it from the application scope. I added a scheduled task to manually handle session expiration based on these variables: session.lastvisit and application.sessiontimeout so that the application scope memory is not permanently used. This means it should use nearly the same amount of memory efficient as the normal session memory storage. Now my app can preserve all the session data after Railo restarts using the native memory storage without modifying the way you access session data in an existing app. There was no need to wrap session with CFCs or other stuff, it's just plain session.myvar in my app still. It would be more convenient if there was a method that returns a reference to the original session struct, but my solution is good enough for now.

I initially saw the session dump file size getting smaller when restarting each time.  This allowed me to realize that some of the sessions that were in application scope were being lost.  I had to combine the application scope copy of session data with the current session data using structappend() to avoid data loss.

It was quick to implement this.

I was able to write and launch this code for Railo shared memory persistence and launch it to the production server in about a day's effort because the code to do this is very short. I think maybe less then 50 lines were added, yet they yield a great impact on start-up performance.

Everything comes up again 10 times faster with no lost session data. Win/win!

This code is part my application, Jetendo CMS, and it will released as an open source project when it's ready for the public.


Bookmark & Share



Popular tags on this blog

Performance |