ThreadLocal + Thread Pool = bad idea (or: dealing with an apparent Glassfish memory leak)

One of the not-so-great things about developing Java web applications is the fact that, after a few redeployments, sooner or later the web container JVM needs to be restarted due to Out Of Memory errors. This has been true in every combination of development environment and web server I have used so far, and until last week Netbeans 6.8 + Glassfish 3.0.1 was no exception.

The cause of Out Of Memory errors in web containers are many, but the most common of all is a ClassLoader leak. Each web application is loaded under its own ClassLoader and once an application is undeployed all references to its classes should be cleared, which makes them (and all objects referenced by them) eligible for garbage collection.

The problem is sometimes (I would say most of the time) some reference fails to be cleared when a web application is undeployed. When that happens the ClassLoader associated with the application cannot be garbage colledted and a number of objects referenced by its loaded classes remain in memory. After some redeployments these objects clutter the heap space causing a OOM exception.

After a few months of development I got fed up with Glassfish 3.0.1 getting slower and slower and finally crashing after a few redeployments. I used jvisualvm to analyze the heap contents and found out that as I expected the used memory  got bigger after each redeployment.

After some analysis it became clear that most of the retained heap was being held in place by a reference coming from the class com.sun.enterprise.security.authorize.HandlerData , which in turn was being refrred to by a ThreadLocal instance.

ThreadLocal variables are, as the name suggests, variables bound to the Thread scope: each thread has a separate copy of the variable. This kind of variable is very useful to propagate context information between application layers.

The problem in this case seems to be that during application deployment some code in Glassfish is creating a ThreadLocal variable holding a reference to an instance of com.sun.enterprise.security.authorize.HandlerData, which in turn seems to hold a reference to a lot of other objects (mostly related to proxy classes generated during EJB and CDI deployment).  Glassfish doesn’t seem to clean this reference after the deployment finishes, which wouldn’t be much of a problem if the thread that deploys the application terminates.

Unfortunately the application deployment thread never dies, because it is not created solely for the purpose of application deployment. It is instead taken from the Glassfish thread pool and is returned to the pool once it finishes the deployment task. That means the ThreadLocal reference never gets cleaned and over time causes the heap to overflow.

Full article https://weblogs.java.net/blog/jjviana/archive/2010/06/09/dealing-glassfish-301-memory-leak-or-threadlocal-thread-pool-bad-ide

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s