Thursday, January 16, 2014

Tomcat, Spring and memory leaks when undeploying or redeploying an web application

In this post I'll talk about a new kind of memory leak in Spring applications involving transaction management and initializing beans.

About memory leaks when undeploying a web application

While developing a web application with Spring and Tomcat I frequently run out of heap space. The web is full of stories about this. There is Aspirin to suppress the symptoms [1] and there are wise people who know why this is happening [2].

The short (and scientifically not 100% rigorous) story is that the web application started either a thread or registered an object with a global registry which it then forgot to stop or unload.

Tomcat 7 added [2] not only detection for many of these cases, but in some instances it also can remedy the situation by forcefully unloading JDBC drivers and terminating rogue threads. 

I argue that it is ok if the application server cleans up after an application - in a way, it is not different than garbage collection (which is now socially accepted). However the sad fact is that it doesn't (yet) work all the time which make Aspirins [1] against the headache necessary and just postpone the unavoidable, namely stopping and restarting the JVM.

A new type of memory leak

The JDBC driver is parked with the application server, any thread pools are being closed when the application shuts down, quartz schedulers are also being cleaned up... and yet there is an entry like this every time the application stops:
SEVERE: The web application [] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Transactional resources] ... but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Apparently a Spring-managed thread local variable is not being cleaned up. After a binary search over all possible components it turned out to be this particular instance:

@Component
public class SomeComponent{

@Autowired
SomeSpringDataJpaDao dao;

@PostConstruct
public void setup(){

      dao.doSomething();

}
}

What had happened here? Spring was instantiating SomeComponent, autowiring a Spring-data-jpa DAO into it and then calling the setup method. This runs a query against JPA without a transaction. Anyone who has worked enough with databases knows this means trouble. In my defence, SomeComponent participates (almost) always in a transaction which is driven by a ServletRequest. Ironically the particular case runs with a timer and thus does not go through the web layer which would have spanned the transaction.

The solution is (almost) fortunately quite trivial: add an @Transactional annotation to either the component or the setup() method.

Note that Spring-data-jpa 1.4.3 seems to have fixed a related bug in LockModeRepositoryPostProcessor.invoke() which used to set the "Transactionl resources" thread local variable but never cleared it.

Resources

[1] Dealing with “java.lang.OutOfMemoryError: PermGen space” error

http://stackoverflow.com/questions/88235/dealing-with-java-lang-outofmemoryerror-permgen-space-error

[2] Memory Leak Protection in Tomcat 7
http://java.dzone.com/articles/memory-leak-protection-tomcat

No comments:

Post a Comment