A while back, we had a customer with a production server exhibiting a curious problem: performance would start well after an application server restart, but over a few days, and sometimes over a single day under heavy load, response times would become longer, until the system would become unresponsive.
Over weeks we analyzed logs files, trying to figure out whether there were unusual patterns in the user requests. We also connected monitoring tools and a profiler to the production system in order to see whether the VM was running out of memory or other resources.
Unfortunately, in spite of a number of false leads, we could not figure out anything conclusive. It was as if the system just became slower for no good reason!
Luckily, after a good deal of additional Googling, we figured out that the culprit was JBoss, or rather, a combination of a particular version of JBoss (JBoss AS 7.1.1.Final) and a particular version of the Java virtual machine (Java 7). 
This version of JBoss forces the
-XX:+TieredCompilation VM option behind the scene, and this option is the direct cause of the problem. It is supposed to improve performance, but turns out to have a very serious adverse effect instead. 
Practically, the issue is solved very easily by setting
Clearly, it was wrong of JBoss to set this option by default, and it seems that this was reverted in subsequent versions of the application server. But many organizations are still using JBoss AS 7.1.1.Final, even though it is more than three years old now, and we hope that this post will be useful to somebody else.
Here are some useful links on this issue: ↩
This option is activated by default with Java 8, and it might be that Java 8 fixes the issues associated with it, but we haven’t tested that. ↩