We encountered the dreaded

java.lang.OutOfMemoryError: PermGen spaceerror

What does it means

– Permgen Space is the space where the class files are all loaded for the first time and there after they are loaded or referenced from there. This error means there is a hell lot of duplicate classes being loaded and not all are being unloaded. These duplicate class instances are cluttering the permgen space and resulting in this error. One expects only only class instance of a java file. This is basically amemory leak.

also read:

• Java Tutorials
• Java EE Tutorials
• Design Patterns Tutorials
• Java File IO Tutorials

Temporary Solution

–
[code]
‘start tag’ jvmarg value="-XX:+UseConcMarkSweepGC" ‘end tag’
‘start tag’ jvmarg value="-XX:+CMSPermGenSweepingEnabled" ‘end tag’
‘start tag’ jvmarg value="-XX:+CMSClassUnloadingEnabled" ‘end tag’
[/code]
– These allow perm gen sweeping and allows class unloading, I guess it works really well when there is no memory leak but low perm gen space allocated, so when there is a space deficiency it automatically unloads few classes and then reloads them when needs. Yes there will be a performance issue but am sure not many will notice it.
[code]’start tag’ jvmarg value="-XX:PermSize=256M" ‘end tag’
‘start tag’ jvmarg value="-XX:MaxPermSize=512M" ‘end tag'[/code]
– you can keep increasing this value between 128 – 512 or more but this will only delay the time after which the error is thrown.

Tools

–
Few tools that I used for analysing this issue

1. MAT – Eclipse Memory Analyser Tool (Also recommended – has some great features to zero in on the suspect / culprit)
2. IBM Heap Analyser
3. JConsole
4. Yourkit – trial version for 14 days (you better solve your issue within 14 days :)) [Highly recommended – it allows monitoring and a hell lot of features]
5. JMAP

Analysis and Solution

–
There are two ways of going about this, you would never programmatically create duplicate classes and load them into the permgen, so the only few options are

• First Way
  Are you using different classloaders or custom classloaders to load few classes? If yes there are cases that same classes are loaded often because of different classloader instances. Check you own code.
• Second Way
  Analyse the various frameworks that you are using, check in particular if you in your code or these frameworks use dynamic proxying, which means creating proxy classes at run time for loosely coupling the classes.

PS: A tip – check if any framework uses CGLIB, in my case it was this.

Approach 1

1. I used jmap on the unix machine and got the heap reproduced in a file called heap.bin.
2. Then open the heap.bin using the free tool IBM Heap Analyser (ha36.jar Let me know if you need it) or MAT or Yourkit.
3. Now you can view different details of the heap, like duplicate class, which objects is being created in large number or objects based on size etc.
4. Now go to class instances and search for duplicate classes or sort the classes by name and then go through the list.
5. You should be able to see at some point few duplicate classes being loaded and they would have different extension

   example-
   [code]
   Class ABC you can see Class ABC, Class ABC$$EnchancerByCGLIB$$9a6119f_32$$FastClassByCGLIB$$123456, Class ABC$$EnchancerByCGLIB$$9a6119f_33$$FastClassByCGLIB$$123453, Class ABC$$EnchancerByCGLIB$$9a6119f_34$$FastClassByCGLIB$$123454, Class ABC$$EnchancerByCGLIB$$9a6119f_35$$FastClassByCGLIB$$123455 etc etc
   [/code]

6. Now you know this class is being created by CGLIB / dynamic proxying and because they are all different classes all together they will be surely kept in the perm gen adding to the clutter and the error.

Approach 2

1. Yourkit attaches to the live running application – the process is given in their site.
2. Monitor the live application and keep checking the permgen and number of classes loaded and unloaded – if the classes loaded keeps increasing even after a threshold we know we are in trouble (jconsole also can be used to see the classes loaded and unloaded)
3. In your kit keep checking the class instances and search for duplicate classes or sort the classes by name and then go through the list.
4. You should be able to see at some point few duplicate classes being loaded and they would have different extension
   [code]
   example- Class ABC you can see Class ABC, Class ABC$$EnchancerByCGLIB$$9a6119f_32$$FastClassByCGLIB$$123456, Class ABC$$EnchancerByCGLIB$$9a6119f_33$$FastClassByCGLIB$$123453, Class ABC$$EnchancerByCGLIB$$9a6119f_34$$FastClassByCGLIB$$123454, Class ABC$$EnchancerByCGLIB$$9a6119f_35$$FastClassByCGLIB$$123455 etc etc</li>
   [/code]

5. Now you know this class is being created by CGLIB / dynamic proxying and because they are all different classes all together they will be surely kept in the perm gen adding to the clutter and the error.
6. This list will only keep increasing as time flies / more requests are being processed.

Solution-

1. Now check which version of the framework you are using and check any recorded bug against it – I was using cglib-nodep-2.1_3.jar and it seems to have a weak reference to all the classes that it creates causing this clutter. I then had to upgrade Spring to spring 2.5.6.SEC01, Hibernate and CGLIB to their production stable releases.
2. After this fix / upgrade I took regular imprints / monitored it live and found there were no more duplicate classes being created.
   • Another way is to take regular heap imprints and you would see that there is pattern say heap1 = 10 MB, heap2 = 50 MB, heap3 = 100 MB, heap4 = 80 MB, Heap5 ~= 80 MB.
     In the case of the error prone application / memory leak plagued application the pattern would be heap1 = 10 MB, heap2 = 50 MB, heap3 = 100 MB, heap4 = 120 MB, Heap5 = 140 MB….

PS: Jconsole can be used to see the heap growth, perm gen growth, class loading, class loading etc at run time. All the graphs can have peaks and downs but must even out in due course, if they keep going north then you know for sure you are going to have trouble.

Few More

[code]
jmap -permstat 18254 > permstats_2.txt
jmap -histo:live 18254 > histo_live.txt
jmap -histo 18254 > histo.txt
-XX:-TraceClassLoading Trace loading of classes.
-XX:-TraceClassLoadingPreorder Trace all classes loaded in order referenced (not loaded).
[/code]
(Introduced in 1.4.2.)
[code]
-XX:-TraceClassResolution Trace constant pool resolutions. (Introduced in 1.4.2.)
-XX:-TraceClassUnloading Trace unloading of classes.
[/code]

We encountered the dreaded

java.lang.OutOfMemoryError: Java heap space error.

How often do we hit this error and look in bewilderment.

What does it means

Means there is a hell lot of objects being created and not all are being marked for GC. These stray objects are cluttering the heap space and resulting in this error. This is basically a memory leak.

also read:

• Java Tutorials
• Java EE Tutorials
• Design Patterns Tutorials
• Java File IO Tutorials

Temporary Solution

start tag jvmarg value=”-Xmx892M” end tag- you can keep increasing this value between 256 – 1024 or more but this will only delay the time after which the error is thrown.

Tools

Few tools that I used for analysing this issue

1. MAT – Eclipse Memory Analyser Tool (Also recommended – has some great features to zero in on the suspect / culprit)
2. IBM Heap Analyser
3. JConsole
4. Yourkit – trial version for 14 days (you better solve your issue within 14 days :)) [Highly recommended – it allows monitoring and a hell lot of features]
5. JMAP

Analysis and Solution

1. I used jmap on the unix machine and got the heap reproduced in a file called heap.bin.
2. Then open the heap.bin using the free tool IBM Heap Analyser (ha36.jar). Let me know if you need it.
3. Now you can view different details of the heap, like duplicate class, which objects is being created in large number or objects based on size etc.
4. Now there is a option to get the leak suspects, it would list a few leak suspects – it would surely list few open source frameworks like Active MQ, Hibernate etc which means you will have to upgrade these to the latest production release from the framework providers.
5. Concentrate on any leak suspect either on your classes or your packages or on any java’s Class used a lot in your class.
6. I was able to find a wrongly declared and used variable related to threadpoolexecutor which was causing a lot of hanging objects. (key is to find the link between a leak suspect and the way it is related to your code)
7. After fixing this I took regular heap imprints and analysed them and found that the leak suspect was missing.
8. After this fix my application starting running for almost double the time but I soon hit the dreaded java.lang.OutOfMemoryError: PermGen space, about which I shall write in the next post.
• Another way of seeing the success of your fix is to take regular heap imprints and you would see that there is pattern say
  [code]heap1 = 10 MB, heap2 = 50 MB, heap3 = 100 MB, heap4 = 80 MB, Heap5 ~= 80 MB.[/code]
  In the case of the error prone application / memory leak plagued application the pattern would be
  [code] heap1 = 10 MB, heap2 = 50 MB, heap3 = 100 MB, heap4 = 120 MB, Heap5 = 140 MB….[/code]

PS: Jconsole can be used to see the heap growth, perm gen growth, class loading, class loading etc at run time. All the graphs can have peaks and downs but must even out in due course, if they keep going north then you know for sure you are going to have trouble.

Few more

[code]
jmap -permstat 12345 > permstats_2.txt
jmap -histo:live 12345> histo_live.txt
jmap -histo 12345 > histo.txt
-XX:-TraceClassLoading Trace loading of classes.
-XX:-TraceClassLoadingPreorder Trace all classes loaded in order referenced (not loaded). (Introduced in 1.4.2.)
-XX:-TraceClassResolution Trace constant pool resolutions. (Introduced in 1.4.2.)
-XX:-TraceClassUnloading Trace unloading of classes.
[/code]