Finding memory problems

Finding memory problems:

When performance testing a java application and want to identify memory problems, below steps will help:

1.   Don’t call it “memory leak” until you are 100% sure-make your program manager jump up and down. (A better term “unaccountable heap usage”, etc.)

2.   There is probably some “unaccountable heap size” from each release, there is no need and no time to look for small ones. Concentrate on the large ones which could cause problem down the road.

3.   There are some caches which is not leakage-watch out for their growth in size. Also check for some session data, which might be cleaned up after user logoff.

4.   Run 72-90 hour stability test under load for each release, monitoring java heap size. Check java heap GC frequency and old gen size. (cat “Full GC” myServer.log |awk ‘/Full GC/ {getnextline; getnextline;getnextline;print $0}’ | awk ‘{print $4}’)

5.   Enable GC Logging in JAVA_OPTIONS by adding appropriate flags:

"-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:~/$HOSTNAME/logs/gc-log-file.log"

               

1)-XX:+PrintGCDetails we activate the “detailed” GC logging mode which differs depending on the GC algorithm used

2)-XX:+PrintGCTimeStamps timestamp reflecting the real time passed in seconds since JVM start GC

3) -verbose:gc activates the “simple” GC logging mode         

4) -Xloggc we can specify location of an output file. By default it is at stdout.

6.   When running stability test for 90 hours, keep all the counters running in the background-(such as, sar –A –o mylogfile 120 100000 >/dev/null 2>&1 &, while [ 1 ];do netstat –s |egrep “inpack|outpack” >> myNetstatlog;sleep 600 done & ) You will need to find out why things go wrong in the middle of the night.

7.   Check all the logs to make sure there are no exceptions in the logs. If there are exceptions, memory can grow. It should be a clean run.

8.   If there is a problem, such as out of memory exception, use your favorite memory profiler-JProbe, JProfiler ..etc.

9.   Look at the source code, see if this is actual “leakage”. If you are not sure, talk to the developer, provide a screen shot.

10.            In profiler, use the filter to look for your application classes first, before you look at the Java classes or 3^rd party classes. Usually it is our code problem, not Java or middleware.