JVM Performance, Part II - JVM Bazaar

As promised, I have results from running some of the tests in other JVMs. The most interesting results were from the Map test and the Method execution test, so I have re-run those on various JVMs and environments.

The first result comes from running with Oracle (Sun) JDK 1.6.23 on the same Windows XP machine. Next I ran on Oracle (Sun) JDK 1.5.5, and then Oracle JRockit 1.6. I then ran the tests on another machine, an old 4 cpu 8 core AMD machine running Linux. The results are very surprising, each different machine and environment seemed to have very different behavior.

Note: These tests are not in any way attempting to compare Windows vs Linux performance, or Oracle Sun JDK vs Oracle JRockit, completely different hardware, JVMs and environments are used.  The goal is to see how different operations compare on the same JVM, and how one JVM and environment can differ with another.  Some of the Windows results are faster most likely because the machine is newer, and a faster CPU, some of the Linux results are faster possibly because the machine has 8 CPUs instead of 2, so can process garbage collection on other threads.

The test uses a Map of size 100 testing instantiation of the Map, 100 puts and 100 gets. The average result is included. The %DIF is between the Map and the Hashtable results for that same JVM.

Map Operation Performance Comparison

Map	WXP-JDK1.6.7	%DIF	WXP-JDK1.6.23	%DIF	WXP-JDK1.5.5	%DIF
Hashtable	357229	0%	379673	0%	219961	0%
HashMap	274587	-30%	403501	+6.2%	238916	+8.6%
LinkedHashMap	269840	-32%	356207	-6.5%	228787	+4.0%
IdentityHashMap	110801	-222%	121691	-211%	60670	-262%
ConcurrentHashMap	119068	-200%	183159	-107%	109912	-100%
HashSet	281960	-26%	407709	+7.3%	228936	4.0%

Map	LX-JDK1.6.20	%DIF	LX-JDK1.5.5	%DIF	LX-JRK1.6.5	%DIF
Hashtable	343250	0%	307227	0%	474378	0%
HashMap	496258	+44%	375805	+22%	642267	+35%
LinkedHashMap	476535	+38%	417273	+35%	595331	+25%
IdentityHashMap	552437	+60%	478350	+55%	716368	+51%
ConcurrentHashMap	293529	-16%	299675	-2.5%	331730	-43%
HashSet	451615	+31%	381843	+24%	633824	+33%

The first good thing to notice is that for the same machine, the results get better with newer JVM's.  That is very nice, and something that most Java developers are familiar with, each new JVM version having better performance than the previous.

The second things to notice is that the Map types have huge variations based on the JVM.  HashMap was faster than Hashtable in 1.6.23 and even 1.5.5, but somehow slower in 1.6.7.  I found this very perplexing, but no matter how many times, or in which order I ran the tests the results were the same.  I also compared the code between the releases, and did not find any code changes that could account for the change in performance.  I assume it has to do with how the JVM manages memory and synchronized methods.  One would think that the code for Hashtable, HashMap, and IdentityHashMap was all the same, since the functionality is the essentially identical, but in fact they share none of the same code, and use quite different data structures.  I assume this accounts for the difference in performance under the different JVMs, so that none of them is inferior, they are just different.

Note that these tests are still single threaded tests.  So although they are comparing the raw performance difference between Hashtable and HashMap, the far bigger issue is the affect Hashtable's synchronized methods have on concurrency.  This is also true for ConcurrentHashMap, altough its raw throughput seems to be worse, it concurrency is far better, and in a mutli-threaded environment it will fair much better.  I will hopefully explore this in another blog post.

Method Execution Performance Comparison

Execution type	WXP-JDK1.6.7	%DIF	WXP-JDK1.6.23	%DIF	WXP-JDK1.5.5	%DIF
Normal	25533130	0%	29608720	0%	18557941	0%
synchronized	13383707	-93%	14373780	-105%	1254183	-1379%
Block synchronized	8244087	-203%	8812159	-235%	1253624	-1380%
final	26873873	+6.1%	30812911	+4.0%	18853714	+1.5%
In-lined	26816109	+5.2%	30815433	+4.0%	19734119	+6.3%
volatile	1503727	-1539%	3290550	-799%	1448561	-1181%
Reflection	159069	-1564%	145585	-20237%	82713	-22336%

Execution type	LX-JDK1.5.5	%DIF	LX-JDK1.6.20	%DIF	LX-JRK1.6.5	%DIF
Normal	4666753	0%	4840361	0%	4257995	0%
synchronized	2250871	-107%	4413458	-9.6%	3465398	-22%
Block synchronized	2252474	-107%	4410721	-9.7%	3672440	-15%
final	4666384	0.0%	4911130	+1.4%	4392700	+3.1%
In-lined	4668538	0.0%	4874539	0.7%	4042412	-5.3%
volatile	3236491	-44%	3231105	-49%	3032121	-40%
Reflection	2645911	-76%	2568412	-88%	132232	-3120%

The first good thing to note again is that newer JVM have for the most part better performance.  This is one of the great aspects of developing using the Java platform.

The second thing to notice is once again the behavior for the various JVMs is very different.  They all seem to be consistent in that synchronized methods, volatile and reflection are slower, but the degree of the difference is pretty major.  Synchronized methods seem to be much better in JDK 1.6 vs 1.5, but in the latest Linux JVM almost have no overhead at all.  Reflection also improved a lot from 1.5 to 1.6, but again the Linux overhead is less than a tenth of the Windows JVM results (except for JRockit).  Volatile has similar differences.  This could have more to do with the hardware than the OS or JVM, as the Linux machine has 8 CPUs, so may be doing some memory managment using its other CPUs (but this does not explain the JRockit result...).

Summary

So what have we learned?  I think the most important thing is that different JVMs and environment can have very different behavior, so it is important to test in the environment that you will go into production in.  If you do not have that luxury, then it is important to test in a variety of different environments to ensure you are not making trade offs in one environment that could hurt you in another.

I would not get too obsessed with worrying about how your applications performance may differ in different environments.  By in large, the main performance optimizations of reducing CPU usages, reducing message sends, reducing memory and garbage, improving concurrency will improve your application's performance not matter what the environment is.