A "quick" microbenchmark
The other day I decided to do some microbenchmarking across several different languages. I used an array of 1 M integers and applied the quicksort sorting algorithm as the test code and also included the native language sort routines where applicable.
Each respective test will run the sort algorithm 5 times (some only 3 because they take so long already it doesn't matter) and the minimum execution time is printed. I ran each one a few times and took an eyeball average of those runs. I am not that interested in a +/- of a few milliseconds when sorting 1 M integers. I am way more interested in orders of magnitude.
|Language||Version||Time in milliseconds|
|1||C++||MS 16.00.30319.01 for 80x86||75|
|2||C# Array.Sort()||.NET 3.5||100|
|4||Groovy api (aka java)||1.8.5||110|
|10||Ruby api array.sort!||1.9.2p290||250|
|14||Node.js (v8) api sort()||0.6.7||480|
|16||Python api sort()||3.1.3||814|
|17||PHP api sort()||5.3.8||1441|
Things I will take away from this exercise
It was interesting to see how easy some of the features were to find in some languages compared to others. Command line parameters, random numbers, benchmarking code etc. Lets take a look at command line parameters in each language. They are all similar (some variation of an arg[sv] array) but yet different (array offset, parsing, brackets, semi-colon). Its these subtle differences that make a lot of language context switching difficult.
C++ int len = atoi(argv); C# var len = int.Parse(args); java: int len = Integer.parseInt(args); scala: val len = Integer.parseInt(args(0)) php: $len = $argv; node.js (coffeescript): len = process.argv groovy: len = args.toInteger() python: alen = int(sys.argv) # len is a reserved word ruby: len = ARGV.to_i
A small but effective change included changing
pivot = arr[(left+right) / 2] to
pivot = arr[(left+right) >> 1]. In some languages it generated a noticeable increase in performance. As well it prevented a necessary cast/floor to an int preventing an array lookup of arr[3.5].
Pythons top google links are tied to v2 rather than v3. Using the old print statement rather than the print() function is pretty frustrating as you view the examples from google and the code seems correct. The error message was not at all helpful either. I have seen this in other languages/frameworks as well.
Then we have the browser wars, aka the js engine war. Chrome posted the best time for the quicksort. For some reason I was expecting some overhead compared to node.js but the in browser version was pretty much identical in both test cases. IE was the most consistent and by far had the best time using the Array API sort(). Firefox was really slow on the API call... no idea why. It was slow enough that I actually got the warning saying the script was running too long. During the algorithm Firefox was "Not Responding" as well. I didn't bother running the code in older versions as that wasn't the goal of this exercise.
Consective runs in IE and FireFox produced consistent results. However, this was not true for Chrome. Chrome's performance decreased by about double for the second run and a little more after that to what seemed to be a ceiling. It probably has to do with GC since on a page refresh the time returned back to the fast time.
Groovy gets to piggy back off of the JVM for API calls.
You can view the code for each implementation on github.
Although I did have the idea before seeing this, the following blog post provided some good information http://stronglytypedblog.blogspot.com/2009/07/java-vs-scala-vs-groovy-performance.html
You can see the results with PHP 5.4 ... A "quick" microbenchmark update PHP 5.4