Readers obviously jump to conclusions based on box plots and summary statistics -- when they really need to look at the individual tasks and look at the program source code, because there's so much variation.
I think there would need to be incredibly tight constraints on minimum program run-time, source code size, and memory use; before summary statistics could be taken at face value.
No, but maybe a rough idea of a specific language implementation's properties.
>> if there are valid criticisms <<
There are plenty ;-)
http://shootout.alioth.debian.org/dont-jump-to-conclusions.p...
Readers obviously jump to conclusions based on box plots and summary statistics -- when they really need to look at the individual tasks and look at the program source code, because there's so much variation.
I think there would need to be incredibly tight constraints on minimum program run-time, source code size, and memory use; before summary statistics could be taken at face value.