I agree. Also it creates a halo effect. And creates more incentive to click either up or down depending on how you view the number relative to your own opinion. Not having a count gives new comments a chance (since you don't automatically look at the "minutes ago" part). So a new comment can be not at the bottom giving it a chance to gain more exposure even though it has no points.