Digital Camera Ratings - Season 2 Episode 1

2008.03.23

Last time on Digital Camera Ratings....

• Neocamera has four ratings: Poor, Average, Good and Excellent.
• Ideally, an Average rating would be given to an average camera. Other ratings should be centered around this one.
• Higher ratings are awarded for image quality first, then ergonomics. Features are not directly taken into account.

The coarseness of our ratings is somewhat arbitrary. Some sites use a precise rating system, with cameras rated from 0 to 10 with one or two decimal points, others have only two ratings. The reason for using a small number of ratings (4 in this case, could be 3 or 5, it's not that important) is because there is a lot of subjectiveness in how important deficiencies are. We can point out deficiencies, but we can't say how important they are to you.

Giving a precise rating for a camera, like 8.7, is in my opinion misleading. It may lead in something thinking that a camera with an 8.7 is better than one with an 8.6 rating. Then there is the question of how much better and if its worth the $$$ differences. How's $75 for 0.1 difference? Note that a rating here is different then a measurement. It is not like saying than an engine is rated at 520hp, that is a measurement, just like megapixels or ISO.

Keeping ratings well distributed is the tough part. The main problem is that digital camera technology does not stand still. The result is a moving average which is impossible to track with fixed ratings. We thought of two approaches to deal with that: periodically re-rating cameras and using a relative scale. The former way is the more logical but proved problematic. We did it once when we had less than 50 reviews. Now that we have over 200 reviews, it is a gargantuan task. The latter way is simply confusing for readers. Imagine that all Good ratings were not equal... Is Excellent from 2 years ago is better or worse than Good from last week?

For those who take ratings very seriously, this poses two problems:

• The average (in the numeric sense) camera gets a better than Average rating. Note that there is always skew here because we do not review all cameras and those that we review are more often than not near the top of their line.
• Cameras rated Excellent differ in quality more than other ratings. Why? Because once the first cameras were given the top-rating, everything better got to ratings too. Over 3 years, that means you get the best camera from today with the same rating as the best camera from 3 years ago.

What readers have to do then, is to keep in mind that ratings are a guideline - that is one of the reason we chose a coarse rating system - and that the assessments, pros-and-cons, express reviews and full-reviews are there to expose the true story.

• Review