For several years, I have written at great length about the problems that arise when Consumer Reports tackles tech gear. Yes, the magazine has a stellar reputation, in large part because tested products are all bought retail, they don’t take ads from other companies, and don’t even allow those companies to quote CR reviews in promoting their products. CR is also run by a non-profit corporation that is funded by sales, subscriptions and even reader donations.
This veneer of incorruptibility means that most everyone takes the magazine’s ratings seriously. That would be a good thing if the reviews were thorough, and the ratings made sense. Quite often they do. So the information in the March 2012 issue about the excessively high calorie and fat count of the buttered popcorn you buy for inflated prices at the local multiplex shouldn’t come as a surprise, but maybe moviegoers will pay attention and seek healthier refreshments. The roundup of LCD and plasma TV sets accurately described the differences between the two technologies, and why plasma is often better unless you want the brightest possible picture.
But when it comes to such personal tech gear as smartphones, tablets, and particularly personal computers, CR falls down on the job. Way down.
A notable example is the curious way in which they handled the alleged “Antennagate” scandal, involving the original iPhone 4 and the possibility that you could kill reception with what became known as a “death grip.” Despite all the visual evidence that a similar phenomenon could be easily duplicated on other phones when held in somewhat different ways, CR decided that only the iPhone 4 was at fault and, despite getting the highest numeric rating in a smartphone feature, still wouldn’t recommend the product. CR was even oblivious to manufacturer warning labels and printed documentation that also cautioned against holding their mobile handsets the wrong way.
To be perfectly fair, yes, the iPhone 4s did get a recommendation due to the superior antenna system and relatively high test results. But other models got higher scores for having larger displays and/or 3D. It doesn’t seem as if CR bothered to consider the usability of a larger form factor, and how the gadget might fit in pants and shirt pockets, not to mention the ease of one-handed manipulation. Larger must be good, period. CR also didn’t bother to actually compare the user friendliness of the various smartphone operating systems, nor the quality of the various app stores in terms of selection and quality of software.
When CR reviews personal computers, it’s not at all clear how closely they try to match the various specs, or whether the basics, such as display size, hard drive capacity, and memory, are sufficient for them to put products in the same overall category. Although CR is aware of the existence of Mac OS X and Windows as separate, distinct platforms, they do not actually compare the two in any meaningful way, so you can decide whether to go Apple or with one of the Windows PC models.
The same issue that carries those reports about movie theater snacks and high definition TVs also contains an article entitled, “Light & lively laptops.” Here, CR seems oblivious to the difference between traditional note-books and so-called “Ultrabooks,” since it seems they rate them all in the same category, defined strictly by display size.
Even then, Macs rate at or near the top in the 11-inch and 13-inch categories. Why a 13-inch Samsung scores a tad higher than a 13-inch MacBook Air isn’t really explained, other than the former having a lower price. Curiously, CR compares two versions of the 13-inch MacBook Pro, which appear to be from different generations, rating one better to the other when it comes to something called “Versatility.” How so? With CR, you never know, because they don’t explain such fine distinctions.
At least on this occasion, the tested Macs were priced comparably to the Windows note-books, except for the lower-rated models. At least these ratings indicate that, yes, you should get better value for a higher price. That’s good as far as it goes, but the ultimate questions of usability and OS elegance are off the table for CR, which only seems to understand specs and raw benchmarks. To them, it appears that there is no Mac versus Windows question, not any reason why, for example, Apple is gaining sales while most PC companies are suffering from flattened or lower sales. Certainly you cannot attribute this to the alleged Steve Jobs reality distortion field, since he’s no longer here, yet Apple’s sales are better than ever.
On the other hand, if I were in the market for a new vacuum cleaner, I suppose CR would be a good place in which to compare the various models, although most are at least good enough. For autos, CR isn’t interested all that much in the “fun to drive” factor, since they are more attuned to basic ride, handling, safety and comfort issues. A car may be supremely comfortable in all respects, an efficient people hauler from here to there without guzzling a lot of fuel, yet be a total bore to drive. But that’s not CR’s market.
Now what’s unfortunate about all this is that CR seems tone deaf to the problems with their reviews. They aren’t asked the hard questions by a fawning media, and thus have nothing to explain. But with all the resources at their disposal, they should do a better job than anyone. Too bad they haven’t figured that out.