Another Memo to Consumer Reports: What About the OS?
January 5th, 2011As most regular readers know, I’m not enamored of Consumer Reports and the way the magazine handles personal technology. Their efforts to simplify and generalize possibly complex information ends up helping nobody. While they no doubt accurately represent reader surveys about product and service reliability, the actual reviews are tragically flawed.
I’m also concerned why so many members of the media accept CR at face value, and don’t question the clear and present flaws in their test methodology. It can’t just be me!
Consider the February 2011 issue, which was just dropped into my mailbox a short while ago. One of the feature articles is entitled: “Tablet computers,” and the subtitle pretty much sums up the conclusions: “Apple and Samsung offer the most for your money.”
The tested products include, of course, the iPad, and the Samsung Galaxy Tab, along with a smattering of lesser products from Huawei, Archos, Coby and Maylong. The later, with a 7-inch screen, retails for a mere $100, and that may be too much, considering that CR concludes that its “Touch screen has poor response. No access to a standard app market.”
Unfortunately, the key areas where you might compare tablets of different sizes and configurations are omitted. Almost as bad, CR tested the iPad with the older 3.2 OS, even though version 4.2, with multitasking and loads of other features, shipped in November. Then again, the differences between the iOS and Android, which powers all but one of the remaining tablets in this lineup (the Archos 9 PCtablet uses Windows 7), are never explained. The closest you get are some generalities about the app markets. iOS and Android are good, but “Third-tier app stores we visited were simply not up to par.”
Maybe so, but CR has an extremely low level of acceptance, failing to point out the obvious differences between the App Store and Android Market, such as far more substantial apps, including games, in the former’s repertoire. Worse, nothing is said about the fact that Google has never certified current Android OS versions for use in tablets. What this means is that the apps are designed for smartphone screens, not even the 7-inch displays tested by CR. As a result, the images, particularly text, might be pixellated when expanded to fill a larger screen.
Then again, the difference between the smartphone display and a 7-inch tablet isn’t such a stretch. Going to 9.7 inches on the iPad required compromises for existing iOS apps, which open in a smaller window, but can optionally zoom to fit the larger borders, with the expected sacrifices in quality. But with tens of thousands if apps that now support the iPad, you’re more likely to find what you need without compromise.
Alas CR, which has managed to stick feet in the mouths of its tech crew with that lame review of the perceived flaws of the iPhone 4’s antenna, doesn’t attempt to examine the usability of a 7-inch versus 9.7-inch tablet. Steve Jobs had made a large issue of this perceived disparity, pointing out that the 7-inch screen gives you only 45% of the real estate of the iPad’s display, which he believes is too small. CR didn’t test this contention, although they cover the difference in passing.
So the iPad “is a tad clunky to carry around,” and I would agree with that statement. The 7-inch Galaxy Tab “is more portable, even if the smaller screen won’t wow you the way the iPad screen will.”
Well, I suppose that sort of covers some of the differences, but the actual experience of consuming content and performing some basic productivity chores isn’t evaluated. The issue of usability is seldom on CR’s radar when it comes to personal computers and related products. They seem too focused on the hardware differences, real or imagined, and the most basic functionality, emphasizing the usefulness of the touch screen on tablets. Beyond that, they have nothing to say.
But that’s been true about CR’s personal computer reviews, where they compare Macs with under-equipped Windows PCs, which simply makes the Mac seem a lot more expensive. Windows? Mac OS X? To CR, they are two flavors of the same ice cream, with only minor differences, or at least that’s the impression the lack of a meaningful comparison conveys.
I suppose if there’s anything you can take away from CR’s mediocre coverage of tablets is that they don’t recommend paying extra for 3G capability, because they believe Wi-Fi access is ubiquitous enough for most users. I don’t disagree, but that’s just me. It’s also clear that price does impact quality. For the most part, the cheapest tablets were seriously downgraded for one reason or another. However, the Archos Windows 7 model, listing for $430, got dinged for poor app access and a subpar touch screen that isn’t “very responsive.” That, however, is likely as much the fault of Windows 7 being nudged into a direction for which it wasn’t designed, as the design of the touch screen itself. But CR doesn’t consider those possibilities.
The review will, naturally, be outdated when the next iPad debuts, probably in a month or two. But I don’t expect CR’s coverage of the next model, and its competitors, to be any better.
Print This Post
It’s CR’s way of being “fair and balanced.”
@Hairy Goomer, No wonder I have a problem with that phrase. 🙂
Peace,
Gene
Unfortunately, what CR is doing passes for genuine journalism these days. “Reporters” (on TV especially) will just tell us things like “Well, this is what this side says, and this is what the other side says” as though they both have equal validity, which is rarely the case. As you stated, CR treats the OSes as though they are little more than clones of one another, with neither of them having advantages over the other. You have documented this behavior well over the year, regarding CR and their “comparisons” of Windows and Mac OS X.
I don’t read anything CR writes. They are faux journalists at best.
@Hairy Goomer, Too many reporters have become copying machines. They don’t know how to — or aren’t expected to — actually do some digging for a story.
Peace,
Gene
To be fair, CR is used to reviewing cars, and they seem to have carried over their automotive review technique to the computer world. To serious car people, there are cars that feel right on the road, and there are clunkers. There are cars that are pleasant to drive or ride in for long periods of time, and others that are uncomfortable. There are cars with good, easy to get at storage space, and their are cars with terrible stowage. Rather than imagining some standard CR driver, they tend to concentrate on the hardware features of the car, the engine, standard test performance, cubic feet and so on. They’ll tag stuff like impossible to get into rear seat, but won’t go much beyond that in trying to predict how the car will be used.
Unfortunately, this doesn’t quite carry over to the computer world. Software fit and finish are much more important in usability. They don’t seem to notice just how hard some computers make it to get in and out of the back seat or turn off the heater. Sure, there is lots of room in the back, and the heater blows lots of hot air, but those specs are less important than ease of use. CR is well behind the times. Their automotive model just doesn’t work for computers.
CR has never really gotten computer technology. They understand toasters and will submit a toaster to 15 tests but for computers, they pretty much turn them on and tap a few keys and call it a day. because of their long lead time, while most toasters might be on the market for a year or two, computers change out quickly – a thing taht seems to have escaped them – basically they were always about a year behind on macs so it was never recommended. It wasn’t until about 2008 before they finally got around to testing and recommending macs to WIN PC’s but they still value price above all else … or look at the iphone thing – to them, squeezing down on a secret part of the iphone produced “lower bars” so it was “obviosuly unacceptable” … not only could 99% of people not figure out how to replicate this (I tried) but a $1 band or a FREE case could resolv this issue even for the 1% who had this mythical problem … it’s like complaining the coin slot is for EU coins so you should not buy a Porsche. They are AR focused on the small issue. Yes, I don’t want a car that rolls over easily but I can live with a car with tiny coin slots …
Trusting CR to do a proper review on Computers and complex Tech items is like asking your neighborhood car mechanic to fix a rocket engine. Yes, they are both ‘engines’, but…. you get my drift 🙂
CR is equally poor at reviewing audio equipment. For example, they seem to think that all amplifiers of similar spec sound alike….
Actually that’s probably true. Other than one’s imagination, amplifiers with similar specs sound the same in level-matched double-blind listening tests when used within their operating range. I was involved in these discussions long, long ago, and that is the case. When it comes to speakers, however, differences can be quite drastic.
Peace,
Gene