Some Analysts Have Doubts About Consumer Reports’ Dismissal Of Microsoft’s Surface Line

On Thursday, Consumer Reports said that it could no longer recommend Microsoft’s Surface tablets and laptops. It based its statement on its annual user survey, which found that a quarter of all Surfaces malfunction within two years of use. That’s a sweeping commentary of a whole line of machines.

Overall, the publication’s sample consisted of 90,000 readers who own PCs and tablets. That may sound like a lot of data points, but that’s for owners of all models from all manufacturers. When I asked Consumer Reports on Friday afternoon for the number of people in the survey that own Surface devices, they would not provide the figure. “In order for a brand to be included in our survey, the minimum number is 300,” said spokesman James McQueen. “That’s according to our internal survey team.”

Writing before Consumer Reports disclosed that threshold, some analysts said that the publication doesn’t have the data to back up its blanket dismissal of the Surface. The two-year ownership window Consumer Reports studied might skew the sample toward older Surface models. It might include many devices older than the Surface Pro 4, and not many of the new Surface Books, noted Creative Strategies analyst Carolina Milanesi. “If I am correct in my assumptions of which models are part of the sample and I look at the most aggressively priced Surfaces, Surface RT, and Surface 3 could make up a large proportion of this sample,” Milanesi wrote in a blog post for the research firm’s clients on Friday. “While the assessment of those specific products might be correct, it is certainly not a reflection of the Surface current lineup.”

Moor Insights & Strategy analyst Patrick Moorhead says he’s used every mobile Surface device ever produced for extended periods. “Besides some momentary issues with early 2012 Windows RT devices and 2016 [models using Intel Skylake processors], which were fixed in software updates, I have not had issues,” Moorhead writes in an email to Fast Company. His personal experience, of course, is anecdotal. But he adds, “What CR is lacking is any kind of root cause or segmentation across product lines that show that a 2017 product should not be purchased based on something arising in 2014.”

And Microsoft came to its own defense early Friday with this statement from Panos Panay, the charismatic frontman who manages the Surface line, about the Consumer Reports judgment:

In the Surface team, we track quality constantly, using metrics that include failure and return rates—both our predicted 1-2-year failure and actual return rates for Surface Pro 4 and Surface Book—are significantly lower than 25%,” writes Panay. And: “Surface also ranks highly in customer satisfaction. 98% of Surface Pro 4 users and Surface Book users say they are satisfied with their device.

But as longtime Microsoft blogger Paul Thurrott points out, Panay’s factoids don’t directly counter Consumer Reports‘s contention that a quarter of Surfaces break down within two years. As for his own experience, “Anecdotally, I’ll point to the fact that the three Surface Book models I’ve used have all had reliability problems,” Thurrott wrote. “And that, contrary to that, my Surface Pro 4 has never had any issues at all. Because that’s why the Surface Book and Pro 4 reliability issues are so vexing: Some never have issues, but others never stop having problems.”

All this matters because Consumer Reports has a strong reputation for being rigorous, thorough, and fair. It’s trusted. Millions of people check in with their research before risking their money on an expensive piece of tech gear. On the other hand, Microsoft has done an impressive job with the Surface line over the past few years, and the machines have improved significantly since first making the scene back in 2012.

Consumer Reports is one of the few sources we have left for independent third-party data on PC reliability—publications such as PC Magazine and PC World having ended such studies in recent years—so what it says matters. That’s also why it’s concerning that it would pan the reliability of a whole line of computers without telling us exactly how many people actually had problems, and what the problems were. When I asked spokesman McQueen for more granular data on CR‘s Surface findings, he said that the publication wouldn’t be providing such details.