Microsoft is still busy doing damage control after Consumer Reports said last week that it would no longer recommend Surface computers, as a result of their high failure rate. The downgrade was based on a user survey that included a total of 90,000 responses across all bands.
An internal Microsoft document about the reliability of the Surface surfaced on Sunday that didn’t immediately seem to help Microsoft’s case.
The document, acquired by longtime Microsoft blogger Paul Thurrott, showed that in the three-month period just after the launch of the Surface Pro 4 and Surface Book in late 2015 and early 2016, Microsoft saw unusually high return rates for the products.
Thurrott recalls that in early 2016 Microsoft executives blamed the high return rates on Intel, the maker of the new Skylake processors used in the Surface Pro 4 and Surface Book. Thurrott remembers the executives calling the Skylake chips “buggy” at the time.
But the blame may have been misplaced. Thurrott:
Since then, however, another trusted source at Microsoft has provided with a different take on this story. Microsoft, I’m told, fabricated the story about Intel being at fault. The real problem was Surface-specific custom drivers and settings that the Microsoft hardware team cooked up.
So the leaked document paints a decidedly unflattering picture of the original build of the Surface Book and Surface Pro 4 and Microsoft’s initial comments about quality issues.
But the document also says something important about the Consumer Reports findings. The publication came under fire from some analysts Friday for downgrading a whole line of computers based on a survey for which it would provide precious little detail.
Consumer Reports said on Friday it would not divulge the number of Surface owners on whose comments it had based its findings, other than that its published results include only brands with at least 300 responses. Nor would it provide reliability scores for specific Surface models. That means that we don’t know whether the Surface Pro 4 and Surface Book problems acknowledged in the leaked document are to blame for CR‘s blanket refusal to recommend any device in the Surface line.
What Is A Failure?
I once managed a large reliability and satisfaction survey at PCWorld, like my colleague Harry McCracken did before me. I can tell you from experience that the line between subjective and objective observations by computer owners is often blurry. Events that survey takers described as “reliability” problems were often better described as “ease of use” problems.
The manager of the whole Surface line at Microsoft, Panos Panay, points out in the leaked memo that this may have been an issue in the Consumer Reports survey, too. The survey scored computers on the number of “failures” reported within two years of ownership. Panay points out that the definition used by Consumer Reports might lead to an easily rectified problem like a temporarily frozen or unresponsive touch screen being counted as a “failure.”
The Consumer Reports downgrade is a very big deal for Microsoft. The Surface has gotten lots of good vibes from the media and become a multi-billion-dollar business, but the machines’ sales have fallen for two consecutive quarters. Watch for the company to go into full-court press to contain the ill effects of Consumer Reports‘ damaging vote of no confidence.