If you are a patient looking for help when trying to find the right doctor, you probably think checking out online reviews would be a good thing to do.
Two studies should give you pause.
The first paper looked at risk-adjusted mortality rates for cardiac surgeons in 5 states (California, New York, New Jersey, Pennsylvania, and Massachusetts) that have published this data. Of the 614 surgeons whose information was public, 96% were rated on one or more of the well-known rating websites.
The average risk-adjusted mortality rate for all the surgeons was 1.68% with a range of 0% to over 16%. The median rating for all surgeons was 4.4 on a scale of 1 to 5, and the median number of reviews per surgeon was 4. That’s right, 4.
From the paper, the figure below shows the risk-adjusted mortality rate on the Y axis plotted against the average online ratings on the X axis.
As you can see, ratings did not correlate with mortality rates, p = 0.13. Some surgeons with the highest risk-adjusted mortality rates had ratings of 5.
The authors understated conclusion was “Patients using online rating websites to guide their choice of physician should recognize that these ratings may not reflect actual quality of care as defined by accepted metrics.”
The second paper compared ratings data from three popular sites for 410 physicians who had been placed on probation by the Medical Board of California to controls matched by zip code and specialty.
Again the average number of ratings was low at 5.2 for the probation group and 4 for the controls.
Doctors with some categories of violations—medical documentation, incompetence, prescription negligence, fraud—had significantly lower ratings than controls. But for other categories including problems of professionalism, drugs or alcohol, crime, sexual misconduct, and personal illness, ratings were not significantly different than those for controls.
For those physicians on probation, average ratings did not differ significantly between physicians whose probation status was published by the ratings website and those whose status was not published. This suggests one of two possibilities—either the raters did not look for sanctions or they did look but did not care.
“The mean rating for physicians on probation was 3.7 compared with 4.0 for controls when all three rating websites were pooled (p <.001). Despite this difference being statistically significant, the absolute difference is quite small with overlapping rating distributions. The vast majority of reviews were positive with small numbers of reviews per doctor.”
On univariable analysis, the overall ratings for those on probation vs. controls differed significantly [odds ratio 1.5 (95% CI 1.001-2.2)] but accounting for gender and age were not significantly different by multivariable analysis [odds ratio 1.4 (95% CI 0.9-2.2)].
The authors wrote, “Web-based physician ratings were lower for doctors on probation indicating that patients may perceive a difference, [but] the absolute difference was quite small”
Some claim bad data is better than no data at all. I disagree. I can’t say it any better than the authors of this study, “Physician rating websites have utility but are imperfect proxies for competence.”
Skeptical Scalpel is a retired surgeon and was a surgical department chairman and residency program director for many years. He is board-certified in general surgery and a surgical sub-specialty and has re-certified in both several times. For the last 8 years, he has been blogging at SkepticalScalpel.blogspot.comand tweeting as @SkepticScalpel. His blog has had more than 3,000,000 page views, and he has over 18,000 followers on Twitter.
