Back in July, an organization called ProPublica released its Surgeon Scorecard, which attempted to give patients a way to compare surgeon performance.
While the effort was well-intended, it was criticized by many, including me. Starting on the day it was published, I wrote several posts (links here, here, and here) detailing what I thought were its flaws.
My objections and those of many other detractors were brushed off by the authors of the scorecard.
Last week, several well-known academics associated with the RAND Corporation issued a report thoroughly dismantling the methodology of the Surgeon Scorecard.
Here are some of the highlights.
Complications considered by the ProPublica authors included only those causing readmission within the first 30 days after the operation. The RAND report pointed out that about two-thirds of surgical complications occur during the index admission (when the surgery was performed). In addition, significant and sometimes life-altering complications that occurred after 30 days were omitted. The Surgeon Scorecard listed only a 2.9% complication rate after radical prostatectomy, but erectile dysfunction (about 50%) and urinary incontinence (about 14%)—complications that really matter to patients— usually occur much later than 30 days postoperatively.
The RAND report explained that emphasizing only complications related to readmissions is not sensitive enough to judge surgeon quality because of “differences in patient case-mix, differences in resources allowing management of complications in the outpatient rather than inpatient setting, and differences in surgeons’ innate propensities to readmit a given patient.”
The RAND group also took issue with ProPublica’s risk adjustment model, which has never been validated by any other study and did not include standard clinical measures.
The data set provided to ProPublica by Medicare contained many errors such as identifying non-surgeons as being responsible for performing operations that ended up in the Surgeon Scorecard.
The RAND report, which also contained suggestions for improvement of the scorecard, concluded, “Given all of the methodological problems detailed herein, there is substantial reason to doubt that the ProPublica Surgeon Scorecard, as currently constructed, will help patients choose those surgeons who are most likely to provide good surgical outcomes.”
ProPublica’s answer to the RAND critique follows in its entirety:
“ProPublica responds that it and its scientific advisers previously addressed the issues raised by the RAND critique here, here and here. Reasonable scholars can and do differ on the best methods for measuring surgical complications. We are hard at work on Surgeon Scorecard 2.0 and appreciate the input from the RAND team.”
The problems identified above along with many other issues raised by the RAND report were not specifically addressed in any of ProPublica’s earlier statements. You can click on the links in the paragraph above to see for yourself.
The scorecard authors have never acknowledged that any aspect of their work might be not be valid.
If there is nothing wrong with the current Surgeon Scorecard, why are they “hard at work on Surgeon Scorecard 2.0”?
Skeptical Scalpel is a retired surgeon and was a surgical department chairman and residency program director for many years. He is board-certified in general surgery and critical care and has re-certified in both several times. He blogs at SkepticalScalpel.blogspot.com and tweets as @SkepticScalpel.
SkepticalScalpel