Patient satisfaction surveys are flawed in many ways. Here are just a few:
1. Sampling is a huge problem. A description of why sampling is an issue can be found here. It’s a bit complex. To summarize, the validity of a survey is strongly related to the size of the sample and the rate of response of the survey. If you have a patient base of 1,000 and elect to survey 500 of them and receive responses from 100, the sample is really only 10% [100/1000] of the population in question. This would result in an accuracy of 95% ± 20%.
Most patient satisfaction surveys include far fewer than 50% of the population in question and have far lower response rates than 20%.
Thankfully, I no longer directly participate in the quarterly hysteria that occurs when patient satisfaction scores are received by hospitals. A national company that many hospitals use sends out small numbers of questionnaires that have a very low rate of response. In addition, they use only a five point scale as a basis for their ratings and report the results as percentiles. [Note: I am not a statistician, but I don’t think it is kosher to report a five point scale in percentiles ranging from 1 to 100.] Usually there are modest up and down variations in these scores that are almost never statistically significant, especially when you consider the margin of error of well over ± 20%. Upon receipt of lower scores, task forces are established, multiple meetings are held, policies are changed, and staffs are browbeaten. Many times the scores improve on the next cycle, and the task force is congratulated. Lost in the euphoria is the fact that there is a 3-month lag between the institution of any policy changes and the receipt of the next group of survey responses. In other words, the policy changes probably were not the cause of the uptick in the scores.
Note that the Medicare Hospital Consumer Assessment of Healthcare Providers and Systems [HCAHPS] survey suffers from many of the same flaws, such as small sample size, poor response rates, and way too many questions.
1. There is no correlation between patient satisfaction scores and complaints.
2. Surveys are more reliable if they are completed as close to the time of the encounter as possible. Most are not done that way.
3. They do not necessarily correlate with quality of care as is demonstrated in papers involving medical patients and patients with heart attacks.
No doubt the facts will not deter the bean counters from mandating that all physicians survey patients for satisfaction no matter how meaningless the data may be. American Medical News recently reported that the AMA and one of the large national surveying companies will be happy to help you with this for a mere $65 per month for AMA members and $85 per month for non-members.
I’m thinking of starting my own patient satisfaction surveying company.
Skeptical Scalpel is a recently retired surgeon and was a surgical department chairman and residency program director for many years. He is board-certified in general surgery and a surgical sub-specialty and has re-certified in both several times. For the last two years, he has been blogging at SkepticalScalpel.blogspot.com and tweeting as @SkepticScalpel. His blog averages 800 page views per day, and he has over 4,200 followers on Twitter.
SkepticalScalpel