Part 1: What Really Happens at a BoP Clinic?: A new study shows surprising results
Looking for a disturbing read? Try the recent study published in Health Affairs on primary care provider quality in India. Using standardized patient methodolgy, it uncovered a number of alarming findings: low levels of medical training, frequent incorrect treatments, rare correct diagnoses – and only small differences between trained and untrained doctors in some important aspects of medical practice.
Rose Reis at the Center for Health Market Innovations recently interviewed Jishnu Das, senior economist in the Development Research Group at The World Bank, and one of the authors of the Health Affairs study. In Part 1 of this two-part interview, they discuss the study’s unique methods and surprising findings. (Read Part 2 of their interview here.)
Rose Reis: You opted to measure quality through standardized patient visits rather than through observation or patient recall. Can you summarize how this works? It must be extremely difficult to implement – is it quite rare?
Jishnu Das: In previous work with Jeff Hammer, we used both direct observation and patient recall to measure quality. The direct observation worked fairly well, and the results, described in this paper and this paper, are similar in some regards to what we document using standardized patients. Yet, we worried about the limitations of direct observations.
Could it be that differences across doctors arose because patients selected different doctors depending on the severity of their illness? Did observing doctors change their behavior–something that Ken Leonard had shown in Tanzania?
And finally, because we never knew what patients came in with, we suspected, but were never able to show, that correct diagnosis rates were low across all types of medical providers.
For this study, we finally decided to make the plunge and use standardized patients to measure the quality of care.
Standardized patients–people recruited from the local community to present a consistent case to numerous health care providers–are widely considered the gold standard in quality measurement. For every provider, they present with the same symptoms (and dress and walk the same way!) and will give the same answers to questions that providers may ask.
Because doctors believe the patient to be “real,” the questions they ask, the examinations they perform, and the treatments and diagnosis they provide are accurate measures of the quality of care, purged of the problems associated with direct observation.
This is indeed a rare study, and it does require a fair bit of work to develop and train the standardized patients; studies thus far have typically used smaller samples to validate the approach rather than provide field-based data on primary care processes.
Working with a multi-disciplinary team including doctors, an anthropologist and a standardized-patient trainer, we recruited people from local communities, developed extensive scripts, piloted the approach among doctors and finally, after more than 150 hours of training, deployed the standardized patients in the field.
Yet, my feeling is that the work required is minimal relative to the power of the information generated. Furthermore, standardized patients can easily become part of a regular system of monitoring in health systems. Costs are high primarily at the development and training phases; if standardized patients were institutionalized and used in a regular feedback cycle, the costs would be significantly reduced.
(Above: Jishnu Das)
RR: What kind of cases did your “patients” come to providers with? Why did you ask them to present with these types of sicknesses?
JD: Our standardized patients presented with cases of unstable angina, asthma, and dysentery in a child (who was “sleeping at home”). These cases satisfied four conditions:
- They are important in the Indian context, where chronic and infectious diseases are both represented in the overall disease burden.
- The symptoms could be accurately mimicked by standardized patients (unlike, say, a high fever).
- None of these types of cases required invasive examinations, which could put the standardized patient at risk.
- All of them had established protocols and treatment guidelines developed by the Government of India, which allowed us to assess the care received against benchmarks obtained through a national consensus.
RR: In your sample, 67 percent of rural medical practitioners reported no medical qualifications at all–even in government clinics, 63 percent of providers your patients met with had no medical training, although the requirements call for a trained public doctor. Was this surprising?
JD: Careful studies of rural health care providers in the past have consistently shown high rates of health care provision by private providers without any medical qualifications.
But the high rate of provision by medically untrained providers in the public sector was indeed surprising, and reflects both a shortage of trained doctors in rural public clinics, and high rates of doctor absence in these regions.
RR: Your study found that providers did not often follow appropriate protocol–did not ask all the appropriate questions about cases they were presented with, did not do many of the necessary examinations, and made incorrect diagnoses in nearly one out of two cases. Based on your study, can you describe a typical interaction for a patient with unstable angina symptoms (chest pain, often a precursor to heart attack) coming to a rural medical provider for care?
JD: The average interaction for such a patient would last three minutes (and remember, there are no nurses to take vital signs or check height and weight before the patient sees the doctor!)
The provider would ask maybe three questions, complete one examination and prescribe three to four medicines. The questions are typically about the location and start time of the pain (but less than a quarter would check whether the patient experienced radiating pain down the arm—a classic give-away for this case).
The exam is usually one of the four vital signs (blood pressure, pulse rate, respiration rate or temperature), and the medicines reflected what the doctor thought the patient had. These ranged from Zantac for stomach ulcer to Acetaminophen, which is contraindicated in the case of angina. A very small minority (4 percent) would give Aspirin, which can be life-saving under these conditions, and around a third would prescribe an ECG or refer the patient to a hospital, or both.
It is commonly believed that the private sector overuses diagnostic tests in India, but our study shows the opposite. Critical diagnostic tests such as ECGs are severely under-used among both public and private sector providers in India.
RR: Why is it important to look at unnecessary prescriptions offered to patients? Do they frequently cause harm? Can you give examples?
JD: I think you want to look both at under-treatment (things that should be done but were not) and over-treatment. Some of the over-treatment could result in direct harm to patients, as in contraindicated medication like Acetaminophen for unstable angina.
Other over-treatment could lead to future drug resistance in the patient, and/or could contribute to a rise in overall drug resistance. This is a persistent problem, for instance, with the overuse of antibiotics in India.
Finally, even without any adverse health consequences, over-treatment represents a financial cost to the end-user, who pays for medicines they don’t need.
RR: Do your results reflect low rates of training or high caseloads? Or would you attribute the results more to something else?
JD: We are working on this further, but what we know so far is that the results do not reflect either high case-loads or the paucity of equipment in these clinics. In fact, we found no correlation between case-loads and quality, or the availability of equipment and quality. Both results are fairly easy to reconcile with our observations in the field.
The typical rural doctor in our sample saw 10-15 patients a day, which meant that for most of the day he/she was idle–idle-time was anywhere between five to six hours a day.
This seems to be a general attribute of rural medical care. For instance, time and motion studies in rural Tanzania and Senegal show that doctors spend an average of 29 minutes a day seeing patients. Similarly, most of the providers in our sample already had the basic medical equipment to correctly diagnose and prescribe treatment for these cases.
Unfortunately, our results also show that training had only a small effect on quality of care. There are a couple of possibilities. One is that the quality of medical education coupled with the lack of re-certification requirements leads to enormous variation in quality even among doctors with medical training.
A second, and this is something that we and others have documented previously, is that training without commensurate effort will still lead to poor quality. You need a trained doctor who is willing to put in the time and effort necessary to follow diagnostic and treatment protocols.
In other words, training and equipment are necessary, but not sufficient for quality care. In follow-up work, we are trying to disentangle these multiple hypotheses using a novel sample of providers who practice in both the public and the private sector, and I believe the results from that work will yield important new insights.