Metrics, evidence, and RCTs in global health: An Interview with Vincanne Adams

The following two tabs change content below.
David Flood
David Flood, MD, MSc, is a physician with the Guatemalan NGO Wuqu' Kawoq | Maya Health Alliance and resident in Medicine-Pediatrics at the University of Minnesota. He received his medical degree from Harvard Medical School and an MSc in international health policy from the London School of Economics.

vincanne adamsVincanne Adams, PhD, is Professor of Medical Anthropology and Vice Chair of the Department of Anthropology at UCSF. She is the author of numerous books and articles relating to global health and development in the U.S., Tibet, Nepal, and China. Her forthcoming edited book, “Metrics: What Counts in Global Health” (Duke, 2016) critically explores the dominant role of metrics and randomized controlled trials in global health research and practice. I spoke with her about her new book and some of her other recent global health writing on slow research, alternative accounting, and the production and uses of evidence.

What has drawn you to write about global health metrics and evidence?

I had been doing a lot of work on Asian medicine for many years, working in the Himalayas. I started in Nepal, and I wrote a book on doctors’ role and use of scientific models in the democratic revolution. Then I went to Tibet in 1993 where I looked at maternal health, safe motherhood, reproductive issues, and the dynamics of medical knowledge under fraught political conditions. I also joined a non-governmental organization devoted to training rural health workers in techniques of safe motherhood. I came on with the stipulation that we would incorporate traditional medical knowledge, as my own research had shown me that Tibetan women preferred to conceptualize their pregnancies and births in Tibetan medical terms. I was committed to the idea that preserving Tibetan medicine was important for Tibetan identity and their future in China.

We started to develop a curriculum and training program for rural health workers that combined Tibetan medical practices with Western medicine. We got a big grant from the NIH to do this. We spent years preparing this project, developing protocols, and getting IRBs set up in Tibet. And then finally we were told that we couldn’t do our project on Safe Motherhood because we couldn’t get the “right numbers”—because not enough women die in childbirth in Tibet to get a good power calculation. It was a shock! At the same time, it was also true; in these rural areas, we couldn’t get the right kind of numbers to do a randomized controlled trial of our intervention.

So I started to look at how evidence-based medicine had become something that was being pushed in global health. I was especially interested in how principles of evidence-based medicine were changing the way people were designing and carrying out global health interventions. I noted how it put new demands on non-governmental organizations of different sizes and scales. It looked to me like a promissory idealism—that randomized controlled trials finally were going to give us the answers we needed for what works and what doesn’t work. And I was critical of that because, in my experience, it was the need for a randomized controlled trial that actually scuttled an otherwise thoughtfully designed project in Tibet.

Is this how the book on metrics came about?

Metrics: What Counts in Global Health

In addition to my experiences in Tibet, I was very influenced by work I conducted in New Orleans after Katrina. My book “Markets of Sorrow, Labors of Faith: New Orleans in the Wake of Katrina” was a very detailed study that included interviews with over 160 people each year over four years. The essential argument of that book was that the neoliberal shift in governance that allowed private sector, for-profit companies to fund and orchestrate the recovery process was a total failure. The first half of the book shows how the recovery was delayed in a massively painful way because of the assumption that the private sector and for-profit companies could do a better job than the government. The second part of the book details how small non-profits and NGOs, who used volunteers, were what enabled the city to rebuild. Ironically, the rise of non-profits is also tied to the neoliberal turn where the government is no longer funding public institutions for the safety net, so it’s being left to charity, non-profits, and faith-based volunteer groups to do that work.

Experiences with this research led me to think about how a similar thing is being seen in the international health or global health world. It’s not quite the same; there’s a long history of non-profit and foundation work ever since the colonial era. But there’s been a real shift with the proliferation of NGOs of all sizes involved in global health work. You have large players like the Gates Foundation and the Global Fund, but on the ground it’s the numerous small NGOs and mom-and-pop shops that have the strongest presence. So the strange thing is that the models of evaluation, accountability, and implementation that are being promoted by global power brokers including the WHO don’t seem to be very well suited for the actual global health work of these small NGOs.

So how should we make sense of the outsized role of small NGOs in global health in light of new standards for what counts as global health evidence?

You have these grand statistical machineries in place, and RCTs are a particular kind of data generation instrument. And then you have the co-production of meta-analyses where people have set a new standard for what constitutes good evidence. It does run the risk of rendering previous kinds of evidence fallible or less reliable. In fact, the people I know who are working at small NGOs are very troubled by this shift because their donors are coming to them and saying “where’s the evidence?” and “what kind of evidence is it?” NGO workers are being expected to generate information that follows these same epidemiologically driven models.

You argue that relying on RCTs as the highest level of evidence can limit what kinds of questions you can ask.

There are two things that happen when you assume that RCTs are the only way to get reliable information about effectiveness and accountability. One problem is that if you can’t set up an RCT for the intervention because you can’t get the right numbers, then you might abandon good projects. This was our experience in Tibet.

The other problem is that when you actually do succeed in setting up an RCT, in order to get good data, you have to ignore so much other information in order to get your results. I’m not just talking about cleaning the data; I’m talking about how so much in an RCT is tuned out as static in the system. If it’s not one of the variables that you’re looking at, then it gets turned off and you don’t hear it anymore. This can be a violent process to the empirical world. You may end up ignoring things that actually matter quite a bit in terms of effectiveness, impact, and implementation.

Your comments on this topic make me think of a problem I’ve observed in Guatemala. The group I work with is very interested in indigenous language advocacy, but we’ve struggled about how to generate evidence that delivering health services in indigenous languages is relevant, or has an “effect size.” We can’t randomize half of our patients to not receive medical care in their maternal language. So we must turn to other methodologies that are considered inferior to the “gold standard” of RCTs, and we worry that our argument will thus be viewed as inferior as well.

That’s a really good example of why we should be skeptical of the RCT as a perfect instrument.

At the same time, I have come around a little bit on RCTs because they can actually be very local. That’s one of the advantages of an RCT. You can get very specific information about a specific intervention in a specific setting. The problem is that the data often gets overgeneralized. But the fact that you can use RCTs in very local ways is something that should be remembered.

"I started to look at how evidence-based medicine had become something that was being pushed in global health. I was especially interested in how principles of evidence-based medicine were changing the way people were designing and carrying out global health interventions." (Photo from Watsi.org)

“I started to look at how evidence-based medicine had become something that was being pushed in global health. I was especially interested in how principles of evidence-based medicine were changing the way people were designing and carrying out global health interventions.” (Photo from Watsi.org)

Adapting the principles of the local food movement, you have proposed the idea of “slow research” in global health. Can you explain this idea and the reaction you’ve received?

It’s less about “slow” research and more about “local” research. All of those things I talked about in regards to how RCTs and metrics organize our knowledge and render some things invisible—how do we reverse that? It seemed to me that if we’re reversing the perspective and thinking about measurable outcomes in relation to local things, then we should look at the successful model we have from the slow food movement. That really has shown one way “local” can work.

The people who read it and love it say, “Absolutely, this affirms what we’ve been trying to say!” I suspect there is a whole other group of people who think it’s a bunch of hogwash. I’m not sure all of my colleagues in global health see the utility in this approach. For example, I had a conversation with one person about scale and scaling up, and they said, “You can’t scale up using a local approach.” That is true. I don’t think you should. That’s why I talk about other kinds of scaling.

One theme that emerges across your writing is that you want to re-prioritize local specificity and local particularities in global health. Is that fair? 

That’s fair.

But also there’s potentially a very interesting conversation coming down the pipeline that I’ve been trying to think more about in relation to precision medicine in the context of global health. Precision medicine started out as “personalized medicine.” In the medical field, it’s a convergence of evidence-based medicine and RCTs with the pharmaceutical industry’s push to get the maximum effectiveness out of drugs for specific subgroups of people. This latter point is explained well in Joseph Dumit’s book Drugs for Life. Big data is giving researchers the tools to figure out exactly for whom one kind of medicine works over another.

Take breast cancer. If you can figure out exactly what kind of cell the patient has, you can deliver the exact right kind of drug to that person. Many differences have already been studied in medicine—racial, age, gender, and tumor differences. These are all variables big data can analyze to make better use of existing drugs or slightly modify drugs in order to make them more effective with subgroups. If you do a drug study and the drug heals 10% of the people, the interest would lie in figuring out what is special about that 10% and then tailoring drug treatments accordingly.

I’ve been thinking about how the principles of precision medicine could be applied to global health work. What would it mean to do “precision global health”?

That’s very interesting. In global health research, you might have a negative multi-site study, but in one of the sites the intervention might have really worked well.

How do you cultivate an instrument that allows you to observe when an intervention is working well in a specific setting and pursue it more there? What are the variables that would indicate that you should use it there?

For example, in the earlier study I mentioned in Tibet, not every piece of the Safe Motherhood intervention worked in every community. In some communities, you had to change things around to get it to work. And I’m not just talking about belief systems, but also whether there is a paved road in a specific community or not, the presence of an existing health care facility or not. There are many variables that you may want to look at systematically as part of a “precision” approach that promotes local specificity.

"What I’m talking about is something more radical, which is that you shouldn’t be more than local. I’m arguing that you have to do local work in local ways." (Photo from Watsi.org)

“What I’m talking about is something more radical, which is that you shouldn’t be more than local. I’m arguing that you have to do local work in local ways. I’m arguing that a truly local approach is not manageable by big funders or policymakers who want “one-size-fits-all” policy.” (Photo from Watsi.org)

This reminds me of another idea you’ve written about that comes from the field of clinical research, the “n-of-1” trial in global health. 

In simplest terms, I was thinking of “n-of-1” trials as a way to counter the need for a large sample size—to emphasize that if something worked for an individual person, then that matters too.

This idea is oppositional to the impulse to scale-up. It’s the kind of thing that gets classified as “anecdotal” information and therefore not valid. But, for example, if you’re working in communities where women are dying of childbirth, you don’t need to have a large n to have a very good idea about what’s causing women to die. The “n of 1” is a way to affirm that every life counts; it’s not only studies with big sample sizes that count.

My thinking also comes from a student of mine, Dana Greenfield, who was working on the “quantified self” movement, which is a movement where people keep track of massive amounts of personal data. They track their heart rate, weight, respiratory rate—they’re using new technologies to intensively monitor their biology. And they keep track of these numbers and patterns. They say, “When I do this much exercise, when I eat this food, when I use this drug—this is what happens.” They’re measuring everything and getting very large sets of data with an “n” of 1. This presents an interesting counterpoint to the view that good science can only happen when you have large numbers. These people do have large data sets, but they’re only for one person.

So, in global health, your idea is about what would it mean to similarly generate vast quantities of data about what works in a specific place—without having a large “n”? And in an “n of 1” trial, each person is his or her own control.

Exactly.

I wanted to ask you about the word “anecdote,” which has become so vilified. You write that anecdotes have become “like the untouchables of India’s caste system” in the hierarchy of evidence-based medicine. But you are trying to reclaim the anecdote.

I think anecdotal information is the best! [Laughs.]

Give me some anecdotes to help explain what you mean! [Laughs.]

In my first chapter for the Metrics volume, I also write about how anecdotes haven’t completely disappeared from the discourse. Anecdotes as evidence get brought back to affirm some kind of moral certainty around the intervention or to create an affective bond around those doing the work and those receiving the work.

So the anecdote still does a huge amount of work, but my argument is that using anecdotes this way—as window dressing to affirm the numbers—is problematic. What we need to be doing is using the type of information contained in anecdotes to actually influence how programs are implemented in specific contexts. Harnessing this idea, some anthropologists like Simon Lewin and Chris Colvin are conducting large reviews of the qualitative literature to make arguments about what interventions are working and not working.

But even that model reflects the impulse to scale up interventions. What I’m talking about is something more radical, which is that you shouldn’t be more than local. I’m arguing that you have to do local work in local ways. I’m arguing that a truly local approach is not manageable by big funders or policymakers who want “one-size-fits-all” policy.

2 Trackbacks