Elie Hassenfeld: [00:00:00] Hey everyone. I'm Elie Hassenfeld, GiveWell's co-founder and CEO. Today I'm here with Adam Salisbury, and we're gonna be talking about data collection and how that influences GiveWell's decision making and other decision making.

This might sound like a fairly technical topic, but because we rely so heavily on data in our decision making, it's really important to understand the data sources we use, their limitations, and what we're doing to try to get better data in the future to inform our decisions and you know, do a better job directing funding effectively going forward.

This is especially timely because one of the key inputs that we rely on in our research and many other decision makers do in the field of global health and development, is what was a formerly USAID funded health survey called the Demographic and Health Surveys, or DHS surveys for [00:01:00] short. They had been funded by the US government and that funding was stopped. You know, this is really a critical piece of infrastructure that decision makers rely on.

And so today, Adam and I are gonna talk about the DHS surveys, how they work, what kinds of information they provide, and some of the limitations inherent in them. And then what we do when the DHS surveys doesn't serve our needs and how we collect additional data.

So Adam, good to chat with you. Maybe before we dive in, you could just share a bit about your background and what you focus on at GiveWell.

Adam Salisbury: Yeah, thanks Elie. I've been at GiveWell for about three years. Before GiveWell I was a research economist at an economics research institute in the UK, mostly working on education randomized trials in low income countries. And before that I was a consultant to the Zanzibar Ministry of Education.

I spent most of my time at GiveWell on our cross-cutting research team, so this is a team which sits above our different grant making teams. Broadly speaking, [00:02:00] we're responsible for two things. One is, upholding the quality of our research, so making sure that our models are consistent internally, making sure our research is legible externally. And second is, looking into grant opportunities which don't fit neatly into those grant making silos. So, a grant making opportunity which doesn't fit neatly into the malaria, nutrition, or vaccines team.

Elie Hassenfeld: Great, thanks for having this conversation. So, you know, one of the sources that I know we rely on a lot in our research and many others do, and I remember right when GiveWell started, this is one of the first data sources that we found was the Demographic and Health Surveys from USAID. Can you just explain what these are exactly and why they're so important to our work?

Adam Salisbury: Sure. So these are surveys predominantly conducted in low- and middle-income countries. They take place about once every five years, and they're representative surveys of the entire population. They were predominantly funded by USAID, but with other funders contributing as well.

And essentially, I think of them as trying to take as holistic as possible, a [00:03:00] snapshot of the health status of the population and the demographic status of the population. So they capture things like under five vaccination rates, people's diets, birth rates, mortality rates, whether people receive bed nets.

Elie Hassenfeld: Without these surveys, would we know this information or not? Like how critical is this information to what we know?

Adam Salisbury: For most of the information we use, I don't think we'd be completely in the dark without the DHS survey, but we would be more so in the dark. In two key areas, the DHS is one of, if not the most important input. And the one I'll talk about is mortality rates, so what percentage of children are dying, for example, in Nigeria?

So taking a step back, the reason we care about this is because we try to maximize the impact of our donations. When we're considering funding a bed net campaign, we wanna try to estimate how many lives this will save. To estimate that we'll need to know how many kids are dying in the first place.

Knowing how many kids are dying in somewhere like America or the UK is much more straightforward because most people die in [00:04:00] hospitals, there's an official recording of the death, so you can just count those up. In places like Nigeria, it's much more difficult because more than half of people die at home. There's no coroner that comes and makes an official recording of the death. So without these large, representative surveys, it's very difficult to just get basic information like the percentage of kids that are dying in the country in a given year.

Elie Hassenfeld: And you said that these happen every five years. So, does that mean that the data that we're relying on for estimating the under five mortality rate in Nigeria is an every five year number? Like what else is going into that estimate that we rely on?

Adam Salisbury: So you're right that it's every five years for the mortality rate. They ask that retrospectively. So they'll go to a household, they'll interview a woman between the age of 15 and 49, I think, in the household. And they'll ask, please list all the births in your life. And then they'll ask about the survival status of each child like, is this child still alive? And if they're not still alive, they'll ask, when did this child die? By doing that, you get a sort of time series of [00:05:00] death that occurred, and then using that you can get mortality rate for each of the previous years.

Elie Hassenfeld: And so when we or others in the global health field are trying to come up with this number which is let's say the under five mortality rate in, for example, Nigeria. To what extent is that number reliant nearly entirely on like this survey and that kind of question, versus any other factors?

Adam Salisbury: Yeah, so the way we estimate this is we tend to take estimates from modeling groups. Those modeling groups essentially take various different inputs, put them through a statistical model and spit out an output. We spoke to some of the researchers at John Hopkins and they said that in Sub-Saharan Africa, 80% to 90% of the inputs are either DHS surveys or something called the MICS, which is the Multiple Indicator Cluster Survey. That's run by the UN, so it's separate, but it's the same basic idea, it's like a representative five yearly survey. Those are by far the most important input. And then between DHS and MICS, I think DHS is slightly more important because there's just [00:06:00] been more of them, so there's more data and the sample sizes are slightly bigger as well.

Elie Hassenfeld: Got it. And so I think like this is something that seems like really important to recognize and understand, and it's surprising to most people.

There's this number that we rely on very heavily and it seems very straightforward. It's how many children are dying under the age of one, or under the age of five. The place we get that figure is, primarily, house to house surveys that ask people to recall the number of births they've had and which children have died and when, as the, you said, 80% to 90% of the input into this figure. And that's adjusted with some statistical modeling overlay.

But really it's like that data that is the underlying foundation of such critical information that I think, I mean, I certainly would've thought this before I got into GiveWell, you just would assume this data exists in some more robust place in some more robust way, and it's not solely based on this house to house data collection.

Adam Salisbury: [00:07:00] Yeah, I think that's right, and even at GiveWell. So when I first started looking into this, I knew the DHS was an important input. I think I've been surprised at like how important it is, not just for mortality rates, but fertility rates as well, which they're less important to our work and they're super important for population projections. Which you can imagine is very important for like how many schools should we build? How many hospitals to build?

Elie Hassenfeld: Yeah, what are some of the other data points that are like important for planning in a country, either for the country governments themselves, or for funders or policy makers? What are other examples of things that come from either, you know, one of these two surveys you mentioned that are basically following the same model?

Adam Salisbury: So I can talk about very general indicators, and I think the fertility rate is a good example of that. You can imagine fertility rates affect population projections, which I can imagine being important for almost every ministry in a government. If I'm in the Department of Transport in Nigeria and I wanna know how many roads to build, like I need to know how many people these are gonna serve. If I'm in the water ministry, I need to know how much water infrastructure we should be aiming for. So there are those very general indicators.

There are also very specific indicators that I think inform [00:08:00] policy as well. And there are some case studies on the DHS website of policy changes that have happened as a result of the DHS. It's a lot to sort of confidently claim causal impact 'cause counterfactuals are hard.

But there's a nice example in Malawi where it was only through the DHS survey that policy makers learned that women in Malawi had a very strong preference for injectables as contraceptives. So before that, most of their family planning programs were geared towards condoms and other forms of contraception. But based on that finding, they sort of shift their policy. And beyond the DHS, I don't think there was any other way that preference data was getting systematically collected in the country.

Elie Hassenfeld: Hopefully this helps make clear why the ceasing of funding for these surveys creates such a gap for low income countries. Because there is data like the under five mortality rate, data like the fertility rate, that's of obvious need to decision makers in country. And then like other indicators that they're collecting that can be useful in planning, that really goes away when the DHS is [00:09:00] gone.

And so, we'll talk a little bit about the fact that there are efforts being made to try to figure out how to protect this data and ensure that it keeps being funded and is ongoing. But absent the DHS, where would this data come from?

Adam Salisbury: So we still have the Multiple Indicator Cluster Surveys, so that's the UN one. That serves a very similar function to the DHS 'cause things like the mortality rate. It would just mean we'd have less data, so we'd have less precise estimates.

Some countries which have the DHS, especially middle-income countries, vital registration systems are not terrible. So, I don't know the exact figure, but a reasonable share of deaths get recorded in hospitals. So we could use that data as well, and people do use that data. I think we'd go from being 80% to 90% dependent on the DHS to being 80% to 90% dependent just on MICS.

Elie Hassenfeld: I'm also curious to talk through like what does it cost to deliver a survey? And I imagine it's fairly expensive. An obvious question someone might have is well, if this is so important, why do it every five years? You know, why not do it annually? And presumably one of the reasons, if not the reason is it's just very [00:10:00] expensive. And if you were going to run the survey five times as often you'd be spending a lot more money. What does it cost as best you can describe it and why does it cost so much?

Adam Salisbury: Yeah, so just to throw some figures at you. So the USAID contract to the DHS was worth $236 million and that was covering a five year period. So it was just shy of about $50 million a year. And that was to cover surveys in just over 50 countries. The cost for each survey varies by country because some countries just require larger sample sizes because they have a larger population.

The Nigeria survey in 2018, which is what I've got a benchmark for, that costs about $13 million to run. Not all of that came from USAID 'cause as I mentioned, there are other people that fund it too, including the Nigerian government. But it's expensive and the reason why it's expensive is one kind of, obviously it's a massive survey. So in Nigeria there was 55,000 surveys administered around 40,000 to women, around 15,000 to men.

And there's a lot of planning that needs to go into these. You need to obviously like hire a lot of enumerators to run that many [00:11:00] surveys. Enumerators are just interviewers essentially. But you need to hire statisticians to design the right sampling strategy so you get the most out of the data you're collecting. I don't think you need to do this, but I think it's helpful to sort of fund dissemination activities with the government. So, ideally, we don't just collect this data and it sits in the vacuum, but policy makers use it for policy and that requires some time and maybe some resource investment as well.

Elie Hassenfeld: Okay, so we're talking about the expense of the survey and you know, it costs about $50 million a year, and it's about 10 countries per year, if I'm getting the numbers right?

Adam Salisbury: About that, yeah.

Elie Hassenfeld: I mean, while there's like this huge variation where Nigeria was more expensive at $13 million, it's bigger. There's some that are smaller and less expensive. You know, we're talking about like millions of dollars per survey country. And then, you know, maybe there are some possibilities of making it cheaper, but it's still fairly expensive for all the reasons that you described.

And I'm guessing that a major reason is the way these surveys are conducted is you literally have to go house to house and sit down with someone. And the survey is pretty long. Do you have any [00:12:00] sense of how long it takes to administer? When I just think about like the content included, I'm sure it's several hours, if not more than that to administer a survey.

Adam Salisbury: Yeah. I'd need to double check the exact figure, but I think a couple of hours sounds right as a rough benchmark. And you're right, these surveys are literally administered by people physically visiting houses. And because these surveys aim to be nationally representative, the footprint of these surveys is the entire country. So you'll wanna have enumerators in northeast Nigeria and southwest Nigeria. So yeah, massive.

Elie Hassenfeld: You know, the travel itself takes a long time because the roads can be poor. Then you actually get where you're going and you need to explain to someone why you're there and what you want to ask them.

And when I was in Kenya last year, we actually like, sat with a survey enumerator and did like a mini survey. Something that I'm guessing would've just been some small portion of a DHS survey, like we did this as an example. It took us an hour just to go through like a small set. And so just I think when you think about 50,000 surveys administered to an individual household [00:13:00] at the time cost of, I don't know, you do two a day, that's a lot of person power that's required to get the data here.

Adam Salisbury: Yeah, I think that's right. And I think there's interesting analogs again with high income countries and just how much harder it is to get information about a population. So I'll give an example, in the UK we had a census quite recently, so a count of everyone living in the UK. The census in the UK is now mostly digital. You get sent a form you can basically log into a website and fill it out. I think about 90% of people did that. About 10% of people sent off a paper-based form, so they got sent something to their address, and then they filled it out, sent it off by the post.

You can't do either of those things in somewhere like Nigeria and DRC. Like most people don't have access to the internet, so they can't fill out an online survey. And the postal system doesn't function very well in very, very remote places. So you can't send surveys out and have people send them back again. So yeah, it's literally a case of driving up to these villages, which in some cases itself isn't an easy task 'cause the road quality's bad. I think it's easy to underestimate how logistically demanding it is.

Elie Hassenfeld: And I [00:14:00] think something that's really salient to me is, often the places that we're trying to direct funding are to the portions of countries that have the lowest resources. And you could easily imagine a situation where someone makes a choice that eases survey response but then is not representative of the whole country because it's looking at three quarters of the population who are the best off, but you're losing that bottom quarter.

And in our work it is very often that bottom quartile who have the greatest needs and so they're the ones we really need to know about. And without making sure that the data we have is accurately collecting their situation, we're gonna make worse decisions about where to direct funds.

Adam Salisbury: Yeah, I think that's exactly right.

Elie Hassenfeld: So I'm really curious, I know there's not a lot to say at the moment, but this data's so important. It seems like it's at risk of going away. You've been leading GiveWell's work to just consider whether and how we could support more. Like where does that stand right now in terms of something you think we should potentially put money towards?

Adam Salisbury: It is something we're still actively looking into. [00:15:00] So I'm speaking with other development partners at the moment who are also interested in stabilizing the project. We wanna make sure we're not stepping on each other's toes, so figuring out how we can sort of work together as opposed to doing the opposite.

Part of those conversations are also about what should be the plan for the DHS long term. So there's the sort of short term, the current contract was up until 2030. There are sort two questions we're asking ourselves. One is, what should we do to stabilize that and make sure that all the surveys that were planned get out the door. But then there's the separate, more strategic question is like, what happens after that? Where should the DHS be housed? Should it be continuing with the current contractor? Should we aim for the same ambition with this survey, or should we try and make it cheaper by cutting the sample size or experimenting with other ways of getting the data? So yeah, those are all live discussions.

Elie Hassenfeld: Yeah. So say more about that. I mean, I think the DHS I imagine has been around for a very long time. A lot has changed about what is feasible with respect to collecting information today relative to what was done. Has the DHS kept current with it's data collection approach? Are there ways that you think would be potential [00:16:00] avenues to collect better data more cheaply?

Adam Salisbury: So you're right that obviously the technology's changed a lot since the DHS started. So it started in 1984, if you think about then, like almost no one in Africa will have had access to a mobile phone. But now, not everyone does, but it's above 50% and rising pretty quickly as well. Mobile phones are a means to communicate with people, so maybe we could utilize mobile phone surveys. I think that's promising and we are looking into that now, which I can speak more about. I don't think these are a substitute for a household, in person surveys, I think of them more as a compliment to.

There are some things you just can't measure very well over the phone. So one example is in the DHS survey, interviewers will go into people's houses and they will ask to see all the bed nets that are hanging up. You can't obviously do that via mobile phone. They will also take photos of child vaccination cards to get a sense of the vaccination rate. You can't do that by phone. And there are just some questions which are quite sensitive like family planning preferences, for example, which I can imagine if people are less willing to give honest responses when it's kind of an anonymous phone interviewer.

Having said that, so there's things we couldn't collect, but [00:17:00] there are obviously advantages for some phone surveys. So I mentioned the DHS is quite expensive. Phone surveys can be done much cheaper. They can also be done much more quickly. So for us, if we want to get information very fast to inform a grant decision, we could do that via phone.

Elie Hassenfeld: There could be other reasons you want to go in person, like making sure you've reached the person you expect to reach in the place you expect to reach them. And I would guess that once you've done the work of going out to see someone in their house that's very far away, that's a fixed cost, and the marginal cost of asking extra questions in person probably is fairly low relative to the marginal benefit of getting better answers to the questions that you're asking.

Adam Salisbury: Yeah, I think that's right, and in the discussions I've been having about it, I don't think anyone's proposed replacing the DHS with phone surveys. The suggestion I've heard is, keep the DHS survey, but once you've made all this effort of collecting this very detailed demographic information about people, why not also collect their phone number? And then rather than survey them every five years for this very heavy touch survey, you can then, in addition, survey them periodically every six months or so via mobile.[00:18:00]

Elie Hassenfeld: So I mean, this seems like really obvious, why hasn't this happened yet?

Adam Salisbury: It's not straightforward for privacy reasons. So mobile phone numbers are personally identifying information. If you look at the DHS, all the data that's public strips out any personally identifiers. So the phone numbers couldn't be made public.

One other thing as well, I don't think this is a reason not to do it, but I think another challenge with phone surveys is people tend to change their phone numbers quite frequently, I think in low income context. So the phone number someone gives you at the time of the survey could change a year or so afterwards.

Elie Hassenfeld: Right, so it could be a lot cheaper, but there are these other problems. Okay. So we've talked a lot about the DHS survey and how valuable it is, but then I also think it has these real limitations.

It's like, we, GiveWell, ultimately end up relying on it for a lot. But then, it's not the case that the DHS survey provides even like a majority of the data that we need. I think I remember at one point trying to match up some malaria prevention activity we funded with malaria rates, you know, as measured by the DHS survey. And of course, just the timing of the campaign we [00:19:00] funded versus this every five year survey made that whole exercise impossible. And so we're doing a lot of other data collection, and I'm just curious if you could talk a little bit about how you are now thinking about supplementing the existing public data with maybe we call it bespoke data to serve, our own needs where we need data to make better decisions.

Adam Salisbury: Yeah, I think bespoke is a good way to describe it. One way that DHS falls short for our needs is what you just mentioned, like it only happens once every five years, and sometimes we want data that's at a more specific point in time than that.

Another reason is it's not customized to GiveWell, it's customized to what the government wants. And there are some things that GiveWell really, really wants to drill deep on, which aren't drilled in as deeply as we'd like in the DHS survey.

So one example of that is bed net durability. So bed nets are one major focus area of ours, but they're just one among a portfolio of investments for a government and other donors. One thing we care about is how long the nets last after we fund a campaign and specifically like how long until they get holes in them, et cetera, which reduces their effectiveness. That [00:20:00] data isn't collected by the DHS, so if we wanted to get an independent measure of that we'd have to fund it ourselves. Worth also noting that our grantee, Against Malaria Foundation, they do collect that information, but if we want an independent check, it'd be by ourselves.

And then the other thing that I wouldn't say GiveWell is uniquely interested in, but I think more interested in than other donors and other governments is, how should we trade off very different outcomes? So just stepping back at what GiveWell tries to do. So we try and maximize impact per dollar spent. And we try and do that across seemingly very different portfolios, so we could give a thousand dollars to someone living in extreme poverty, or we could give a thousand dollars to support a bed net campaign, the effect of which would be to save lives.

In an ideal world, what we try to do is put those into one overarching framework so we can make head to head comparisons between them. In order to do that, we need to come to some, quite ultimately, like philosophical judgements about how much should we value increases in consumption versus reductions in mortality risk. One way we could do that is just sort of come to [00:21:00] those judgements ourselves. A way that I would prefer is to actually ask the communities where we fund these programs, what they would prefer. And the DHS doesn't ask that question, and there are no other sort of third party resources which ask that question. So that's also an area where I'm interested in funding more research.

Just to flag, we have funded some of that research before. So in 2019, we ran a survey of people living in Kenya to ask about how they value increases in consumption versus reductions in mortality. But we haven't done anything else since then at a large scale, so I'm interested in that.

Elie Hassenfeld: Right. So this is like one example of information that we'd really like to get that would help us make better decisions that isn't publicly collected, and so we would go out and collect it ourselves. Just on that point specifically, like I think a question someone might have is don't we see people's revealed preference? If someone preferred I don't know, a malaria net over having a few dollars they would go buy the malaria net and the fact that charging for a malaria net significantly reduces uptake. [00:22:00] Why doesn't that just tell you that we already know their preference, which is cash over a malaria net?

Adam Salisbury: Yeah, I mean, that's a great question. So there are reasons why people might not buy the bed nets, one is imperfect information. If people don't realize how effective bed nets are at reducing mortality then when they don't buy the bed net, that may not be reflecting the sort of value they're placing on life. It could just be some misinformed belief about how effective these are at actually preventing mortality.

I think another reason is even if you assume people have perfect information, I think there's pretty good evidence, not just in low income countries, but also just humans in generally are quite bad at dealing with small probabilities. If you look at people cycling around the Bay Area, lots of people don't wear helmets. And I think if you showed these people what the base fatality rate is of dying without a helmet and with a helmet, it would imply that they put very low weight on their lives. I don't think people actually, well I'd be very surprised if people put actually that low end in their lives. I think people just sort of can fall into the tendency of thinking, oh, this is such a small probability, it won't happen to me.

Elie Hassenfeld: Alright. Thanks a lot. That all makes [00:23:00] sense.

Hey everyone, it's Elie again. So I think this is a really interesting example of some of the infrastructure that underlies the work that GiveWell does and that all donors need to rely on. I imagine for many people, these DHS surveys and the way in which they work and the data they collect is really an unknown part of international development.

I also think it's a great example of just how hard it is to know the things that we need to know. You know, for GiveWell, so much of the work that we do is aiming to support life-saving programs, especially for children under the age of five. And as Adam said, in many places where we work, it's very hard to know how many children under the age of five are even dying. And these surveys, which have now had their funding cut, are the main way that we know what's happening with the health of young children in the countries that we aim to support.

But really it's a good example of the kind of information, you [00:24:00] know, we at GiveWell need to understand. And we need to dig into so that we're able to do our work well. You know, right now we're faced with this additional question of, how should we value ongoing support of these surveys? Is this something that we should support? And how should we think about trading off between supporting additional data collection, which will inform our future decisions on others, versus using that same money to just help people as much as we can. You know, that's the kind of decision that we're often faced with, we're certainly faced with here.

We appreciate your interest in our work and your support. If you're sure that you know you want to give and just help people and don't want funding to go to data collection, then supporting our Top Charities via the Top Charities Fund is a great way to go. All of that money goes right on to the organizations that we support. On the other hand, if you're open to either helping people directly or potentially funding data collection where it's needed, the All Grants Fund is a great place.

With the All Grants Fund, we're able to support a broader range of programs where we often, you know, know less [00:25:00] about exactly the impact that those programs will have.And I think the possible support for data collection and these DHS surveys is a good example where. We know they're important, but we can't come to an estimate of the cost per life saved for them in the same way that we would for our Top Charities Fund. And so, for that kind of support, the All Grants Fund is really great.

Regardless, I just appreciate your interest in our work and for your ongoing support and interest, thank you so much.