Challenges of Programs Evaluation of Health Interventions in Developing Countries

by Barbara Wynn, Martha I. Nelson, Arindam Dutta

Overview

Examines various approaches, methodogies, and issues related to evaluation of the impact of health programs in developing countries and ways to make program evaluation more rigorous.

Product Details

ISBN-13:	9780833038524
Publisher:	RAND Corporation
Publication date:	11/01/2005
Edition description:	New Edition
Pages:	112
Product dimensions:	7.06(w) x 8.92(h) x 0.34(d)

Read an Excerpt

Challenges in Program Evaluation of Health Interventions in Developing Countries

---

By Barbara O. Wynn Arindam Dutta Martha I. Nelson

Rand Corporation

Copyright © 2005 RAND Corporation
All right reserved.

---

Chapter One

Introduction

Purpose of the Study

A recent editorial in the Journal of the American Medical Association noted that the effectiveness of many health interventions in developing countries has not been proven. The editorial called for increased international support and collaboration to provide the infrastructure to evaluate global health interventions and move toward evidence-based global health (Buekens, 2004). Interventions that are effective in developed countries may not be effective in developing countries that have differing social, economic, cultural, and infrastructure factors that may affect how a project is implemented and the project's outcomes. Rigorous program evaluation of interventions in various resource-limited settings is needed to determine which interventions will work most effectively and to spend scarce resources wisely.

This report is intended to promote understanding of why evaluation is a critical component of any health intervention and to stimulate discussion on ways to increase the capacity to evaluate health projects and make evaluation of health interventions in developing countries more rigorous. We provide an introductory overview of various approaches, methodologies, and issues related to program evaluation for health projects in developing countries, andcite other sources of information on various evaluation techniques. We conclude by identifying research and other actions that could be taken by funding organizations that would facilitate impact evaluations of large-scale health interventions.

Program Evaluation in Developing Countries

Evaluation should be conducted throughout the various stages of an intervention, starting with the project design and ending with an assessment of ultimate outcomes. At any given stage of an intervention, the focus, scope, and technical issues will vary depending on the intended purpose of the evaluation at that stage. A well-designed evaluative strategy generally involves the following steps:

a formative evaluation during the project's developmental phase to clarify objectives and to refine the project design (including the evaluation strategy and data requirements), while taking into account the cultural environment and other local factors that influence how a project is implemented process evaluations throughout the project implementation phase to provide timely feedback on how the intervention has been implemented and what might be done to improve it operationally to achieve desired outcomes an impact evaluation to assess the net effects of the intervention and whether the intervention's goals were reached.

Impact evaluations establish whether there is a causal chain of events (or "causal chain") between an intervention and observed outcomes. There may be a direct cause-and-effect linkage in the chain (e.g., vaccination for polio has a direct effect on an individual's immunity from polio) or the cause-and-effect linkage may be indirect (e.g., the impact of polio immunization campaigns on the national infrastructure for disease surveillance). The causal chain may involve a series of cause-and-effect linkages, some of which may be upstream (i.e., precede the intervention under evaluation), horizontal (i.e., cause-and-effect factors that involve individuals, such as family members, and organizations not directly participating in an intervention), or downstream (i.e., take place after an intervention, such as longer-term impacts on an individual or the community). To establish the causal chain and attribute observed changes to an intervention, it is important to understand what changes would have occurred in the absence of the intervention, all else being equal. The challenge in determining the effectiveness of an intervention is to control for any other factors that might explain the observed changes and to identify and measure the indirect effects of the intervention.

In the subsections below, we provide an overview of the public health issues on which we focus in this report-vaccination campaigns, Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome (HIV/AIDS), malaria, and tuberculosis-and the evaluation challenges they present. The interventions for each of these four public health areas have unique characteristics that pose their own set of challenges for impact evaluation.

Vaccination Campaigns

The World Health Organization (WHO) estimates that more than two million deaths in 2002 were from diseases for which vaccines are already available, and almost 1.5 million of those deaths were children (WHO, 2003b). A 2003 Global Immunization Score Card showed that only 26 percent of developing countries meet an 80 percent benchmark for routine immunization coverage, and a majority of developing countries has not yet introduced newly available vaccines, such as a vaccine for Hib (Haemophilus influenzae type b, which causes some forms of pneumonia and meningitis), which killed 413,000 children under the age of 5 in 2002. Over the past decade, immunization coverage of DTP3 has stagnated far below targets for sub-Saharan Africa and South Asia after rapid improvement in coverage throughout the 1980s, according to WHO/UNICEF estimates (Vaccine Assessment and Monitoring Team, 2004).

Because the efficacy of vaccinations is well established, and because vaccinations target diseases that occur early in childhood, evaluations of the effects of vaccines follow relatively short causal chains. Evaluators generally assume what the immunization effects of vaccination will be and rely on simple counts of vaccinated individuals to measure effectiveness; evaluators do not need to measure differences in disease prevalence.

However, persistent shortfalls in vaccine coverage have two implications for program evaluation. First, impact evaluations should consider not only whether the goals of an intervention were reached in terms of numbers of immunizations but also what accounted for the success or failure of a vaccination campaign. More information is needed on the broader determinants of the success of a vaccination campaign so that lessons learned and best practices can be considered when designing other vaccination campaigns. Second, more attention should be given to the downstream effects generated by immunization campaigns. The polio eradication initiative illustrates the indirect benefits of a health intervention. In addition to eradicating the disease from many areas, the polio program strengthened both immunization systems in general and the global health infrastructure, including worldwide networks of laboratories, surveillance systems, human resources, and vaccine-delivery equipment (Davey, 2002; Levine and What Works Working Group, 2004). Polio immunization campaigns also serve as platforms for providing basic health care services to children who had never before received medical attention. Currently, both polio and measles vaccination campaigns serve as channels for delivering vitamin A (Davey, 2002).

Broader impact evaluations that consider the indirect effects of vaccination campaigns could potentially demonstrate any additional benefits from those campaigns (and whether any trade-offs with routine immunizations and other preventive services are associated with vaccination campaigns) and lead to increased political support for them.

Malaria

Malaria causes more than 300 million acute illnesses and at least one million deaths annually, primarily among children in the tropical and subtropical regions of the world. Deteriorating health systems, increased resistance to drugs and insecticides, and environmental changes have led to an increase in malaria over the past two decades. Malaria poses intervention challenges as a vector-borne disease with no consistently effective vaccine and high resistance to treatment.

The public-health debate on malaria centers on the relative value of bed nets as an effective long-term strategy for malarial containment. To measure the effectiveness of this intervention, program evaluation must go beyond counts of distributed nets to substantiate a downstream causal pathway-i.e., the evaluation must also determine the degree to which people use bed nets appropriately and whether treating the nets with insecticide has a noticeable effect on the disease burden of the individuals using nets. Researchers might also try to determine if applying insecticides to bed nets lowers the population of mosquitoes in the environment or alters evolutionary pressure on viral character, which would confer indirect benefits on the community in which the intervention took place. The question of disease burden raises an issue regarding whether the unit of observation for gauging an intervention's effectiveness should be just the individuals who were provided with bed nets or the entire community in which the intervention takes place.

The long-term sustainability of the bed-nets intervention has been in question in part because some study findings have suggested that bed-net use in infancy might increase mortality in older children though delayed acquisition of immunity to malaria. Results from a two-year community randomized trial in Kenya, followed by continued surveillance of adherence and mortality rates for an additional four years, are encouraging in this regard. The researchers found sustained bed-net usage over the six-year study period and found no differences in the mortality rates in older children who had bed nets as infants and those who had not (Lindblade et al., 2004). The evaluation design for the Kenya study illustrates limitations that ethical considerations may pose for evaluation. For reasons of equity and community acceptance, insecticide-treated bed nets were distributed to control households at the end of the initial two-year trial, and bed nets were retreated with insecticide throughout the post-intervention surveillance period. Thus, the question of what happens to the mortality rates for children who are protected by bed nets in infancy after they are exposed to normally high malaria transmission remains unanswered.

Tuberculosis

WHO estimates that in 2002 eight million persons developed active tuberculosis (TB) and two million died from the disease, with 90 percent of the active cases and deaths occurring in developing countries. The persistence of TB over the past decade has been primarily due to lack of government commitment to TB control, poorly managed TB-control programs, poverty, population growth, and a significant rise of TB cases in HIV-endemic areas (WHO, 2002a).

Directly Observed Therapy Short-Course (DOTS) is an internationally accepted cost-effective strategy to control TB that consists of five key elements:

Government commitment to sustained TB control Detection of TB cases through sputum smear microscopy among people with TB symptoms Regular supply of high-quality anti-TB drugs Six to eight months of regularly supervised treatment Reporting systems to monitor treatment progress and program performance.

Globally, only 37 percent of the estimated number of TB patients received treatment under the DOTS strategy in 2002, with an 82 percent average success rate for treatment. As a broad-scale social intervention, the DOTS strategy does not lend itself to randomized controlled trials and poses evaluative challenges-for example, determining why TB rates in the former Soviet Republic have fallen in recent years and understanding the implications that these falling TB rates might have for TB-control programs in other countries (Dyer, 2005).

Multi-drug resistant (MDR) TB is far more costly to treat than TB strains that can be treated with standard short-course therapy. Cost-effective treatment regimens still need to be identified for MDR TB. Because interrupted or discontinued standard short-course therapy can lead to drug resistance, whether improved standard short-course chemotherapy can reduce the incidence of MDR TB and the epidemiological and economic impacts of poorly implemented DOTS programs both need to be evaluated (Dyer, 2002).

Tuberculosis is the leading cause of death for people with HIV/AIDS. In the past, interventions for TB and HIV/AIDS have been viewed and evaluated separately. Program evaluators need to establish norms and guidelines for evaluating the increasing number of "joint" programs for HIV/AIDS and endemic diseases and for addressing the measurement and evaluation challenges they pose (WHO, 2003a).

HIV/AIDS

The Joint United Nations Programme on HIV/AIDS (UNAIDS) estimates that in 2004 nearly 40 million persons were living with HIV and 3.1 million died from AIDS, about 95 percent of whom live or lived in developing countries. Perception of HIV/AIDs not as an isolated medical condition but one within a larger social context has important implications for the complexity of evaluation of HIV/AIDs programs.

At one level, the complex array of sociological factors that contribute to modes and rates of HIV/AIDs transmission generate substantial challenges in attribution, i.e., establishing the link between the intervention and individual HIV rates. Programs that indirectly affect viral transmission, such as HIV education programs, require robust evaluations that capture multiple elements along a lengthy, complex causal chain between intervention and disease burden. However, as researchers begin to appreciate the extent to which HIV's toll on society far surpasses disease rates through its leading toward greater economic and social upheaval, the impacts of evaluation shift from medical outcomes toward the larger societal goals of preserving the workforce, the family unit, and community function. This appreciation of the larger social and economic benefits of HIV prevention also provides additional incentives for investment in prevention programs.

When public health programs are operating in developing countries, the indirect effects generated by those programs may be less predictable than the effects in more-familiar Western settings. For example, a hypothetical vast mobilization of antiretroviral treatments (ARTs) for HIV patients could have substantial economic, political, psychological, and behavioral impacts far beyond those from a medical perspective. In terms of social impacts, the commitment and ability to treat HIV patients could have substantial impacts on (1) the stigma and fear associated with HIV, (2) native perceptions and distrust of Western activities in developing countries; (3) increased recognition of AIDS as a real disease (many birth certificates still record only the ultimate cause of death, such as tuberculosis).

In terms of economic impacts, the urgent needs of ART programs to train new staff to dispense medicines could generate a new class of rapidly trained clinicians. Keeping people sufficiently healthy to hold jobs, support their families, and keep their children in school could have vast economic implications. In terms of biological impacts, reducing HIV patients' viral load (the amount of HIV virus in an individual's blood stream) reduces the probability of additional infections and the chance of further viral transmission to children as well as to sexual partners.

The political effects could also be substantial, as national governments cooperate with international aid organizations on programs they both endorse, rather than on controversial condom-distribution and sexual-education activities. These effects are posed as hypothetical conjecture to illustrate the vast social context of HIV/AIDs and, as a result, the importance of taking these social and economic impacts into account when evaluating HIV/AIDs interventions.

(Continues...)

---

Excerpted from Challenges in Program Evaluation of Health Interventions in Developing Countries by Barbara O. Wynn Arindam Dutta Martha I. Nelson Copyright © 2005 by RAND Corporation. Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.

---