by Louisa Hackett - For those of us trying to make the world a better place, figuring out a way to study, understand and assess the interventions we make is critical to improving the work we do.
Being accountable for the money we spend by knowing whether our programs work is a necessary burden. Yet, given the trade off between spending money on evaluation versus delivering an essential service, the research questions we ask better be relevant, the data collected better be valid and – if we are really trying to create positive social change – we better be open to hearing what the data says, good or bad.
The New Yorker magazine's May 17 issue profiles an MIT professor, Esther Duflo, who believes something can be done about poverty. She also believes that data is needed to understand the effectiveness of different strategies being used to solve the problem. Duflo co-founded the Abdul Latif Poverty Action Lab at MIT in 2003 which has conducted over two hundred experiments world-wide, testing social policy theories.
Some of Duflo’s studies appear to prove the obvious: increased education results in higher wages; students’ whose teachers were absent scored lower on tests than those whose teaches showed up for work. However, other Poverty Lab studies produced less predictable results. For example, giving away protective bed nets is more effective in preventing malaria than selling them at a low price. And, studying the effect of hiring low-cost remedial teachers in primary schools resulted in a policy which now serves 33 million children in India.
The Poverty Lab uses a research method called randomized control trials by which results of an intervention with one group are compared against another group picked at random. The method is said to be a simple, valid measure. It can avoid ethical issues of denying a service to a control group by comparing the ‘’treated’ group against the wider population. And, both groups if large enough allow the researcher to be certain that a change is a result of the intervention or treatment.
Perhaps one of the most intriguing issues in the article was the wary response to an evaluation of micro financing. Internationally, microfinance has been touted as a miracle: an effective way to ‘solve’ poverty. Make a loan, and businesses will start, income will rise, the loan will be paid back, and people will move themselves out of poverty. And, the ‘banker’ can even make money. Microfinance is capitalism working for the larger good.
Yet, the Poverty Action Lab’s study of 7,000 households in India showed that microfinance did not fix everything. Getting credit to the poor worked, but an overall improvement in consumption, a measure of economic well-being, did not happen. Duflo became convinced that not having a steady job most likely prevents a person from having an easier life. Again, this seems obvious. However, the wider field promoting and running microcredit programs was wary, questioned the research findings and defensively provided anecdotal evidence that microfinancing works.
The article does not state the actual cost of the microfinance research; however given the number of people required to conduct 7,000 interviews in India both before and after the program and the time to organize, analyze and report findings, it is safe to assume the expense was significant. While the agency agreeing to the study was interested and wanted follow-up research, the microfinance field in general was not; which suggests the feedback has not been widely used within the sector to improve programs.
How does all this affect our work here in New York City? Getting data about our efforts to make a more fair and just City is needed. Knowing what works and what does not is useful. But, we should not bother spending time and money on evaluations, if we’re not ready to hear what’s wrong with our programs. Evaluations should not be prompted just because a funder requires it, but because we want to learn from and make our programs better. If we are not willing to learn, the evaluation expense will be wasted. If we are open to constructive feedback, the time and money spent can be well worth the effort.
Subscribe via RSS feed 
Comments
It should be noted however that formal evaluation is different from performance management. Performance management allows for the collection and analysis of data in real-time to understand if your efforts are helping to move you closer to your objectives. This allows you to make mid-course corrections to improve. Evaluation, is very valuable but is more about what you accomplished yesterday whereas performance management is about what you will do tomorrow.