Talking about Truth and Power in Evaluation

For EDC’s Leslie Goodyear, being a good evaluator is about much more than collecting and presenting pristine data. Instead, it’s about building human connections.

“You can have the most perfect study design and data, but if you haven’t managed to develop a good relationship with the people running the program you are evaluating, then they might not even listen to what you have to say,” she says.

This year, Goodyear began her term as the president of the American Evaluation Association (AEA), the field’s largest professional organization. In this role, she will be leading efforts to improve evaluation practices and increase the use of evaluation for decision-making in fields such as the sciences, humanities, education, and international development. Here, she talks about some of the challenges facing evaluators, as well as what she wants to achieve in her new role.

Q. As new president of AEA, what are you excited to be working on this year?

Goodyear: One thing we are currently doing is reviewing and revising the Guiding Principles for Evaluators, which is a set of guidelines that help evaluators practice in an ethical manner. As part of that process, we have heard a lot about big data and social media. Advances in technology have really changed some of the ways evaluators think about collecting data, because we can now pull from social media and the back ends of websites. We need to think about the ethics of that.

Evaluators take protecting human subjects seriously—it’s important to protect the confidentiality of the people who are telling you about their experiences with a particular program. So the question, How do you protect human subjects when using social media to evaluate programs? is very current. We’ll be rolling out the new guidelines by the end of the year.

Q. The AEA conference in October is titled “Speaking Truth to Power.” What does that mean in the context of evaluation?

Goodyear: The phrase was first used as a title for a 1950s American Friends Service Committee pamphlet about nonviolence. I found it to be an interesting, relevant notion that speaking what we actually believe to the people who have the ability to change policies can change the world.

Evaluators often talk about “speaking truth to power” as a core belief. If I am evaluating your program, and I see that it’s not really working the way you intended it to work, then I need to be clear with you about what I see happening and how that’s affecting your participants. It can be uncomfortable, but that’s our job as evaluators.

Q. Program evaluation seems like it would be fairly straightforward—you figure out what works and what doesn’t. Are there times when this isn’t the case?

Goodyear: I’ve been in situations where a funder will provide money for a program that seeks to improve education, for example, but then balks at changing anything when the program evaluation shows that they could be having an even greater impact. So that can be uncomfortable, because as the evaluator, you are sort of forcing funders to reevaluate how committed they are to a certain issue.

Speaking truth to power also carries some risks for the people who are benefiting from the program that is being evaluated. If an evaluator reports that a particular program isn’t working well, or that it was based on faulty assumptions, then that could mean the program gets shut down. And that has ramifications for the people who were in that program—students, teachers, nurses, whoever. They may be left without a program that was important to them. So I think we, as evaluators, need to be cognizant of that. We need to be aware of speaking the truth, while also understanding the power that our evaluations have.

At the conference, we’ll be discussing several questions: Is it our responsibility as evaluators to speak truth to power? Or do we just take contracts and deliver data? Do we have a moral responsibility to make the world a better place? What’s the best way to give feedback to our clients, and how can we help them take responsible action? And I think those conversations will be great ones, because there are a lot of evaluators who have examples to share.

Q. If we’re sitting here a year from now, what does success look like?

Goodyear: There are a number of things I want to have done. Ratification of the new Guiding Principles for Evaluators is certainly one of them. I also want the organization to be even stronger than it is now. On that point, I think we are already making progress—we’ve just hired a new executive director, who will help AEA implement its strategic plan. I want to have had a successful annual conference, too.

But more than anything, I think success looks like an engaged community of evaluators. So if I can help strengthen AEA while helping evaluators continue to improve the work they do every day, then I think this year will have been a big success.