SSIR@PND

Through an agreement with the Stanford Social Innovation Review, PND is pleased to be able to offer a series of articles and profiles related to the "business" of improving society.

Time for a Three-Legged Measurement Stool

Time for a Three-Legged Measurement Stool

For decades, people have framed the conversation about measurement in the social sector in terms of monitoring and evaluation. They shorthand it as "M and E" and serve it up as a generic, two-dimensional description for measuring nonprofit performance. Monitoring is the routine data collection and analysis conducted by an organization about its own activities, while evaluation typically means the kind of data collection and analysis conducted by an independent third party.

In many respects, these two complementary parts of measurement have matured and strengthened over time. Aided by technology, monitoring has developed into the collection of information about who is being served and with what level of frequency and intensity, and even the tracking of short-term outcomes — all of which can inform decision making. Meanwhile, the evaluation field has become more nuanced, with new approaches available to answer a wide range of questions about outcomes, impact, and the factors that enable or inhibit change.

But for all they represent, these two building blocks are insufficient. We need a third leg of the nonprofit measurement stool to achieve more balance; we need feedback. Distinctly focused on the customer or constituent experience, feedback involves systematically soliciting, listening to, and responding to the experiences of nonprofits (or government direct-service providers) and customers about their perceptions of and experiences with a service or product. By listening to customers, we can gain unique insights that will help us improve the quality and effectiveness of our programs.

Innovation, Revelation, and Amplification

Certain organizations are already leading the way in using feedback. Many have embraced customer perspectives as a crucial component of their work to source innovation, to surface hidden problems, or simply to amplify marginalized voices in systems of service delivery. Examples of these three advantages underscore how integrating feedback into the measurement process can benefit both programs and clients.

Sourcing innovation | Some organizations implement all three legs of the stool. One such organization is Nurse-Family Partnership (NFP), an evidence-based program that recently used feedback to question its assumptions about what its clients actually wanted. NFP began in 1977 as a research project in Elmira, New York, whose studies determined that when a nurse regularly visits with a first-time mother for two years, providing a range of supports and information, the arrangement produces multiple benefits, including better birth and early child outcomes and improved parenting. The program has been externally evaluated for forty years and expanded into forty-two states and six tribal communities. But in 2015, CEO Roxane White and new Chief Communications and Marketing Officer Benilda "Benny" Samuels determined that even evidence-based programs need periodic innovations to reach new mothers and retain participation. So NFP decided to participate in Listen for Good, a systematic feedback tool that asks mothers about their experiences with the program — from whether they would recommend it to other new mothers to what they saw as its strengths and improvable areas. 

Some staff were skeptical that mothers in the program would want to participate. They worried that others wouldn't want to use their data-plan minutes to respond to the Listen for Good survey via text message. But when staff sent the feedback survey to ten thousand recipients, they received almost a thousand responses in twenty minutes. And the first thing they noticed was just how much the mothers appreciated the invitation to provide feedback; they saw it as a sign of respect. While those who responded provided positive feedback about the program overall, they also had innovative ideas for improvement, such as connecting participating mothers with one another (not just NFP staff); creating an app focused on NFP's print materials; and, counter to staff's expectations, asking to be able to communicate with the nurses both via text and in person.

These recommendations led to innovations now being tried at NFP, including creating a new feedback team that is not only engaging participants in the program but also inviting feedback from staff, volunteers, and partner organizations. NFP, the gold standard for evidence-based programs, has integrated feedback as a third leg of its measurement stool to unlock new insights and drive continuous improvement. 

Surfacing hidden problems | The Second Harvest Food Bank serves millions of people each year in Silicon Valley but never systematically solicited feedback from its customers until 2016. From its first feedback efforts in ten locations, Second Harvest learned that customers from different cultural communities were having vastly different experiences with food and service at the food banks — white and Latino clients were markedly more satisfied than Asian clients. This insight led Second Harvest to experiment with more culturally sensitive approaches to its work, including new-volunteer recruitment and training, food choices consistent with traditional Asian diets, and even a new location for food pickup, to better serve the Asian community.

Second Harvest, like many nonprofits, will likely never invest significantly in an expensive third-party evaluation, but rigorous systematic feedback has bolstered its understanding of client experiences and preferences. If the organization can improve clients' experiences, it will be better positioned to accomplish its mission of reducing hunger in local communities.

Giving voice to those who are least heard | Epiphany Community Health Outreach Services (ECHOS) is a nonprofit ministry of the Episcopal Diocese of Texas that provides health and social services to the growing population of immigrants and refugees in Houston. ECHOS provides a range of critical safety-net services, including English-language classes and, more recently, Hurricane Harvey relief services. Through its first efforts at customer feedback, ECHOS learned that clients were waiting excessive amounts of time, which made its services difficult to access.

ECHOS staff realized that the experience of its clients contradicted the organization's stated intention to treat all immigrants and refugees with respect. Consequently, ECHOS is changing its registration process so that clients no longer have to stand in line to be received. It is also instituting expanded hours and making workflow improvements to increase efficiency. In fairly short order, the organization has transformed how it manages the flow of people and created a more positive and respectful customer experience.

Iterations on Feedback

When I began my career in applied research thirty years ago, I was taught that client satisfaction surveys were useless. They were seen as "lite," in contrast to "hard," outcomes. Because of the power differential between nonprofits and their clients, evaluators assumed that satisfaction measures would always be positive and therefore not meaningful.

It's a new day. Across the social sector there is a growing recognition of the importance of being human centered — of putting the people we seek to benefit at the center of our problem-solving efforts. Those human-centered design principles should also apply to nonprofit measurement.

Many funders are already interested in connecting more with the communities they aim to serve, as well as finding new measurement tools. Indeed, a recent study by the Center for Effective Philanthropy found that foundation CEOs believe that listening more to the people they are looking to help is essential to their organization's success. The Fund for Shared Insight, the philanthropic collaborative that has been the driving force behind Listen for Good and other feedback efforts, has grown rapidly in the past four years, from six participating funders to seventy-eight co-funders, and counting.

U.S.-based funders like the Plough Foundation in Memphis, Tennessee, have found feedback to be a powerful tool for their grantees. "The organization had never asked [its] populations what they wanted," says Plough Foundation board chair Diane Rudner. She describes the learning from feedback, particularly from an organization serving people with developmental disabilities, as "so valuable... it’s amazing!"

The internationally focused Omidyar Network partnered with the Acumen Fund on "lean data sprints," in which the organizations gathered feedback from approximately thirty thousand customers of sixty-eight Omidyar investees in eighteen countries. Like Listen for Good, the Acumen tool uses the Net Promoter system, which involves a calculation of customer experience scores. Based on that first attempt at being more customer-centric in its measurement, Omidyar and its investee organizations generated actionable insights about each relevant sector (e.g., independent media, education, financial inclusion), as well as each organization's perceived strengths and weaknesses.     

Omidyar Network still relies on traditional impact evaluation when it can, and on elaborate dashboards to monitor the progress of the organizations and businesses it supports. But because the organization has added a third dimension, customer feedback, to its measurement stool, it's better positioned to let customer experience drive improvement.

Let me be clear: I am not arguing against monitoring or evaluation. They are both important tools. Evaluation helps us gain a deep understanding of what works and why, while monitoring helps us track our progress and provides useful signs for course correcting. But not every organization can invest equally in each leg of the stool. The advantage of feedback, when properly integrated, is that it is both information-rich and affordable. The insights, ideas, and preferences of our ultimate beneficiaries can unlock new possibilities for operational improvements, programmatic innovation, and more respectful engagement. 

As a mentor of mine once advised, "Let not the abuse of a thing be an argument against its proper use." It's time to stop denigrating satisfaction surveys and unleash the power of feedback in new ways. Let's strengthen our measurement tools so that they are reliable, comparative, simple to use, and capture both quantitative inputs and qualitative comments. And let's start listening to gain insight, improve, and innovate. 

Fay Twersky is the director of effective philanthropy at the William and Flora Hewlett Foundation and the former director of impact planning and improvement at the Bill & Melinda Gates Foundation.