So, What Does It All Add Up To?

Submitted by Carlisle Levine, Sherine Jayawickrama, and Tosca Bruno-van Vijfeijken.

International NGOs have well-established practices for measuring results at the project and program level. But how to measure results at an agency level across numerous countries, sectors and projects? What does it all add up to? Are there meaningful, cost-effective ways to tell?

NGOs pursue a variety of approaches to measure results at the agency level. Some approaches are top-down, starting with agency-level measures that align with a mission or strategic plan, and asking staff to track indicators associated with these measures. Others are bottom-up, starting with data already collected by country offices or programs, and aggregating this data for an agency-level perspective. Some focus only on program results, and others include finance, management, and other results. Some have custom-designed data management platforms and others use software like Excel.

Across this diversity of experiences, some key lessons stand out.

Shifting Organizational Culture

Agency-level data is well-used in organizational cultures that value data and reward evidence-based decision-making. Few NGOs currently have such cultures. Shifting an organization’s culture in a supportive direction requires concerted change management.

Understanding the Audience

Agency-level measurement systems that work well are clear about their purpose and audience. Some systems try to deliver benefits for too many constituencies. While these systems might try to meet the data needs of multiple stakeholders, they typically evolve to meet the priorities of headquarters-based stakeholders (i.e. boards, senior executives, marketing and communications staff).

Underestimating Time

Most NGOs underestimate what it takes to design, develop, and execute an effective agency-level measurement system. A comprehensive system with a custom-designed data management platform may take close to two years to design and an additional three years to produce quality data.

Focusing on Outputs

Despite the aspiration to measure “what it all adds up to,” most systems end up capturing outputs (rather than outcomes or impact) because outputs are more readily measurable. Some NGOs do aspire to measure outcomes in the future. For output data to be useful for learning and decision-making, it must remain specific and clearly defined once aggregated, and the link between it and related outcomes must be well established.  

Thinking at the Agency Level

International NGOs often work in project, sector, or country silos, and the process of developing agency-level measures is sometimes a rare opportunity to think and act across those silos. This can help staff recognize how their work contributes to the broader whole. The design process often reveals long-neglected M&E weaknesses, and prioritizes investments to improve M&E capacity.

So What?

We found that effective, well-utilized agency-level measurement systems are not easy to achieve. International NGOs must be clear-headed about what they expect from agency-level measurement systems, and how realistic those expectations are (given the time, resources, capacity and trade-offs it takes to get there). For these systems to be meaningful, well used, and worth the investment, an organization must have a well-defined strategy, a learning culture, executive buy-in and use, and sufficient data collection, management, and analysis capacity.

Carlisle Levine and Sherine Jayawickrama are independent consultants, and Tosca Bruno-van Vijfeijken is the director of the Transnational NGO Initiative at Syracuse University. Learn more about measuring results at the agency level at their InterAction Forum Learning, Knowledge and Leadership track workshop, "Measuring Agency-Level Results: Can it Work, and Does It Matter? Findings From an InterAction White Paper" on Tuesday, April 19.

Editor's Note: This blog is part of a series highlighting the various workshop tracks at Forum 2016. Check back every day until April 15 for a new blog!

Read more

Think small: What is the future for small NGOs? (Posted April 6, 2016)

Excerpt: It was an interesting, thoughtful study but, frankly, I was annoyed.  Its sole focus was big NGOs; all of the consultants’ interviews were with representatives of the 50 largest U.S. NGOs. What about the rest of us?  How do smaller NGOs prepare for the future when we are seeing more disruptive change, coming us at a faster pace, than ever before? Read more.

Three Myths About Passion That Are Keeping You Stuck (Posted April 8, 2016)  

Excerpt: As a leadership coach, "passion" is a word I hear often in my conversations with clients.  After working with dozens of young professionals in the non-profit world, I've found that there are three big myths that keep them from feeling fulfilled and happy in their careers. Read more. 

Top Ten Online Advocacy Mistakes (Posted April 11, 2016)

Excerpt: Do you want to succeed at online advocacy? Of course you do! Fortunately, thousands of us have been toiling away in this field for years. In the process, we've done just about everything wrong you can imagine. But you don't have to repeat our errors. Read More.

A Data Revolution for Development (Posted April 12, 2016)

Excerpt: We see it in every humanitarian crisis. When migrants crossing the Mediterranean reach land, they reach for their smartphones to tell others they are safe. But they can’t use those phones to access basic information on their rights and the services available to them. Read more.

Supporting Refugees’ Right to, Access to, and Conditions of Work (Posted April 13, 2016)

Excerpt: Every day, different nationalities arrive in Greece and other shores, with nothing to their name, looking to settle somewhere in safety to start their lives over. One means of sustaining themselves and their families is through informal and formal labor and employment. Read more.

Become a Better Problem Framer through Design Thinking (Posted April 14, 2016)

Excerpt: Design thinking is making its entry into the international development mainstream, with big design companies promoting human-centered design to solve the world’s most pressing challenges. However, the actual process of doing design is sometimes hard to grasp. Read more.

Open Learning for Inclusive Development (Posted April 15, 2016)

Excerpt: To understand how countries grow and develop, it is essential to know how they learn and what can be done to promote the translation of knowledge into practical tools. But here is the dilemma: having knowledge is not the same thing as using or disseminating knowledge. Read more.