Keep your impact measurements honest, even when you don’t want to

This edition of The Organizer is written by Krystyn Tully, CEO and co-founder of Entremission.

In university, my least favourite professor was teaching my favourite class. This man had a legendary temper. In first year, he tore into a student for recording a lecture, not noticing the student had limited sight and needed the audio to study. He bullied another student into confessing his virginity to the entire class. You never knew what he’d say or who he’d try to shame next.

My turn came in a fourth-year senior seminar. The professor was giving a lecture on the power of brands. To illustrate the difference between a brand and a product, he used the example of Nike.

At the time, the Nike brand seen in advertisements was synonymous with strength and excellence. In the news, meanwhile, Nike was being criticized for human and labour rights violations overseas. The reality of how the product was made was very different from the brand.

Just because it sounds good, doesn’t mean it is good

I’d never heard of a “brand” before, and I found the topic fascinating. Our small, informal class of about fifteen people sat around in chairs listening as the professor spoke. He lectured, and we took it in.

All of a sudden, the professor stopped talking. He looked straight at me, and he barked, “Tully. You have a look on your face. What are you thinking?”

My heart sank. The teacher with the famous temper had just turned his sights on me. Everyone else was quiet. I felt my mouth open, and words came out.

“I’m trying to figure out how you can know what you know and be wearing those shoes,” I said, pointing at his sneakers with the popular logo splashed across the side. The room got quieter.

It wasn’t a rhetorical question. I didn’t mean to be provocative; I was genuinely curious. Maybe another professor would have given me the benefit of the doubt, or laughed, or talked to me after class. This one got angry.

“Get out,” he said. “Leave.”

So I left. As I walked down the hall and home through Allan Gardens, one question kept rolling around in my mind: Knowing what we know, how can we do the things we do?

Years later, I still wonder. And I still think it’s a great question.

If you do social impact measurement, the most important question is:

Social impact work is about — you guessed it — impact. It behooves us to pay attention to our actual impact (our product), and not just getting caught up in the story we like to tell (our brand). 

The whole point of impact analysis is to try to spot the connections between what we meant to do and what actually happened. You ask big questions, like: 

• When did our knowledge, intentions, and actions create positive social change? 
• When did they create harm? 
• When did they achieve nothing at all?

Organizations define and collect impact data because human intuition is not enough. Our guts can lead us astray. 

If you want a real, positive impact, don’t ignore the messy bits

Consider these lofty mission statements:

“[We] give people the power to build community …”

“[Our] mission is to help humanity thrive …”

“To enable people … throughout the world to realize their full potential.”

What kind of organizations did you picture when you read those words? Maybe a community engagement organization, a health or economic assistance organization, and an education organization?

They each sound like they represent organizations doing great, important, respectable social change work. 

Except these mission statements don’t come from social impact organizations. These mission statements belong to Facebook (a social network), Asana (project management software), and Microsoft (productivity software). 

Not one of these companies would continue to exist or pursue their mission without adequate funding, support, or status. Yet they talk like nonprofit organizations committed to social change. 

Facebook, Asana, and Microsoft and countless other organizations choose to see themselves how they want to be seen: as good and moral companies. They tell stories that reinforce this image. They present case studies, testimonials, and other impact information that paints a picture of their ideal customer, and then they ignore the rest. 

These companies aren’t the only ones captivated by their own social impact tale. The language of social change is easily appropriated by people who are not changing much at all. It’s remarkably easy to do: just focus on the mission statements and anecdotes that paint an idealistic picture of your work. 

Good impact measurement doesn’t start with a narrative

Good impact measurement doesn’t prop up a pre-written story. Real impact measurement seeks to understand what has happened, regardless of whether it fits a narrative. In the data collection stage, information is just information. It all helps. 

When we pick and choose the information we want to look at, it’s hard to know the truth. There is a huge gap between what humans know and what we actually do. Intellectually, we understand complex issues and support big values. Then we act in ways that contradict what we know and believe. 

If we look only at our own intentions and values, we miss out on all the real ways our actions affected the world around us. We choose illusion over reality. 

It takes courage to look at the messy bits of your organization’s work, especially when it means recognizing the moments when good intentions led to harmful outcomes. It can undermine your idea of who you are or the truth of your organization’s mission. 

You’ll survive. 

Acknowledging things the world doesn’t want to see is part of social impact work. The willingness to look, to learn, and to change ourselves is what separates real social change from illusion.