Americans have built an enormous nonprofit infrastructure around the men and women who served in the military.
There are organizations helping veterans find jobs, navigate benefits, improve their physical and mental health, reconnect with their communities, pursue education, overcome homelessness and make the transition from military to civilian life.
More than 40,000 veteran-focused nonprofits have operated across that landscape, according to previous research cited by Time.
Yet the sheer number of organizations, and the billions of dollars that flow through the broader veteran-support ecosystem, raises a deceptively simple question: How do we know which programs actually work?
For many veteran service organizations, or VSOs, answering that question can be far more difficult than it sounds.
Organizations routinely track how many veterans attend an event, receive a service or participate in a program. They can count volunteer hours, distribute satisfaction surveys and collect powerful testimonials from veterans whose lives have been affected.
But those measures don’t necessarily answer the harder question: Is a veteran actually better off because of the program?
It’s a distinction that Team Red, White & Blue has spent roughly a decade trying to understand.
“We knew from the very beginning that we wanted to be a very data-driven organization and look at outcomes,” Team RWB Deputy Director JJ Pinter told Military.com.
The result is an effort that has moved Team RWB beyond simply counting participation and toward measuring changes in veterans’ lives over time, an approach its leaders believe could also help other organizations across the veteran community better understand whether their programs are producing the outcomes they promise.
A Massive Veteran Nonprofit Ecosystem
The size of the veteran nonprofit sector has been a concern for years.
A 2016 Time article, citing research from the Center for a New American Security, reported that there were more than 40,000 veteran nonprofits. Seventy percent reported less than $100,000 in annual revenue, while fewer than 1% reported more than $1 million.
The report described a highly fragmented ecosystem filled with small organizations competing for funding while often lacking the resources necessary to consistently measure their impact.
The issue isn’t necessarily whether those organizations are trying to help. Pinter emphasized that most are.
“I think most veterans organizations or VSOs are run by people with good hearts that genuinely want to do the right thing,” he said.
But intentions and outcomes aren’t always the same thing.
“In some instances, they don’t know how to measure outcomes, and in some instances, it’s just too hard,” Pinter said.
Team RWB’s own experience illustrates just how difficult that process can be.
“It took us 10 years working on this,” Pinter said, “to get it to the point that we could actually use it to help run the business.”
Counting Veterans Isn’t the Same as Measuring Impact
The distinction comes down largely to the difference between outputs and outcomes.
An organization can count how many veterans attended an event, how many meals it provided or how many people participated in a program. Those are outputs.
An outcome asks what changed because the veteran participated.
Did their health improve? Did they become less isolated? Did they develop stronger relationships? Did they find greater purpose? Did the program produce a lasting improvement in their life?
That question has become increasingly important to donors as well.
Team RWB Executive Director Mike Sullivan said philanthropic organizations are becoming more discerning about where their money goes.
“What does your data show in terms of the outcomes of where our money’s going?” Sullivan said, describing the kinds of questions funders increasingly ask. “Because we really want to have an impact.”
For Sullivan, that also means Team RWB’s most important organizational measurement isn’t simply membership, event participation or fundraising.
Number one is impact, he said. We’ve got to be able to show the things that we’re putting effort toward programmatically are doing the things we want them to do for veteran health and wellness.
That’s particularly important for an organization such as Team RWB, which uses physical activity and community as primary mechanisms for improving veterans’ well-being.
Sullivan acknowledged that the model can appear simple from the outside. “You’re just a running club that could meet in any park,” he said, describing one perception the organization has faced. “Why should we give you money? Anybody can do that on their own.”
Proving that those activities actually produce meaningful changes requires something more.
Building a Way to Measure a Veteran’s Life
Team RWB eventually developed the Enriched Life Scale, or ELS, working with researchers including Syracuse University’s Institute for Veterans and Military Families.
The scale was designed to measure five dimensions of an enriched life: physical health, mental health, genuine relationships, sense of purpose and engaged citizenship.
Research validating the original 40-item ELS was published in peer-reviewed journals, including Translational Behavioral Medicine and Frontiers in Psychology. But developing a scientifically valid instrument turned out to be only part of the challenge.
Team RWB had to create something veterans would actually use in the real world. The original 40 questions were simply too many.
“The 40 questions were too long,” Pinter said. “We couldn’t get people to take it.”
Team RWB went back to work and ultimately shortened the instrument to 18 questions while maintaining the statistical standards the organization wanted. Then came an even bigger challenge.
A single score tells researchers only so much. To determine whether Team RWB was actually influencing someone’s life, the organization needed veterans to complete the assessment repeatedly and needed to understand what those veterans were doing between assessments.
That required years of building the digital infrastructure necessary to connect ELS results with other information about how members interact with Team RWB, including the events they attend, how frequently they participate, their digital engagement and other characteristics.
“If we just had ELS scores in a silo, it wasn’t really going to be that useful,” Pinter said. Now, Team RWB is beginning to have enough longitudinal data to ask the question it set out to answer years ago: Are veterans getting better?
The Data Is Beginning to Provide an Answer
The answer so far appears encouraging. Team RWB tracks veterans who have completed the ELS multiple times, with assessments at least 90 days apart.
Pinter showed Military.com the internal dashboard the organization’s leadership uses during its monthly business reviews. Team RWB had set a goal of obtaining at least 500 veterans with multiple qualifying responses and had already surpassed it. Pinter said the organization expected that figure could approach 1,000 by the end of the year.
But getting veterans to complete the assessment is only the first step. The next question is whether their scores improve.
Across veterans for whom Team RWB has multiple measurements, roughly 60% are showing improvement, according to data Pinter presented to Military.com.
That number requires context. Someone who enters Team RWB with an ELS score of 94, for example, has little room to improve. Maintaining that veteran’s already-high level of well-being may itself represent a positive outcome.
So Team RWB is paying particular attention to veterans at the other end of the scale.
An initial ELS composite score below 60 is considered “unenriched,” according to Pinter. Team RWB established a goal of increasing scores among those veterans by an average of 15%. So far, the increase is closer to 18%.
Those veterans also receive particular attention because the data can help Team RWB identify people who may benefit from additional resources.
We want to make sure we’re improving everybody, Pinter said, but then we want to focus on those.
Turning Measurement Into Action
The potential value of the data goes beyond proving that Team RWB works. It can help the organization determine what works, for whom and under what circumstances.
Because Team RWB can examine ELS scores alongside other information about member participation, it can begin looking for predictors of improvement. It can also start to answer questions like:
- Do veterans who attend more events improve more?
- Are certain kinds of activities associated with better outcomes?
- Does in-person participation produce different results than virtual participation?
- Does the frequency of engagement matter?
Those are questions Team RWB can increasingly test instead of simply speculate about.
“If we hadn’t taken the time to put data systems in place, we would ask these questions, but we wouldn’t be able to answer them,” Pinter said.
One application is already helping connect veterans with additional care. Team RWB added a question about unmet mental health needs to the end of its assessment. If a veteran indicates an unmet need, the organization can ask whether that person wants a referral and help connect them with outside resources.
That allows Team RWB to remain focused on its own mission while helping veterans find specialized services elsewhere.
The organization is also beginning to incorporate other measures into its evaluation of impact, including a physical fitness assessment and activity information from wearable devices.
The goal is to triangulate multiple sources of information rather than rely on a single survey. Eventually, Team RWB wants to use that information to provide more personalized experiences to members.
Pinter said a future version of its app is being built to use available information about a veteran to provide more relevant and contextual recommendations.
A veteran with exceptionally high scores may be encouraged to volunteer or take on additional leadership opportunities. Someone struggling in a particular area could instead be connected with programs or resources suited to that need.
A Model Other Veteran Organizations Can Use
Team RWB leaders are quick to acknowledge that what they’ve built wasn’t easy. It required years of work, investment in technology and research, support from funders and an academic partnership.
Pinter also credited Syracuse University’s Institute for Veterans and Military Families and former Team RWB research director Caroline Angel with helping make the work possible.
“We could not have done this without Syracuse and the IVMF because we had to have an academic partner,” Pinter said.
That may also explain why sophisticated outcome measurement remains beyond the reach of many smaller veteran nonprofits.
When 70% of veteran nonprofits in the earlier CNAS analysis reported annual revenue below $100,000, building research infrastructure, technology systems and longitudinal datasets can be a daunting proposition. But that doesn’t necessarily mean every organization needs to build its own system.
Team RWB has made versions of the Enriched Life Scale available for others to use, and Sullivan said other veteran organizations have already used the tool.
“We want people to use it,” Sullivan said. “That’s going to help them validate what they’re doing.”
That may be one of the larger lessons from Team RWB’s experience.
Every veteran organization doesn’t need a decade-long research project or sophisticated data platform. But organizations can begin with a much more fundamental question:
What change are we trying to create in a veteran’s life, and how will we know if it happened?
That question matters not only to donors deciding where to invest their money, but to organizations deciding where to invest their own limited people, time and resources.
It also means accepting that data could reveal that a popular program isn’t producing the expected result. For Team RWB, that’s part of the point.
The organization isn’t collecting data simply to produce a better fundraising pitch. Leaders say they’re using it to decide what to do more of, what to do less of and where veterans may need different support.
And Sullivan cautioned against confusing measurement with the mission itself. The data is a tool, not the objective.
Team RWB still revolves around veterans meeting one another, exercising, building relationships and finding community. Measurement is intended to help the organization understand whether those experiences are producing meaningful results.
In a veteran nonprofit ecosystem containing tens of thousands of organizations competing for limited dollars and trying to solve some of the most difficult problems facing those who served, that distinction could become increasingly important.
Because counting how many veterans an organization reaches answers one question.
Knowing whether their lives are better afterward answers a much more important one.
Read the full article here
