Tip:
Highlight text to annotate it
X
Welcome to Making a Difference: What We Know About Using an Evidence-Based Approach.
It is our hope that after this session you'll have learned, or been reminded of, what a theory of change is
and what evidence is, what the scope of evidence is in veterans and military family programs,
and what can be learned from studies identifying the needs of veterans and military families.
We will also present a framework for continuous evidence building.
And lastly, you will hear from other non-Corporation veteran and military family serving programs
and what has been learned from their evaluations that might be of use in your work.
But before we start the official presentation,
Koby Langley from the Corporation for National and Community Service would like to give you the official welcome from the
Corporation and tell you how this session fits within the What We Know series that started in May.
Welcome, Koby.
Thanks.
Thanks, everybody.
Thanks, Education Northwest, for hosting the webinar for us.
Thank you to everybody who has called in today,
and thank you to the people that have participated over the course of this webinar series over the past year.
We've had well over 2,000 participants participate in these webinars over the course of the year,
and we're currently averaging about 100 to 150 participants every month.
The participants, as Debbie mentioned, are very disparate in terms of their background and area of expertise.
But we're all here committed to one thing, and that's the idea of leveraging national service
and volunteers to improve the lives of veterans and military family members.
We're also interested in leveraging the talent, skills, and abilities of our veteran
and military family community in leading those efforts in our communities.
Those are our two strategic goals.
This webinar series is a part of meeting those goals in that it hopes to improve the level of knowledge
and understanding around performance measures and outcomes and impact. And, more specifically,
to highlight--for our grantees, our potential grantees, our members
and the veteran military family community--our program design
and how it is that AmeriCorps grantees develop programs
and how members implement those programs in a way that measures success in outcomes
and provides impact to the communities that they serve in.
So this is the intent of this series.
It's an important topic area in that the more impactful and the more outcome-driven that we can be,
the greater level of support that the Corporation can gather for funding these kinds of programs.
We are a Federal agency.
Our appropriation is largely dependent upon our ability to provide proof points around outcomes
and impacts just as every other Federal agency is.
The good news is that our outcomes
and our impacts are a very cost-effective program design that has been improving the lives of veterans
and military family members for many, many years.
It just so happens that 2009 was when the Corporation was first asked to focus on this as a strategic priority.
So the slide that you have in front of you talks about the series and the sessions.
This information was gleaned through three, I would call, different formative areas of research.
The first one was in a general assessment and review of the veteran and military family programs
and grantees that have been performing work in this space over the past three years.
We reviewed a lot of the program activity, a lot of the annual reports that our grantees provide to us,
as well as some of the evaluations.
Not all of our programs have a requirement of a robust evaluation; but the ones that did, we took a look at those.
The second piece of research that we used to inform this series and, more specifically,
the last part of this series, was a review of 100 of our grantees over the course of the past three years--since the
Serve America Act was implemented--who were operating in the veteran and military family space--more specifically,
who were aligned with the activities that the Serve America Act asks that we align our veteran
and military families activities around.
The review of those 100 grantees informed the folks here around what I would call "emerging" practices--
not necessarily best practices--as well as some interesting challenges that some of our grantees have presented in terms
of their program design and evaluation and impact data.
The last thing we did was look at some of the more successful grantees over the course of the past three years,
and looked specifically around program design--not necessarily around outcomes,
but just how well did they tie their program design to impact and outcomes
and how well did they perform in comparison to some of their peers in the grant competition based upon how well they
articulated that program design.
So a lot of that information was gathered and provides the background for this webinar series.
This is not an exclusive list of the research that we did to perform the series.
Obviously, we reached out to a lot of our external stakeholders at work in this space.
We've even reached out beyond the folks that we normally partner with in the Corporation,
and are actually happy to have one of those presenters here with us today from RAND who will be talking about a
community evidence-based approach that supports military-connected kids.
I'm not going to steal Anita's thunder; but when she talks about it,
I'm sure folks on this call will be very interested in that program design and how it was crafted.
We're very lucky to have a full-time staff lead that works on program evaluation
and who actually helps lead the Corporation's work in moving our program evaluation in the veteran
and military family space from two very general outcomes--namely,
expanding the number of veterans that are serving and military family members that are serving in our programs,
as well as the number of programs that support the veteran and military family community.
Those were our two initial strategic goals,
and Carla was instrumental in tying back those goals to some very specific activities that could potentially provide
more specific outcomes by aligning
and prioritizing those activities in our grant-making competition with specific activities defined in the
Serve America Act.
Carla was very formative in making those changes.
For our grantees that are on the call
and for the folks that are looking at potentially becoming grantees in the future,
you'll notice that the Notice of Funding Opportunity has changed somewhat as it relates to the performance measures
around the veteran and military family space.
I would encourage everybody to take a look at some of those changes.
It's really just about narrowing down the scope of activities to areas of impact that we've been asked to focus on by
members of Congress.
Since we're a Federal agency, we do what Congress asks us to do.
It does leave a lot of space around program design for things that aren't specifically defined
or delineated in the Serve America Act, but it does help to narrow those activities down quite a bit.
Carla is a Senior Program and Project Specialist here with AmeriCorps State and National.
AmeriCorps State and National, as some folks know, has over 80,000 AmeriCorps members nationwide.
We work with over 2,000 grantees on an annual basis.
Those include nonprofit and state organizations.
Carla actually comes to us after having served as both an AmeriCorps member, as well as
an AmeriCorps VISTA member herself.
So she has experience both from the corporate level as well as the field level--having been both an AmeriCorps
and an AmeriCorps VISTA member.
She's actually managed several successful national service programs herself
and also provided consulting on performance measurements to a number of nonprofit organizations around the country.
So we're very lucky to have Carla here with us today.
We also have Dr. Chris Spera here today, who is our new Director of Research and Evaluation here at CNCS.
I guess I can't say "new" too much longer, Chris,
since he's been with us for over half a year now.
He is chiefly responsible for developing the evaluation strategy for the Corporation and informing the field
and our grantee networks of best emerging practices in that area.
Everybody here knows that we invest heavily in community-based programs
and prioritize interventions around behavioral health, school dropout prevention, disaster response,
military veterans, literacy, housing, mentorship, economic opportunity, and environmental stewardship.
Chris is responsible for ensuring that the Corporation's work around performance evaluation aligns well with all
of those activities.
He develops and overseas the agency's entire research portfolio, manages a robust staff here in D.C.,
and is actually responsible for representing the agency to OMB to ensure that we're aligned with what OMB wants us
to do in terms of measuring outcome and impact.
We're lucky to have Dr. Spera here with us and appreciate his area of expertise.
He's very well published, has over 15 peer reviewed publications, and (of course) has a Ph.D. in human development
and educational psychology from the University of Maryland.
So we're very lucky to have Dr. Spera here to talk with us today.
Lastly, but certainly not least, we have Dr. Chandra.
Anita comes to us from RAND. She was one of the key authors of one of my favorite studies about Operation Purple,
or Camp Purple as some people know it, back in 2008.
It really kind of blew the lid off of what I would call some of the assumptions about the impact of deployments on
military-connected kids. It really did a deep dive on the true impact that deployment had,
as well as some interventions that worked for military-connected kids to help deal with some of the stresses that they
undergo living the life of a child of a mom and dad who's gone for a year or two sometimes--
or sometimes multiple times over the course of their adolescence.
She currently co-leads studies in deployments of military families for RAND.
She focuses on community resilience and child health and wellbeing.
She has co-led studies on social, emotional and academic wellbeing of military youth and families.
She's actually also co-sponsored a longitudinal study on military youth and on deployed spouses funded by NMFA,
the National Military Family Association,
a formative nonprofit organization that most folks know about here on this call.
She currently serves as the principal investigator for the Deployment Life Study,
which is a DOD-funded longitudinal study.
She is also the principal investigator on an evaluation study of Non-Medical Counseling Programs,
also funded by the Department of Defense, more specifically, the Office of Military Community and Family Policy,
which is the formative Federal leader in this space in terms of impact and interventions for military-connected kids.
Dr. Chandra received her doctorate from Johns Hopkins in Bloomberg.
She also has a degree in Maternal Child Health from the University of North Carolina-Chapel Hill,
and has a bachelor's in Child Development from Tufts.
We're very, very lucky to have Anita here with us today.
So, with those introductions, I'm going to turn it over to Carla,
who will talk to you a little bit more about performance evaluation.
Hi, everyone.
We're going to start by getting back to basics a little bit and talking about theory of change
and really trying to get a handle on what we mean when we say "evidence."
When I started managing national service programs, I came into this having been a national service participant
and an English major.
I didn't have a lot of background in this stuff. It can seem a little scary if you don't know the lingo
and you aren't really sure what it is we're telling you in terms of the requirements that we have at CNCS.
So anyone can learn this material; you don't have to be an expert in evaluation.
So we're going to kind of just go back to the basics.
For some of you, this may really be basic; and you may have a good handle on this material.
But we wanted to make sure that we were all starting out with a basic foundational knowledge of what it is that we're
talking about.
We're going to talk a little bit about the theory of change.
The theory of change is basically a theory about how your program works.
It has three primary components: there's a problem, an intervention, and an outcome.
The problem, which we also sometimes call the "community need,"
is really just the negative condition that exists in the community--
the problem that you're trying to fix with whatever sort of intervention you're providing in terms of member activity
or national service participant activity.
When we talk about need, we're just talking about what is the extent
and severity of the problem--how prevalent is this problem in the community?
The intervention is the activity
or the set of activities that are delivered by national service participants, in the case of CNCS programs;
and those activities are delivered with some sort of design and at some particular frequency, intensity,
and duration depending on program design.
Then, those activities lead to some sort of outcome--a change in knowledge, attitude, behavior,
or condition in the beneficiaries that the program serves.
Theories of change are really about cause and effect. They're really about a cause
and effect relationship between the intervention and the outcome.
The idea here is that if I find a good intervention that's going to be effective, there will be a cause
and effect relationship between the delivery of that intervention and the intended change in knowledge, attitude,
behavior or condition of the beneficiaries.
So it's really an if/then kind of thing--if I deliver this intervention,
then this particular change is going to happen in the community.
There are a couple of different ways that data informs our theory of change.
We don't just kind of like make this up out of thin air, of course;
and so we look at data to help us figure out the problem
and figure out what is the best intervention to solve the problem.
The first way that we use data is to help us understand and document the need.
We want to know typically, again, how severe is the problem; what is the extent of the problem in the community?
So we get demographic information, we get various kinds of statistics about what the community looks like and
what the situation is in the community.
We also try to unpack a little bit why the problem exists.
If you don't actually know why a problem exists, then it's hard to solve it.
So when we think about learning about the community need, we're really trying to figure out,
"Why does this problem exist?" If it were easy to solve this problem, somebody would have already solved it.
So what have we missed?
What is the missing piece that is going to help us address this effectively?
And this is really important,
particularly for programs where you're needing to innovate because there hasn't been a lot that's been done that's
been tested before.
I think this is really true in the veterans and military families area.
There haven't been a lot of interventions that have been particularly well tested.
So it's important to figure out and really understand the problem as you're beginning to innovate and move forward.
Chris will be talking about that a little bit more later on in the presentation.
But we're actually not going to spend much time today talking about the need
and the problem, because the real evidence that we're concerned with in this webinar is evidence that supports the
choice of intervention.
As I said, there should be a cause and effect relationship between the intervention and the outcome.
So when we talk about evidence, what we're really trying to look at is,
"Is there actually evidence--research--that suggests that there is in fact a cause
and effect relationship between the intervention and the outcome?" So I said, "If I provide 'X' intervention,
then this outcome is going to happen as a result." Is there actually research that supports that that actually is true?
So is there research looking at the effects of particular interventions to see if they brought about the desired
change in knowledge, attitude, behavior or condition?
As I said, evidence helps tell you whether the intervention is an effective way to solve the problem.
We talked about that--when you're innovating, you start with a solid understanding of the problem
and then you select an intervention that you believe will be successful.
Over time, you gather evidence to determine whether or not you were right in your theory.
So, let's move on to the next slide.
When we say an evidence-based intervention--
and Chris will provide some more technical definitions of evidenced-based
and evidence-informed later--but basically when we're talking about an evidence-based intervention,
we're talking about service activities that are supported by research
and that demonstrate a high likelihood that those activities will address the need and result in the intended changes.
Evidence demonstrates that the proposed intervention--and here we're talking about approach, design
and dosage of the intervention--is likely to lead to the outcome.
It's important to understand that approach and design and dosage are important.
It's not enough to say, for example, "We think it's important to provide employment assistance to veterans."
That's kind of a broad class of intervention.
It would be like in medicine if you were saying, "Antibiotics are the way to help you when you're sick."
Well, one particular antibiotic might be the way to help you when you have one particular type of illness;
but it's sort of a broad class.
So when you think about what is the best way to provide a particular intervention--
what is the best way to provide employment assistance to veterans--you have to understand the problem
and you have to really design an intervention that gets at the heart of that problem
and that is provided with a frequency and an intensity and a duration that is likely to make a difference
and be able to achieve the level of change that you're trying to get at.
So that's important when you're thinking about evidence
and thinking about how you design an intervention that's based on evidence.
Next slide?
So how do we actually use this in the real world?
In a couple of different ways. If I am wanting to start a new program,
I've identified a problem in the community
and I think that I need to figure out some kind of intervention that's going to help solve the problem.
One thing that I can do is to look at what's been done already and what has worked.
And if there are in fact models--different interventions--that have worked, then I can think about adapting that
or replicating that to achieve the outcome that I'm looking for.
If I'm an existing program and I'm wanting to test my own theory of change
and really make sure that the intervention that I'm delivering is effective, then I gather evidence.
My evidence might tell me that my intervention is very successful.
It might tell me my intervention is okay, but maybe there are some places that I could improve it.
It also might actually tell me that my intervention doesn't work, and that's really important to know.
We don't want to continue doing things that don't work.
Sometimes people think that maybe it's a failure if evidence comes back
or you gather evidence that suggests what you're doing isn't working;
but, it's really not, because it allows you to change course and do something that might be more effective.
Okay, next slide.
So places that we find evidence--and there are levels of evidence. Chris will talk about that later as well.
Evidence can be performance measurement outcome data, like we talked about in the last webinar.
It might be results from an impact evaluation of your program.
It might be research that documents the outcomes of similar programs,
or it might be impact evaluations that document outcomes of similar programs.
There are lots of different types of evidence.
Some evidence is stronger than others, and so an impact evaluation is likely to be stronger
and more compelling than performance measurement data.
But where you are in the evolution of your program is also important.
You don't do an impact evaluation right out of the gate; it's not the first thing that you do.
You begin by gathering other kinds of data and building up over time to test your theory of change.
Chris is going to talk a little bit more about what that evolution looks like
and what the different levels of evidence are.
Next slide?
Good evidence--and this won't be news to anybody--good evidence is relevant;
so you're looking at interventions that are similar to the intervention that you are doing.
It's compelling--it shows a clear likelihood of success for the intervention. It's up-to-date. And it's objective--
so it's important that someone without a stake in the outcome of the study was responsible for that.
It's great if it was published in a peer reviewed journal or by some other credible research organization.
Next slide?
Some sources of strong evidence might be university or research organizations, professionals
and experts in the field that you're working in, looking at other programs and their descriptions
and the evidence that they have, as well as articles that look at multiple studies
and sort of tell you what works across a particular field and across multiple different interventions.
Next slide.
We've talked a little bit about theory of change
and about, sort of at a very basic level, what we're talking about when we're talking about evidence.
I wanted to show you a couple of resources where you can go to find more information about this
and other topics related to data collection and performance measurement.
The screenshot that you're looking at here is the Corporation for National
and Community Services' Performance Measurement landing page.
This landing page is the first stop in terms of information about our national performance measures
and how they roll up to our strategic plan,
but there are a couple of interesting places that you can get from this landing page.
If you look at the bottom right-hand corner of this page, you see a little box that says Program Measures
and icons for all of the different programs.
So, if you are an AmeriCorps program or thinking about becoming an AmeriCorps applicant,
you can click on that icon to get more AmeriCorps-specific information and the same for the other CNCS programs.
Once you get to the AmeriCorps pages, you can also find a link to this Performance Measurement Core Curriculum page.
The Performance Measurement Core Curriculum is an agency-wide curriculum that is suitable
for all of the CNCS programs.
It basically covers all of the basic performance measurement topics that we think applicants
and grantees need to know in order to meet our requirements for performance measurement,
and also to use performance measurement themselves as a management tool to improve
their own service delivery and performance.
We have lots of different topics that are available on here.
We have the basics of performance measurement--performance measurement 101 kind of information.
We have a theory of change.
What we have are all tutorials. You can take them online at your own pace.
Then we also have the slides on here and sets of skill building activities with facilitators' notes.
You can take the courses online to learn the material yourself.
You can also download the other supplemental materials if you want to train other people in your organizations
on this content.
If you go to the next slide, you'll see--in addition to theory of change--we also have sessions on evidence.
So if you want to know more about this topic, we have a whole tutorial on it--quality performance measures,
and then a couple of sessions on data collection and instrument design.
So I encourage you, if performance measurement is something that you are interested in learning more about,
these are really great resources to help you do that.
Now that we've talked about some of the basics, I am going to turn it over to Chris;
and Chris is going to put this back into the context of the veterans and military families area.
Okay, thanks, Carla.
Hi, everyone.
This is Chris Spera, Director of Research and Evaluation at CNCS.
It's a pleasure to be speaking today on the webinar.
As you may know, the Serve America Act was passed in 2009.
As Koby mentioned, the Serve America Act specified veterans, military members
and families as one of the six core priority areas for national service.
The Serve America Act defined the service activities most suitable for CNCS service programs addressing the needs
of veterans and military families which are depicted in this nice,
circular graphic depicting the three core areas of health, employment and education.
As you probably know, national service programs operate in three ways.
They create service or volunteer opportunities for veterans, military members and families
and can be a key strategy in the transition of military members from military service into civilian life who have
service as a core component of their being.
We actually did a recent survey of our AmeriCorps and Senior Corps members. What we found is
that 4.5% of AmeriCorps members are veterans and 11% of Senior Corps members are veterans.
What we hope to see is those numbers increase over time to get more veterans, military members
and families involved in serving through national service programs.
National service programs of course provide services to the population of veterans, military members and families.
On the next few slides, you'll get a sense of where those programs are being delivered
and the kind of programs that they are.
And then finally, national service programs provide a community coordination function
which is an important part of CNCS programs.
Next slide.
As we mentioned, national service programs focus in the area of employment, health and education.
These next few slides give you the landscape of current national service programs that involve military members,
veterans and families.
This slide indicates that 58 programs across AmeriCorps, VISTA
and Senior Corps provide volunteer opportunities to military members and veterans.
21 programs provide services to this population,
and another 17 provide a community coordination function in the veteran, military and family space.
Again, we expect to see CNCS programs involvement in this area only increase over time
as we grow our national service programs.
Next slide.
Helping veterans transition into the workforce is a key component of national service programs,
as you can see on this slide,
with 43 programs focusing exclusively on this particular topic of transition back into the workforce and relatedly,
25 focusing on education and certification topics such as the new GI Bill.
Furthermore, another 26 focus on expanding access and knowledge of VA benefits available to veterans.
I've actually done a lot of work for the Department of Veterans Affairs--a lot of research for them--
and one of the things I learned early on in my research with the VA is about only half of the nation's 22 million
veterans actually have interacted with the VA around benefits.
So it's a critical area; and many veterans, believe it or not,
are not aware of the benefits that they're entitled to based on their service.
So we think that's an important role that national service programs can play.
As you can also see on this slide,
wellness--including behavioral health--is the top area for national service programs working in this area.
Of course, issues such as PTSD and Traumatic Brain Injury have been central issues for returning veterans.
And actually, new research that I was involved in based on the DOD survey of health related behaviors has actually
shown some increases in poor coping behaviors such as alcohol use and prescription drugs.
The latest survey that just came out, as an example,
showed that one-quarter of Service members reported cigarette use in the past 30 days,
and about one-third of active duty Service members reported binge drinking in the last 30 days.
So obviously, wellness is a big focus for CNCS programs and working on the behalf of military members and families.
Next slide.
Wellness programs working in national service focus on the areas such as housing, morale boosting activities
and behavioral support. You can see how that disperses across Senior Corps, AmeriCorps and VISTA.
Next slide.
So that gives you a sense of the landscape of where national service programs are currently operating in the veteran
and military family arena.
I want to transition back to the evidence base and what we know.
As an applied researcher and evaluator, I've worked in numerous fields over my career.
My experience is that all these fields are really in different stages in their development of building an evidence
base for identifying and then documenting the outcomes of particular interventions.
As an example,
a field like education--which is a big focus area as you know for CNCS--has a more established evidence base than
something like disaster assistance--
although researchers in both of these fields are working *** building a knowledge base for what works.
In this presentation,
when I say evidence what I mean is that the intervention is effective, versus evidence of the need for a service of
an intervention. I just wanted to make that distinction.
Researchers in the military and veteran area are working hard at identifying interventions that have been effective.
And while the field is still growing and current evidence is limited,
there are really a few good examples of programs that have done pretty good studies, some causal in nature,
and have documented some good outcomes.
You're going to hear about two of them today: one that Anita is going to go over which is the Operation Purple Camp
and the second which I'll go over is the Minnesota National Guard Yellow Ribbon Reintegration Program.
Next slide.
Chairman of the Joint Chiefs of Staff, Admiral Michael Mullen,
has often talked about the "Sea of Goodwill" across America,
referring to communities stepping up to meet the needs of Service members and families.
So on this slide here are two good sources of information really documenting the need for national service programs
to do work in this area--so, good resources for you.
Next slide, please.
In my role as Director of Research and Evaluation at CNCS,
I've had the unique opportunity to be involved in many cross agency discussions so far.
Many of them have been facilitated by OMB or the Domestic Policy Council of the White House.
And actually there's quite a robust conversation going on right now in government about building a common evidence
framework across the Federal Government, sort of recognizing that we need a common lens and we need a common framework.
In fact, there are currently six tiered evidence-based programs
within the Obama Administration that really are at the
forefront of this discussion, including the Social Innovation Fund which is run by CNCS.
Next slide.
This next slide depicts a current draft of the evidence-based framework being used by CNCS and other Federal agencies,
such as the Department of Education, in their i3 Program or Investing in Innovation Program.
It's what we're using for the SIF Program--the Social Innovation Fund--at CNCS.
A first point to make on this slide is that there is a growing belief that evidence-based grant making should
subscribe to a model of higher evidence being associated with greater Federal investments.
So you can see the arrow on both the vertical
and the horizontal access sort of depicting that relationship between greater levels of evidence/greater investments.
In fact, the President himself has said in numerous speeches
that the government should scale up programs that have an evidence
base and reduce programs that have not established evidence of effectiveness.
I think that was very present in the recent budget that was released by the White House
for this upcoming fiscal year.
I'm just going to walk you through a little bit of this framework and make a few points.
Next slide.
Most programs, including a lot of programs operating in the military veteran
and family space operated by CNCS are at an early development, meaning that they are routinely collecting anecdotes,
good stories, developing lessons learned from implementing their intervention,
developing early tracking systems for monitoring performance measures.
I think one of the things that CNCS has done a really good job, really led by Carla over the last few years,
is developing good performance measures and training our grantees to start collecting them.
So we're pretty proud of the work our grantees are doing in this area, and we want to continue to strengthen that base.
Next slide, please.
At the second stage of this model is really where programs move from a developing program really to a promising one.
During this transition, organizations are typically establishing and documenting key outcomes, developing
and refining their theory of change that Carla talked about earlier,
and developing a Performance Management Information System to capture performance metrics.
What we would like to see over time is our CNCS grantees starting to progress along this continuum of evidence.
Next slide.
At the third stage of this model is the movement from a promising program to a proven one.
There's actually a lot of debate right now about when an intervention can be labeled proven.
But really the way it's currently talked about is that this model is based on programs
or interventions that have done a third-party evaluation using a causal model, either a randomized control trial
or a good quasi-experimental design with a matched comparison group.
Next slide.
And at this top level stage, or Level 4,
of this model is the movement from a proven model to one that should be scaled,
such that significant capital should be invested to maximize returns.
During this stage, organizations are conducting multiple third-party evaluations,
and in some cases cost benefit studies.
Next slide.
So for those of you on the phone that run programs,
hopefully you can think about kind of where you fall on the continuum and ways to advance to the next level.
What we hope to do from the Research
and Evaluation Office within CNCS is work pretty hand in glove with the field in progressing this continuum
and documenting the evidence base for our national service programs.
Next slide.
One thing--and Carla alluded to this earlier--
on this last point I wanted to make before we go to the two program examples is
that there are two terms we use:
one is evidence-informed and one is evidence-based--and just a little bit on the difference.
What we mean by "evidence-informed" is a program or approach that is based on
or adapted from a similar program that has evidence from an evaluation.
What we mean by "evidence-based" generally is when a program has evaluation findings for their own program
or approach.
Next slide.
So we're going to go into two programs that have been evaluated, that have documented some good results.
The first is Operation Purple Camp, which will be discussed by Dr. Anita Chandra from RAND
and the second is the Minnesota Yellow Ribbon Program which I will go over.
I'll turn it over now to Dr. Chandra to go over Operation Purple Camp.
Thanks, Chris.
And thanks to the Corporation for inviting me to participate in this webinar.
I just wanted to say a couple of things at the outset.
Certainly I've been working in this field for the last seven or eight years.
When we started working and trying to document the impacts of deployment on military families,
there wasn't a lot of research to really guide which programs worked and which didn't.
People wanted to make sure to address the needs of kids and families as quickly as they could,
which made a lot of sense.
But now we're at a place where we understand, kind of, the needs of these families,
and we're starting to really look more closely at how to evaluate programs in that context.
And it's helping us to really marry the high-quality programs with the specific and targeted needs of Service members,
veterans and their families.
So it's really an exciting time for program evaluation.
In addition, I'm glad that we're highlighting the Operation Purple Camp evaluation
because we're working on a couple of large-scale
evaluations that were mentioned at the outset in my introduction--one that's looking at big,
non-medical counseling programs and military family life counseling programs
and other kinds of veterans programs right now.
The Operation Purple Camp evaluation was on a relatively small scale.
It is a nonprofit program that is implementing Operation Purple Camp,
but it actually yielded some really critical findings that not only improved the camp in subsequent summers but also
gave real credibility and evidence to what matters for kids and families, even in week-long summer programs.
Next slide.
For those of you who aren't familiar with Operation Purple,
it is a one-week free summer camp program that was started in 2004
and administered by the National Military Family Association, which is a large,
nonprofit organization that works on behalf of Services members, veterans and their families.
The goal of this one-week program was to try and help military-connected kids get connected to each other,
particularly those from the Reserve and Guard component who may not be connected to other military kids.
Ideally, the underlying theory was that it would not only engage those kids in understanding military culture,
but it would improve their ability to communicate about what they're going through,
feel comfortable sharing those other experiences with other kids at camp,
and bolster not only their communication but their coping skills.
In addition, there was this overlay of fun.
They had funding sources from the Sierra Club and other organizations to build up outdoor education skills.
But also in the context of really helping kids to connect to each other
and understand what their families were going through was not unique,
that there were others experiencing those same things.
So our evaluation was really intended to understand how much a one-week camp program could improve kids' wellbeing
in those four outcome areas.
We wanted to understand the perspective from both kids and parents.
We also wanted to just understand how well the curriculum was being implemented.
It was a pretty flexible curriculum; but over the years,
Operation Purple was trying to understand what parts of the curriculum were working,
what should they keep from year to year, how were the camp counselors working out.
All of these kinds of things were all very unknown going into the Camp. They had been actually pursuing the Camp
and implementing the Camp and running the Camp for about six or seven years before we did the Camp evaluation.
So it was really time for them to understand what was working
and what was not--not only for their own program improvement, but in order to maintain funding
and sustain the program, particularly in fiscally austere times.
Next slide.
We used a couple of approaches.
The reason why I like this evaluation is because we still preserved the integrity of the program in the sense that
we didn't make big determinations of who could attend the camp or not.
First and foremost, we wanted to make sure that people had a chance to go to camp as they had gone in the past;
but there were always kids who weren't going to be able to attend the camp just because of enrollment numbers
or they just weren't able to attend.
So we used a method to actually match campers--people who went to camp
and those who didn't go to camp--but match them on characteristics that mattered: their gender, their age,
their family's deployment experience, whether they were active component Guard or Reserve.
It allowed us to really tease apart what was the impact of the camp.
In addition, we were able to not only capture kids before they went to camp but one month and three months later.
That was a really important component.
It was a fairly brief survey that assessed kids along those dimensions that I mentioned earlier,
but it really yielded a lot of critical information.
In addition, we also wanted to supplement it to understand about curriculum implementation;
so we asked some open-ended questions in the survey to understand what kids' experience was with the camp,
as well as what parents felt the camp provided to their children.
On top of that,
we added some of the information that the camp was already collecting in terms of After Action Reports,
what was happening in terms of on-the-ground implementation.
We used some of that data to pepper kind of and provide some context for the findings.
Next slide.
So one of the things, if you might remember, one of the key dimensions was our communication and coping skills.
What we found was that we asked kids about whether they felt better about talking to other military kids,
whether they felt that they could actually develop and implement strategies to talk about their feelings,
whether it was talking to a friend or an adult or writing those feelings down in a journal.
We have had several years of research, as mentioned at the outset today,
starting to document the fact that some of the kids who have experienced many deployments
and family deployments had started to have increase in anxiety symptoms, feelings of depression and so on.
So understanding how kids were able to learn how to cope was really critical.
What we found is that while there were some differences between youths, it wasn't really statistically significant.
But perhaps more importantly,
the camp parents--so the parents of the kids who were able to drop off their child at the camp
for that week's period--felt that the camp helped their young person know how to cope with those feelings.
Then in the subsequent follow-up periods,
parents felt that now their child was coming to them to express their feelings,
that there was a stronger bond in terms of talking about what it meant for that child
to experience a parental deployment.
Next slide.
We also, of course, wanted to capture the really rich kind of qualitative information that you can.
We weren't able to do full-on interviews, but we captured a lot of things through open-ended survey responses.
As you can note from these illustrative quotes,
parents noted a lot of the things that were also noted on the last slide--that the children were able
to talk more about their feelings, that Operation Purple allowed them to have those kind of discussions,
to know how to place their fears and anger, know how to identify healthy coping strategies.
Young people also said similar things--that it was okay to talk about it.
This is really critical in a space where we were trying to get kids
to reflect on the stress that they were experiencing.
They may not be needing yet full-on interventions or formal mental health services,
but they do need some kind of support to understand how to deal with anxiety symptoms
and feelings of restlessness and depression.
Next slide.
Another thing that was really important--particularly for Reserve and Guard families who, as many of you know,
have been really impacted by the last ten years in terms of the kind of engagements that their families have had in
the wars--it was important to understand how much the youths felt that they had been connected to other military kids,
how much they felt that they were part of a community, that they understood what military culture was like.
These were important learning opportunities that the camp afforded.
We did find that some change from the baseline survey to the survey that was one month later that youths felt that
they had more experience talking about military life,
that they had connected with people outside of their family who also had military connections.
That was really critical for a sense of calm and a total sense of community.
As a side note,
the National Military Family Association has explored ways of trying to connect campers afterwards through social
media, through online platforms, through other events. And many of the kids have maintained those connections,
particularly when they may be the only military child in a given jurisdiction.
Next slide.
So, in short, we've learned a lot from a fairly short survey--a fairly well-bounded and limited evaluation effort.
We understood that youth felt that they could cope better.
This was really important because there was a lot of research, including research that we led at RAND,
talking about deployment stress.
But we did not know if there were programs that were actually, truly affecting that stress, mitigating that stress.
We had theories about programs that worked, but we didn't have the hard data.
We also learned a lot actually about parents feeling that the camp provided a reprieve
and that there may be a reprieve kind of value to parents as well--something that we started to capture but not fully
because it wasn't a primary focus of the intervention.
But since that time,
the National Military Family Association has thought about ways to provide additional support for parents.
So it was a secondary outgrowth of the evaluation that was still critical.
There were other things that we noted that we would explore in future analysis which I think may be a part of other
kinds of family support programs--the benefits that it confers to kids to feel like they can be confident in sharing,
confident and independent in separating from their parents when it's appropriate.
Those kinds of things were really critical.
It gave us a lot of insight into process in terms of implementation issues.
There were concerns that there wasn't enough training for the people who came to the sites to observe whether the
curriculum was being adhered to, or there wasn't enough ongoing training to support the camp directors.
But since that time, Operation Purple has really invested a lot in quality improvement activities over
and above continuing to evaluate the impact of the program.
Thank you.
Thank you, Anita.
We have actually gotten a few questions for you already here in our Chat panel.
Participants, please go ahead and ask any more questions for Dr. Chandra while we ask a couple of these for her.
First, one of our panelists asked if your presentation would be posted for Operation Purple Camp.
So I was letting them know that the presentation that we're doing today is already posted on the Veterans
and Military Families Knowledge Network; but I also wanted to know,
is there a bigger presentation that the RAND Corporation has done or a report about this camp that is documented
and posted somewhere else?
Yes.
This question also asks,
"Would you be willing to give a presentation just to their own personal 4-H military liaisons or other groups?"
Sure, first of all, there is a full report that is available at www.rand.org.
You can search for it with either my last name or it's called, "Assessing Operation Purple." That's a full report.
It's got many more data and findings, lots of tables and charts.
We've also included the instruments that we used, if people find that interesting.
Everything is always free/downloadable on the RAND website.
So we have a lot of not only evaluation studies,
we have a compendium of innovative evaluations that are ongoing right now called, "Innovative Practices."
Then we have a lot of the research studies as well, if people are interested,
in terms of military family/veterans issues and so on.
So please take advantage of those resources, and they're all free/downloadable.
If you don't find anything, you're more than welcome to e-mail me; and I can locate them for you.
I'm more than happy to share this presentation
or a broader presentation with any audiences that would be interested--4-H or other youth programs.
We're very happy to do that.
And I think there are some lessons learned about how we designed this particular evaluation that may be particularly
relevant to programs like 4-H.
Thank you very much.
We also got a question from Jake Cummings. He said, "Do you think grantors ever value practical
(i.e., the children coping better), versus statistical significance in making funding decisions?"
It's a great point. I know that Operation Purple
and the National Military Family Association really needed these data in order to sustain the program.
They've had to move forward with a more limited scope in terms of a number of camps, for example,
that they're offering this summer.
But there was a real challenge in terms of their funders continuing with this program just because, obviously,
what is measured often is what gets funded--as many of you know on the phone.
So I think the funders were compelled by the statistically significant findings;
but they were also compelled by some of the trends that we saw--some of the qualitative work that we saw.
I think that they wanted to know that there was something that they could hang their hat on, and even one
or two of these key findings about coping was enough.
Even if we didn't have statistically significant findings across all four dimensions,
things were sort of in the right direction. So they valued that,
and I think it started to put some practical parameters on what the Camp can and can't do.
The other thing for Operation Purple that I think is important to note,
particularly for any of you who are running programs that are short-term,
is we wanted to also be realistic about what a one-week program could reasonably impact.
The fact that we were able to see even findings one month later suggests the value of the program
and also suggested opportunities for expansion or booster sessions and that kind of thing.
So I think that the funders that have supported Operation Purple have been interested in those kinds of findings.
For the actual sites for Operation Purple, and mainly their grantees as well, they volunteer to be a site,
but sometimes there's a payment sort of hybrid method.
A lot of 4-H sites, for example, Boys and Girls Club, YMCA sites become Operation Purple for a week or two;
and I think they valued the implementation data because they wanted to obviously improve programs
and know that they were making a difference.
So, even something as sort of relatively smaller than an evaluation as we did in this particular study,
had that kind of value.
Great, thank you so much.
Those are the questions that we have so far for Dr. Chandra.
We can turn it back over to Chris for his next set of presentation slides, but feel free to continue to ask questions.
Dr. Chandra has agreed that if there are more that come in for her after she's left the phone call,
she'll answer them via e-mail and we'll get them out on our edited chat and posted to the Veterans
and Military Families Knowledge Network.
Thank you.
Great, thanks, Anita. That was a fabulous presentation
and I think is a great example of the kind of work that some of these programs are trying to implement in the field of
national service.
She talked a little bit about deployment stress, particularly on kids.
But the next program I'm going to talk about is in a similar area of dealing with deployment stress but more focused on
National Guard members and their families as those folks return home.
I'm going to briefly talk a little bit about the Minnesota National Guard Pilot Reintegration Program
or otherwise known as Yellow Ribbon. You may be familiar with it.
I'm going to speak on behalf of two of my former colleagues at ICF International who were the PIs on this study.
They would have loved to present today, but they had another commitment,
so I'm going to present on their behalf.
This is a pretty interesting study and program.
I think most states have been implementing a Yellow Ribbon Program over the last few years,
but, as far as I know, this is the only state that has done a systematic evaluation of the Yellow Ribbon Program
or the Reintegration Program.
I think it's a fabulous effort in Minnesota and a really good study.
It really stemmed from the return of a pretty large group of soldiers from Iraq in the summer of 2007 that had just
returned from a 16-month deployment which, if you know anything about deployment cycles, is quite a long deployment.
It was actually unexpectedly extended by four months,
compounding the challenge of the deployment experience for both the unit members and, of course, the families at home.
So the Minnesota National Guard implemented this program called Yellow Ribbon.
It included a variety of formats for soldiers and their families, including workshops, briefings, discussions,
round-robin service provider stations and interviews.
But it was all around a program to help ease the transition back from deployment into their communities for
both the soldiers and the families.
Next slide, please.
This slide depicts the components of the program.
You can see there were generally four different components of the program,
including an initial training event that was done in a four-hour block when the soldiers returned home
and then subsequent trainings post deployment including a 30-day training, a 60-day training and a 90-day training.
For those of you that have served in the military or know someone that has served in the military,
one of the downsides during this time as folks come back is they're often given a lot of post-deployment trainings.
As you can imagine, they're just real excited to be home; they can't wait to be with their family.
They're often inundated with information, and that was one of the challenges that this particular program had--
can they develop a program that really puts home the salient points to help folks transition back
and also follow-up with them over time in the 30-, 60-,
and 90-day points which are critical points in the transition both, again, for the soldiers
and the military families in which they reside.
Next slide, please.
So various outcomes were gathered from soldiers and family members. They were compared, as you'll see in a minute,
to similar populations that were part of other states' reintegration programs, as well as soldiers
and families that did not receive any reintegration support.
A soldier and family member's survey gathered data on reintegration outcomes such as the quality of relationships,
wellbeing after returning home, and then coping behaviors--both positive, like exercise, and negative ones,
like alcohol use--as well as the transition back to work.
Next slide.
The Minnesota National Guard also wanted to gather some pretty direct feedback from soldiers
and family members on the training,
so there were post-training surveys that were done immediately following the sessions.
You can see, the data was gathered from over 5,000 soldiers
and almost 1,000 family members completed these surveys; so the sample size was quite large.
Next slide.
So as I mentioned, the treatment group was compared to two groups and a host of relevant outcomes.
Essentially what was found is that compared to soldiers and family members from other states,
a larger percentage of those in Minnesota reported receiving reintegration materials--so that was good.
In turn, higher percentages of Minnesota soldiers
and families reported attending post-demobilization reintegration training.
They also tended to have better outcomes post deployment compared to the comparison groups.
What the study found essentially was those who attended this particular program reported a less severe worsening
of issues like anger, alcohol consumption and financial status.
So those were all significant differences between those who received the program and those that did not,
as well as comparison to soldiers and family members from other states.
Next slide.
I think this program was a really good example of a way to provide soldiers
and family members with post-deployment support. It also measured a good host of outcomes
and documented some important findings that I know the state incorporated into their program going forward.
Then other states inquired about doing evaluations of their own.
Next slide.
Let me open it now for any other questions on either Anita's example or the Minnesota one,
and then we'll conclude with some concluding remarks.
So far, Chris, in the chat feature we've had one question. It goes back a little bit further.
Joanna had asked Carla if the theory of change model was also used at SAMHSA.
I'm not really familiar with what SAMHSA uses.
The idea of a theory of change is a pretty universally understood concept in this work.
We tailored our training to be very specific for the Corporation for National and Community Service volunteer programs.
Chris, you may have a better sense of what SAMHSA actually does than I do.
I know SAMHSA spends a lot of time also training their grantees on theories of change and so forth.
I doubt they use this particular set of slides and way of depicting it,
but it's a pretty common framework for depicting theories of change.
Thank you.
We got another question from Jake Cummings again.
He said he was curious about what alpha level was used to determine significance.
That was both for you and Dr. Chandra.
I think at the Minnesota National Guard it was P less than 0.05, so a 5% chance.
Yeah, for the Operation Purple it was P less than 0.05, but we tended to only really push on findings
at P less than 0.01. For people who like our stuff, we did a lot of corrections
and sort of weighting so a lot of things that might have looked significant fell out because we were pretty stringent
in our threshold.
Thank you--those are the only two questions that have come in so far.
Okay, great--so I just wanted to wrap it up.
I think Koby Langley had to drop off to take care of another issue or attend another meeting.
I wanted to thank everybody for joining today.
We hope that you got a better sense of the landscape of what the national service programs are doing in the area
of military veterans and families.
I think the two programs that were covered were really good illustrations of how to do good evaluations
and document outcomes in this area.
I hope the work that Carla presented on theories of change
and logic models really depicted some of the good cornerstones of really delineating program interventions
and measuring and documenting outcomes.
I believe this concludes the fourth in our series of webinars on military veterans and families,
but you can be certain that we'll back to offer more information to this group.
I wanted to thank everybody for joining today's call.
Thank you very much, Chris.
This is Debbie Ellis again, and I just wanted to thank you all for joining us today.
I'll remind you about the Veterans and Military Families Knowledge Network.
It contains the information that Carla told you about earlier about performance measures,
and you can find that there as well as all of our past webinars
and the information that will be gained from this webinar today.
We'll post the recording as well as the closed captioning and edited chat
and transcript for you on the
Veterans and Military Families Knowledge Network as soon as we get those things back.
If you haven't joined, we will send out information for joining this with our thank you note.
This is a really great place to connect with others who are doing similar work as you are.
You'll find lots of data and resources and documents available there.
The next slide will just let you know a little bit about the planned next webinar series we're going to do.
We're looking into giving a series of three sessions on military families called, "We Serve Too."
Our first session we're hoping will be the last Wednesday in July: An Overview of Military Families,
followed up by a closer look at supporting and engaging military children and then supporting
and engaging military spouses.
So that should be really exciting
and engaging information about those two groups of family members that we try to serve.
And then with that, we can just go on and say thank you very much for attending today.
I want to thank Koby, Emily, Chris and Carla from the Corporation
and Dr. Chandra from the RAND Corporation for their participation today and the information they've shared.
And thank all of you for joining us.
Please fill out the information in the evaluation.
It will be helpful for us in making these sessions more relevant and pertinent for you in upcoming topics.
Thanks so much and have a great afternoon. 794 01:06:07,000 Bye-bye.