Tip:
Highlight text to annotate it
X
>> BRENDAN KELLY: Good Morning. Thank you all for coming to our first Break Out session.
This session is entitled Career Pathways Evaluation Design. With us we have three esteemed colleagues
all from Abt Associates: Howard Rolston, Alan Werner and Laura Peck. Their bios are in the
program book so I wonít say all the good things I could say about them, and instead
will reserve time for presentations and then questions and answers. First will be Howard
Rolston. Howard is going to talk about the ISIS Evaluation Design.
>> HOWARD ROLSTON: Is this on? Okay, good. Thank you, Brendan. Itís a pleasure to be
here. Now, bright lights and my brain donít get along too well, so Iím mostly going to
be looking down. But if you need to get my attention, you can just say, you know, ìYo
Howardî or something like that and I will look up. So what I hope to do is to accomplish.
Oops, this is the wrong, this is, this is, oh, okay. Thatís all right. Should I go back,
or just leave it alone? Why donít I just, Iíll start talking and when itís ready to
get to Vi-slides, just let me know.
So what I want to do this morning is to try to accomplish two things. One is to briefly
describe the subject matter. What is a Career Pathways Program? The second thing I want
to do is having described sort of in general what a Career Pathways Program is and given
some examples of the kinds of things that are involved is to say how the Career Pathways
conceptual framework drives the design of ISIS. ISIS, for those of you who donít know,
is a project called ìInnovative Strategies for Increasing Self-Sufficiencyî which is
sponsored by ACF and evaluation is being conducted by Abt Associates and a number of research
partners.
So in addition to describing the conceptual framework that underlies Career Pathways is
for the purpose of saying how it undergirds the ISIS Evaluation Design. I hope it will
also serve as an introduction to this session because itís relevant to both of the following
presentations too. So to do this and what will be somewhere in the range, I hope, of
15 to 20 minutes, I will at the outset say be oversimplifying, and I donít think that
that will create too much of a problem, but if it does, hopefully, it will be corrected
by others, or in the discussion part of the session.
So first I wanted to sort of mention what I think are sort of three critical, if you
like, facts that are central to the ISIS Design. First, the, thereís a general peer Career
Pathways conceptual framework which is very central to understanding what Career Pathways
program is and also to understanding the, why weíve designed the evaluation as we have.
Itís not a program model. Weíre not operating with a program model that undergirds all the
programs that are in ISIS, they follow different program models all under the general Career
Pathways framework. So I think one way to get at this is to think of career pathways
is a systematic set of responses to the diagnosis of a problem. So itís a broad theory of change
that is, undergirds a whole set of potential responses. A response being a program that
falls under this framework. The third thing that is sort of in another dimension is that
the design, the basic impact design of ISIS is a random assignment study. So it randomly
assigns individuals who are eligible for a program, either to have access to that program,
a Career Pathways Program, of course, or to have access to other services in the community.
Itís whatís known as a single arm design. Itís not as if thereís an multiple different
programs you could be assigned to. Here, youíre assigned to the Career Pathways Program, or
youíre assigned to the control condition, which is access to other programs in the community.
So what these will mean and that Iíll talk about several times in the course of the discussion
is that each program within ISIS; the nine programs that I reference there, will be the
subject of its own evaluation; that there will be some cross-sight evaluation subjects,
but the core focus will be at the individual site. In describing sort of the, the Career
Pathways Program framework that I hope it will become apparent why weíve moved in that
direction. So to describe the problem and the diagnosis which Career Pathways is a set
of responses, Iím going to tell what I believe is an accurate, but vastly over-simplified
story. But I think the story will be useful at getting at the heart of what the substance
is.
So once upon a time there were, or at least in the neighborhood I grew up in, community
I grew up, something called junior colleges. And these junior colleges were like senior
colleges in that they had a strong, but not exclusive liberal arts perspective. Junior
colleges served or were intended to serve as less expensive feeders to senior colleges.
Especially for those who were perhaps less prepared for college, but nonetheless college
bound. There were typically two-year schools and the expectation was at the end of two
years people would transfer from a junior to a senior college. Over time these schools
became more occupationally focused, as did their sort of bigger brothers too, the senior
colleges. And more and more adults seeking to improve labor market outcomes started to
attend them.
And I think on average if the notion that originally the junior colleges were 50, but
who might not be fully college ready, and were less prepared, even more unprepared people
seeking to improve their economic prospects entered these schools. So the problem with
this is the approach the colleges were not entirely well-suited and it did not work so
well for adults seeking to increase their skill level. And the big problem that emerged
over time is that there were extremely low completion rates. Not only were there low
completion rates so that many adults attempted to improve their skill levels, but did so,
were unsuccessful in getting the skills, let alone improving their labor market outcomes.
Those that did usually proceeded very slowly so they spent a lot of time doing that.
So this problem gave rise to a lot of diagnosis. All of which I think have important truths
in them. First of all, one notion was that the programs were too sequential. The expectation
was that for people with low basic skills that they would first remedy those basic skills
before they could get into the occupational part. A second problem is the programs, the
programs offered were often incompatible with work and that work conflicts, especially,
for people who might not have regular work schedules, created difficulties in completion.
A third problem identified is that they were too opaque. It was too hard to navigate how
to get from here to there and how to come out at the end with a credential that could
be of value in the labor market. Connected to that. is there are too many choices, which
is too hard to understand what the impact was of making a particular choice. Third problem:
too little financial support. A lot of the people who attended were low-income; they
that got into problems working and supporting; they might have to work more and that meant
they had to study less. Too little social supports. We werenít dealing with people
who may be focused only on school, many would be working, there was little support from
others, mostly people who were sort of ended up as commuters. A final thing, and thereís
more on the list I could do, but in interest of time, is there was too little link to employment
and employers so one could finally at the end, end up with something and it may end
up having less value in the labor market.
So we can look at Career Pathways, I think, most illuminating is a systematic set of responses
to attempt to deal with lack of high rates of completion. So key Career Pathway ideas
that I think are in some ways implicit in some of the um, what Iíve said so far. First
of all, they deal with a wide range of skill and other needs. So the skills that are involved
are potentially anywhere from basic foundational academic skills to very refined occupational
specific skills. But itís not just skills that the programs attempt to deal with, they
attempt to deal with these other needs. Ah, for example, financial supports potentially.
If you noticed here, Iíve already introduced the idea of adults and youths so you can see
one indication of where my story was too simple, um, but weíll just pass over that for the
moment.
A second element of Career Pathways Programs itís very prominent is to create management
well articulated training steps. In as thatís really where the notion of the pathways comes
from, so instead of something thatís an opaque course catalog, the ideas is to substitute
transparency, a well-lit pathway that the person can see where they are on the pathway
how it connects to future steps and then it all makes sense. So there is choice, but it
is limited and itís structured. So the third element is that Career Pathways Programs have
an occupational focus. Itís omnipresent. Ah those occupations can vary greatly um depending
on the program, but thereís always something uh, an occupational focus included.
And finally in getting back to the idea that people need a multiple systems to sort of
navigate through um, Katie, does that mean Iíve used five minutes? Okay, thank you.
Um, that it needs to have a broad group of partners to be responsive to those needs.
So very briefly, I want to show you a diagram which I am not going to be able to spend a
lot of time on, but if you sort of look at it sort of going from left to right from lower
to higher skills, um, and hopefully to lower to higher employment and earnings, what it
really reflects is a broad range of skill-building. So the programs arenít necessarily confined
to a single area, like, you know, a basic bridge program which would deal with remediation.
Um, second is thereís education and thereís employment all the way along the arrow. Um,
itís not like employment just occurs at the end of, of the arrow. You can get on and off
at various places. And if you look at the nine programs in this study, in fact, all
except one, the programs donít deal with the whole area, the whole era, they deal with
parts of it, but not the entirety of it. So not everything is going to be universal. The
whole sort of pathway to the bottom to the top isnít spelled out. I hope people can
see this, although I donít want to focus too heavily on the whole thing, Iíll come
back to it several times. So the important thing right now, I want you to focus on is
the left-hand side. This is a theory of change for Career Pathways. It really drives the
ISIS evaluation design. So for the moment what I want to focus on in finishing up my
description of Career Pathway Programs is the left-hand side was program inputs. Because
these are what are, what we refer as the signature strategies, sort of what the programs are
and what they ah, um, how theyíre structured. Again, we use the term signature cause these
are not universal. Itís not like every program has every aspect here. And in order to sort
of again, ah not spend that 15 minutes on this slide Iíll just call attention to a
couple things.
One is, one of the key elements is assessment. If people have a broad set of needs that go
beyond just their academic situation and weíre going to deal with those needs to have a more
successful outcome, we have to sort of look at both academic and non-academic skills,
we may have to sort of look at how much money somebody has to undertake this training program,
what their budget is and do they have the money to actually, to be able to support themselves
while theyíre doing this and potentially a family. Um, second there are a whole lot
of kind of key strategies in terms of instruction that are often present. I will just focus
on a couple of them because contextualization acceleration, but if we sort of go back to
the idea that people may be in the kind of regular world of community colleges lost in
the jungle of remedial work never to emerge, never actually even to make it to occupational
training.
One way, of course, to remedy that is to try to introduce occupational training earlier
and link it to the uh, basic skills remediation thatís, you know, contextualization. There
are many different notions that go with it, but the idea that people might learn better
by learning both at the same, and it might also help them maintain their motivation.
Ah along with that is the idea of acceleration, if the path is so long that people canít
um, uh likely emerge from it um we need to shorten it. Okay, I wonít go through all
the other things. In the interest of time, Iím not going to show you the map.
So what this means is that um, skipping to the second bullet, all the ISIS sights exemplify
the central Career Pathway ideas. But they vary greatly in how they do it. They vary
greatly in their subject population. Some may focus on people who donít have a high
school degree. For another, it may be necessary to have a high school degree. Some programs
focus on people who are close to college ready perhaps, reading at the 10th or 11th grade
and not quite there. Others may start with people who are at the sixth grade level. So
these big basic sort of differences in target populations lead to sort of different needs
for a variety of training steps. Um, uh, the also just to sort of make clear that the uh,
um, this is not just talking that lead organizations, but that the, as community colleges, but some
of them are but there are also others that are community based organizations, others
that are work-force investments boards, all in some way or other theyíre linked to colleges.
So the upshot of this is that the number of factors differing across site in the small
sample sites really precludes statistically identifying explanations for site difference
across site differences. Um, Iím mentioning that because as youíll see, this is a different
perspective in, in, in the design that Laura and the design that Alan will describe. Um
ours, in ISIS will be a deep dive into non-separate programs and we will produce nine separate
reports for each side. Weíll take advantage of a common core of data collection, but what
we will, use will, vary by uh, by a site, by program.
Okay, so three elements to the study that Iíll describe briefly and their data sources:
implementation study, impact study, cross benefit analysis. So the goal of the sort
of thinking back to the design, Iíll just skip forward here, the goal of the implementation
study is to understand in depth the left side of the diagram. The program input um, to document
the design of the Career Pathways Program to define how it was implemented related to
the design and very importantly, for both the implementation, the impact study to describe
differences in the level, type, duration, content, and completion of services received
by the treatment and the control group. ëCause essentially given that this is an experiment,
the treatment is really defined not simply as what the treatment group got, but what
the treatment group got compared to what the control group got because we know many will
seek services outside of the program who are in the control group. And weíll also, of
course, though want to very carefully document um, the ah, experience of the treatment group.
So we have a wide variety of uh, data sources for the implementation study uh, so weíll
be talking to, interviewing and having surveys for program staff, instructors, case managers,
employers and other stakeholders and supervisors and people who come out the other end. Very
importantly, in the fifth bullet, we will have a 15 month follow-up survey of treatment
control group members and thatíll be a main way that weíll establish what really did
the treatment group get over and above what the control group uh got. Um, weíll also
have in-depth interviews with treatment control groups and this information will be uniform
across all the sites. What we will use in a particular site may vary, but we will have
this uniform data and in addition we will have using program administrative data, data
that are unique to the site. Where am I supposed to be pointing? Hmm, oops, I clicked the wrong
button, [laugh] too many green buttons.
Um, so what are the key questions related to the impact study? Iím gonna go past this
to the a um, theory of change diagram again. So the main the primary questions for this
study are the things that over in the right hand side um, we want to understand how each
programs, the net impacts on primary outcomes and we define primary outcomes as educational
attainment. Do people, in fact, for example, end up with more degrees? Do they end up higher
numbers of credit hours? Um, we also, of course, want to know because fundamentally the purpose
of these programs is to increase peopleís position in the labor market is, do people
earn more money? Um, that is really a fundamental question for defining the success of the program.
Uh, but weíre interested beyond just things like their earnings, weíre also interested
in do they get, you know, greater levels of uh, of uh employment benefits? Do they persist
in employment? Are they more likely to be working in the occupational area in which
they were studying. And the other thing that comes out of that on the right hand side that
weíre interested in is, does this ultimately improve individual and family well-being?
If peopleís earning go up, did their income go up with that? Ah do they end up with greater
assets or potentially less, perhaps they end up taking on debt uh that doesnít offset
them. Do their children do better? So this is sort of the ultimate kind of tests are
these primary outcomes on the right hand side.
But what goes on in the middle is also important because these are the devices by which people
have designed these programs with the intention of improving the things on the right hand
side. So these include uh, well, I should have mentioned very briefly on the left hand
side, just go back to reminding you that we will understand the program inputs from the
perspective of not only implementation, but also impact, but we have a bunch of things
in the middle that we hope are the programs of, are precursors of the uh outcomes on the
right hand side. So will foundational academic skills be improved? Will occupational skills
be improved? Uh, cycle social factors, will people have demonstrated greater ability to
stick with and fulfill their plans? Will they have more academic self-confidence? Uh, will
they understand better what it is to be involved in a career and how to move up through it.
Will they show, if especially in a program that has ah an emphasis on financial supports,
will they show less economic hardship and more ability to manage their uh, their uh
household budget?
So what we will have then is a uh, um describes the data sources of how weíre gonna get at
some of these uh, questions. Um, first of all, we have a baseline uh, it includes things
like, uh, you know, demographics, age, sex, educational background, household structure
and so on. Uh, thereís a self-administered questionnaire. This is something thatís done
in privacy where people ask a variety of questions about-answer a variety of questions related
to things like academic self-confidence uh, and so on. Uh, career orientation like, what
are their educational aspirations? What kind of, how far do they expect to go and so on
and other elements. There also those will be asked and are being asked plus all the
sites. They will also use where theyíre available basic skills assessments that the sites administer
for programmatic purposes.
So weíll also have follow-up surveys because we canít answer everything uh, uh, well weíll
have two sources of follow-up information. One are surveys. The first survey is at 15
months. We expect a second somewhere around 30-36 months and a final one potentially of
60 months. The 15 month survey, again, the focus will vary by program. If we have a program
thatís designed to primarily help people attain two year degrees, we have a very different
sort of window of expected impacts than one where the focus is in getting somebody a short-term
certificate. So weíll have the same information across the various sites, but weíll use different
information depending on the particular program model that a site, that a site is following.
So um, in some shorter program a focus on, on uh, employment outcomes might be important.
On the other hand, although weíre gonna be interested in employment outcomes in a longer
program, um, weíre gonna have different expectations. We may well expect that peopleís earnings
will go down while theyíre making this investment in education. Uh, finally, um, you know, 30
and 36 months the focus will change more to the right hand side, but not necessarily exclusively
and if the longer term follow-up that uh, we hope the study will involve uh, will really
turn to uh the right hand side pretty much exclusively. So we also will have administrative
records data. Um, weíll have uh as available related to particular sites uh, state and
local sources of college records information. Um, weíll have uniformly from the national
student clearing house enrollment information, institutions attended and nature degrees completed
uh and very importantly weíll have wage records on everybody in this sample.
So, in summing up, before I get just very briefly to the cost benefit analysis, what
we have then is sort of a very comprehensive uh, data collection that will focus on all
the elements of the Career Pathwayís theory of change. We wonít focus in every site on
every element of that theory of change, weíre going to sort of particularize that theory
of change through the particular strategies that our program has adopted, and then weíll
focus on that data that is pertinent to the particular program.
So then finally, uh weíll have a cost benefit analysis. In the long run do the programís
economic benefits outweigh its cost? Thatís going to be a critical question to answer
where these programs are making investments in people and we want to know ultimately whether
those investments pay off. So just to get the end for a minute, skipping a couple, thereís
some contact information if you would like further information on this study. Thank you
very much.
>> BRENDAN KELLY: Thank you Howard. So one point Howard didnít mention, which I think
will segue into Allenís presentation is the ISIS project started without ACF sort of telling
Abt this is what we want to do. And in fact, it evolved through a project of consultation
and, and really seeing where the field was, was at, and over time the team got to focus
on Career Pathways, is that focus was sort of evolutionary, but it was also opportunistic
because then with the Affordable Care Act, ACF received the funds to create the Health
Professions Opportunity Grant Program, which is a Career Pathways Program that, and Sectoral
Training Program that Alan Werner is now going to describe.
>> ALAN WERNER: Thank you Brendan, and thank you Howard for laying the foundation of the
Career Pathways framework, saves us some time. So as Brendan told you, my name is Alan Werner
and Iím the Co-principal Investigator, can people hear me in the back? Iím the Co-principal
Investigator of the Health Profession Opportunity Grants National Implementation Evaluation,
or the HPOGNIE, and Iíll be describing the studyís background, context and design. Actually,
I should wait until, can you, yeah, thank you, the project is sponsored by OPRE, Molly
Erwin and Hillary Foster are the project officers and the Urban Institute is our partner on
the study. Iím just gonna go ahead, oh, there you go, thank you.
So um, Iím gonna cover a number of topics about the NIE today, including its policy
and program context and some detail on the studyís goal and design, including the research
questions, the design framework, the data collection strategy and the various analyses
that weíll undertake the address the research questions. So what is HPOG, well, itís a
number of grants for those of you that donít know, itís a number of five-year grants to
support developmental education and vocational training programs designed to train low-income
individuals in high demand occupations in the health profession, such as in nursing,
nursing assistants, medical records personnel and so on. The grants were authorized by the
Affordable Care Act and in HPOGís first fiscal, 2011, the Office of Family Assistance at HHS
ordered $67 million to 32 grantees in 23 states. Twenty-seven grants went to programs targeting
TANF recipients and other low-income individuals, and five went to tribal organizations. The
target five-year program enrollments range from 165, big range to 5,000, and annual grants
range between $1 and $5 million.
This slide presents some of the key features of HPOG programs. As I mentioned, theyíre
designed to serve low-income individuals and particularly, those with relatively little
work experience, or vocational skills, and often with low educational attainment, the
same target populations for ISIS. The target population also includes those currently working
in the healthcare industry, but at low level jobs with little chance of career advancement
without additional education and training. The programs were designed to address two
growing problems. The shortfall in healthcare professionals and the growing need for some
post-secondary education to get a decent job. The programs are implemented by variety of
institutions as in ISIS, the most common grantees being community colleges or work, workforce
investment boards with the two types of institutions often in partnership.
This slide presents some of the more, some of the features of HPOG programs, specifically
important to elements of program design. The HPOG programs implemented by the grantees
vary in design, but they include core elements of the Career Pathways framework that Howard
introduced. This slide lists the core elements and I, Iím not going to go over these again,
since Howard did such a great job. Oh, sorry, so here you go. These are reading glasses
and I canít see 10 feet in front of me, but I see my notes, so I apologize for that. So
yep, so these are the, Howard talked about these features and they ought to look pretty
much like the ones he showed you for ISIS. And in fact, we drew on the Career Pathways
that was initially developed for ISIS. It was very helpful for us.
Um, as I said, the HPOG and IE is sponsored by OPRE and is one of a number of HPOG research
projects that comprise a multi-prong research agenda, and among the other projects are the
HPOG Impact Study, which my colleague, Laura Peck, will describe; the evaluation of the
HPOG tribal programs being connected by NORC; the HPOG Implementation Systems and Outcome
Evaluation, not to be confused with the NIE, but that developed the design for the NIE.
The ISIS project, which I donít know if Brendan mentioned this or not, I guess you did. It
includes three HPOG grantee programs and weíre going to use those data from the ISIS project
in our, both in the NIE and the impact study. And then finally, the University Partnership
Research Grants for HPOG, they had their meeting yesterday here, which includes awards to five
university and HPOG grantee partnerships studying specific aspects of HPOG programs.
Now, the research questions, the NIE is designed to address the following questions. Iíll
just read them off. How are health professions training programs being implemented across
the grantee sites? So what are these grantees doing? What changes to the service delivery
system are associated with program implementation? A big concern of HHS was that low-income people
were not getting enough good access to training programs in any profession, much less, healthcare
professions, so we want to see whether or not that has improved. What individual level
outcomes outputs and outcomes occur? So what are the results of participating, of enrolling
in HPOG, and then finally, what key components and factors appear necessary or seem likely
to contribute to the success of these programs?
So this slide presents a very high-level overview of the NIE design framework needed to address
those questions. And it entails the measurement, description and analysis of the HPOG contextual
factors, the relevant populations and their demographic and socio-economic characteristics,
program administration, the specific program services, and trainings available to participants,
the guts of the HPOG program and the results. Program outputs and outcomes. Importantly,
the design allows for the exploration of the entry relationships among the variables and
constructs measured within each of these domains. And thatís the statistical-statistical relationships,
not just qualitative.
This slide presents a logic model showing the several domains and their hypothesized
causal relationships. Over the next few slides, donít, donít worry about this, Iíll go
over whatís in the boxes. So among the HPOG contextual factors in the NIE, that was a
box in the upper left, are relevant community demographic and socio-economic factors, as
well as the relationship of the HPOG programs to other similar training opportunities for
low-income populations. Weíll also be collecting data about the granteeís institutional identification
and culture and experience with similar programs and target populations. In addition, weíll
be examining the granteeís program, partners and stakeholders.
The study will also describe the local healthcare labor market and its influence on HPOG program
design. The NIE will describe the relevant characteristics of the eligible populations,
including education and employment backgrounds, as well as household income. And for 23 of
the 27 low-income and TANF grantees will have information on the education and work-related
attitudes and aspirations of HPOG participants, thanks to a supplemental baseline survey thatís
being implemented in the grantees and the ISIS project and the HPOG impact project and
that supplemental baseline overlaps a bit with the self-administered questionnaire that
Howard described in ISIS, but itís really for a different purpose that Laura, I think,
will explain in her talk.
And then, finally, among the contextual factors, weíll be describing each granteeís management
and administrative structure, including the degree to services and trainings are contracted
out, or provided in-house. Also important, is how centralized program administration
is. Is there one central administrative location or service location, or is the program decentralized
across multiple local service sites, and do these sites operate distinct programs? Programs
that are simply different than the other sites under a granteeís egis. This slide presents
the types of program components that services and trainings that will be describing and
measuring in each grantee and service site.
These are, of course, the core of the HPOG initiative and the program factors, as Howard
explained, closely related to participantís success in education and employment. Moreover,
these program factors are those most directly amenable to tweaking in the service of achieving
better results for participants, so I think itís really important to measure these variables
and constructs accurately and in standard ways across grantees and sites. So they have
to cover a lot of variation in target populations and available services and trainings. And
as youíll see, this measurement is very important for part of our design in the impact study.
In the interest of time, I wonít go through this list. Howard talked a bit about these
components. Iíll point out that weíll also be measuring the degree to which HPOG programs
fit into the Career Pathwaysí framework. So they do the things that framework says
ought to be done to serve this non-traditional population.
Moving along, this uh, hmm, this slide presents the core output measures for the study, or
the more immediate results of training, and they include measures of, very similar to
ISIS. Additional hours are both basic and vocational education as well as benchmarks
for course completion and achievement, degrees, certifications, licenses and such and the
like. Additional program outputs include exposure to work-based learning and experience, such
as apprenticeships, internships and training in a career awareness and work related attitude
and behavior. Here are the program outcomes that weíll be measuring for the NIE. Among
the shorter term outcomes are included a summary measure of education and training accomplishments
related to employability, and other outcomes, including employment, employment and healthcare
importantly, and wages and benefits on the job.
For longer term outcomes, weíll be looking at further education and training, for example,
our participants are on a true career pathway. Do they make their way along the rungs? And
going back periodically to upgrade their skills. Finally, as an optional test, weíre preparing
to measure changes in child and family well-being over time for grantees in the impact study.
Itís mainly in the service of the impact study, but itís an important descriptive
element for us in the NIE as well, potentially.
Now, as you might imagine, a project like the NIE that seeks to measure and describe
such a wide range of individual characteristics and outcomes and grantee and site-based factors,
that may be related to outputs and outcomes requires an equally wide ranging data collection
strategy. And the NIE will collect both the administrative and survey data. This slide
presents the administrative data sources. One of the main sources of the MIS developed
for HPOG, the performance reporting system, to some the PRS, the PRS collects the administrative
data at the individual grantee and site level. Among the individual level data are a broad
set of demographic characteristics collected at program, application or intake, including,
for example, age, gender, ethnicity, family and household status, education, employment
and background, and current status.
The PRS also includes a post-intake record of services and trainings received by each
participant, as well as the completion status for each training course and the degree, certifications
and licenses received. Finally, the PRS also records employment status at program entry
and exit, as well as in, program changes in employment, and employment as of six months
following exit. At the grantee and site level, the PRS includes information about the granteeís
institutional identity, its service providers, as well as, most importantly, the services
and trainings available to program participants. The next line on this slide is a typo, I donít
know how it got there, so just forget itís there.
Another important source of administrative data as in ISIS, is the National Directory
of New Hires, the NDNH. The NDNH is maintained by the Office of Child Support and Enforcement
at HHS and it includes the quarterly employment and wage record, earnings record for any individual
working for an employer that reports to its state quarterly wage reporting system. Itís
unemployment insurance system, which is nearly every worker in the United States. The NDNH
will be the major source of information about individual level of employment and earnings.
It wonít rely so much on the six month follow-up information that-that the grantees will be
collecting for that. As I mentioned, the NIE will be using, will collect both administrative
and survey data. The surveys we developed for the study are pretty extensive in their
scope since they have as their mission, the collection of standardized codable information
about all of the variables and constructs included in that logic model and all those
boxes, or many of those boxes.
The grantee survey is the largest and will gather comprehensive and comparable data across
all grantees and sites about contextual factors, program administration, and program components,
including marketing, intake, assessment, counseling, supportive services, trainings, and all other
program activities and services. The management and staff survey will explore management and
staff approaches to key program services to the delivery of key program services and activities,
as well as beliefs and attitudes that the HPOG program and this target population. These
data are important in exploring the degree to which work, client interactions may be
associated with the program results. In large part, to flesh out the contextual description
of HPOG programs as well as the level of systems change.
Weíre also fielding a partnersí stakeholders survey, we also sometimes call that the network
survey. In the study, we distinguished between the program partners who play an active role
in program implementation and stakeholders who have some institutional interest in HPOG
and interact with the program in some way, for example, a local or state healthcare employer
organization, or a nurseís society would be a good example of an HPOG stakeholder.
The major purpose of the survey is to describe the network of agencies and institutions in
which HPOG operates and to assess the system supporting HPOG and its goals, and any systems
changes since HPOG began and that might theoretically be ascribed to HPOG.
Weíre also conducting an employer survey to assess local healthcare employers familiarity
with HPOG and for those who have hired HPOG graduates, the degree satisfaction with the
program. Another important survey which I inadvertently omitted from this slide, but
that Howard talked about, is the 15 month follow-up survey of program participants.
Although this survey is mainly part of the impact study. The NIE will be using information
from the survey and describing participant outcomes besides those that we get through
the NDNH. Finally, as an optional data collection task, we may conduct case studies of selected
grantees and/or program components that appear to be more effective in realizing participant
goals for peer education and employment.
Now, Iím gonna move on to the various analyses that weíre going to undertake to address
those research questions that I talked about earlier on. The descriptive implementation
study will address the first research question, how are health professions training programs
being implemented across the grantee sites and in answering the question, the descriptive
study has two related goals. To develop, first of all, a comprehensive description of each
HPOG grantee, as well as the HPOG initiative overall being able to characterize the national
initiative, and develop codable measures of program design and implementation strategies
for use in the NIE outcome study, and as weíll see, in the HPOG impact study as well. Iím
going to change the order of the slide a little bit and move on first, to the systems change
analysis, which addresses the second research question, what changes to the service delivery
system are associated with program implementation?
So the major goal of the systems change analysis is to examine HPOG grantees partnership and
network structure, and whether and how itís changed under HPOG, since the beginning of
HPOG. In particular, this study component will assess the degree to which these changes
are associated with the goals of preparing HPOG participants for healthcare jobs, while
accommodating all of their support service needs. The analysis will also examine the
extent to which HPOG created or improved accessible entry points into the health professions workforce
to the target population. Thatís a big, important concern.
Finally, I list the outcome study, which addresses the third research question, what individual
outputs and outcomes occur? So the outcome study will first of all, describe program
outputs and outcome, both within and across all grantees and program sites. Second, the
outcome study will conduct a comprehensive participant flow analysis, which is a quantifiable
account of the degree to which participants reach program benchmarks. For example, enrolling
in a training program, completing a training program, attaining a two-year certificate,
finding a job, finding a job in healthcare, and so on. This kind of analysis can be a
powerful tool in focusing on variable rates of success by grantees, institutional types,
participant characteristics, types of training and you can, you can make your own list. Any
of the other, by any of the other constructs that we will be measuring. Finally, the outcome
study will break down the descriptive analysis of outcomes by looking at outcomes by other
variables, such as sub-populations, again, grantee type, grantee choice of trainings,
other administrative choices and so on.
This account that I just gave of the analyses leaves open the issue of the fourth research
question, what key components and factors appear necessary or seem likely to contribute
to the success of these programs? This research question would be addressed in part by the
three NIE studies I just mentioned. Individually and together, theyíll help generate hypotheses
about program design and implementation strategies. Additional analyses that provide evidence
on the association of aspects of the HPOG program and outputs and outcomes also, under
consideration, but finally, the HPOG impact study which my colleague Laura Peck will now
talk about is designed to test causal hypotheses about how training program features affect
participantsí success.
>> LAURA PECK: Thanks, I, I retrospectively yield a portion of my time to my distinguished
colleagues and express my gratitude for their having set the stage in describing what we
mean by Career Pathways and also the context of the HPOG program. That said, with my slides
up, I will um, what Iíll talk about today is a little bit more HPOG context, but in
particular the research questions that the impact study will be asking and the sources
of variation that we get to play with in this study. And then Iíll describe the two main
strategies that we will follow in examining site level and individual level variation.
The HPOG impact study includes 20 grantees serving TANF and low-income populations and
what Alan referred to as service sites, over a hundred service sites. To date, we have
about 12 sites who have begun randomization into the impact study with almost 400 in the
sample. We will be building over the next year plus to a sample size of just over 10,000,
and as Alan described, and the same situation exists in these HPOG sites as does in the
ISIS study, thereís a lot of variation out there.
These sites while they follow the Career Pathways framework, theyíre doing lots of different
things with the varying levels of intensity. The, we can see that in, across the HPOG sites
there is what weíre referring to as natural variation, that is each of these programs
has their configuration of services in place that they have implemented, but then weíre
fortunate in the context of the impact evaluation to be able to establish some planned variation,
that is, in some sites, we will be establishing a second treatment arm. One that experimentally
will test the effectiveness of a particular enhancement. And in this case, the enhancements
that we are implementing are peer support groups, emergency assistance and non-cash
incentives. And those are features that also vary naturally across the HPOG site, so those
sites that did not yet have those features in place have the option to implement them
as part of an experimental test if they were willing to ration access to those through
randomization.
And in addition to the site level variation that exists, we also have lots of variation
across individuals and their experience of the program. The research questions that the
HPOG impact study will address include most generally what impacts to the HPOG as a group
have on the outcomes of participants and their families, and to what extent do those vary
across selected sub-populations, but what weíre focusing most on in our design innovations
are the last two questions posed here, which locally program components influence average
impacts? And to what extent do participation in a particular component or components change
the impacts that individuals experience.
So Iím going to talk about two approaches to getting answers to those last two questions.
The first will refer to the site level analyses, and the goal of our site level are the analyses,
I guess, motivated by the fact that we have massive site level variation is that we were
hoping to estimate the impact of offering specific program components, that is understanding
what the average impact, whether the average impact in the site is changed by adding a
particular component to the HPOG package. And as I mentioned, we have this great opportunity
by having lots of variation in the world, both in terms of the natural variation and
the planned, or induced variation that exists. And in part, because of this, we have what
Iíve termed up here as the bonus of all of this, which is weíll have an experimental
test of the effectiveness of these three components and also be able to look at what the non-experimental
evidence shows about the effectiveness of those components and by blending those analyses
together, we will be able to strengthen the analyses that we have of the components in
situations where theyíre not experimentally based.
So consider, for example, a world in which there are four possible configurations of
how the components are distributed across sites. We can see that components may exist
in all sites. Labeled here F might be something like comprehensive assessments. And then we
also have components that exist in some, but not all sites and within that group of component
configurations, we might see some component that exists only through natural variation,
not one of those things weíre testing experimentally, for example, proactive case management, but
then weíll have this induced variation that exists, for example, with peer support, and
some of those, actually, all three of the enhancements that weíre testing were chosen
in part because they already exist in the field and we have the opportunity to observe
them and in both experimental and non-experimental settings. And each of these configurations
of components offers its own opportunity for inferring the influence of a particular component
on the impact magnitude.
So the design then, that we are using here, if I wasnít clear, a two or three-arm experimental
design in several sites we will have three-arm design where individuals are randomized to
experience the standard HPOG treatment; randomized to experience the enhancement that is everything
that HPOG offers plus the new feature, or to a control group and then in the two arm
sites, individuals will be randomized to the HPOG program or to a business as usual control
group, as Alan and Howard, both referenced whatever else is out there in the community.
So thatís what this first sub-bullet here identifies the services that are available.
Any of the control group members will have access to those services as well as HPOG members,
for that matter and that will reflect what the counter factual state of the world is.
And then we will have these added elements, the EFHG, the assessments, and proactive case
management peer support, other components and our intent is to estimate the impact,
the relative impact, the added contribution that those components make relative to the
impact overall.
So each of these coefficients on a particular component listed is an estimate of the effective
offering that component on the treatment group, and we will have in some instances, experimental
evidence on this point and in many instances, non-experimental evidence. The experiment
will measure the effect of the overall package of these components together, which will constitute
the full program that exists in the two or three arm site and then individual, those
individual components are what weíre really focusing on in some instances, weíll be able
to look at the effectiveness of them through experimental evidence and in some instances,
only through non-experimental evidence.
One of the challenges we face, of course, is that what if proactive case management
is only ever combined with assessment, how do we tease out the relative effectiveness
of those pieces and thatís what our design plan seems [word] to tackle and as I mentioned,
we have this bonus opportunity here, where we will have experimental evidence in the
same situation where we have non-experimental evidence to bear on the same question. And
so our interest is to bring this information together to improve the quality of the non-experimental
evidence by knowing what we think of as truth from the experimental evidence. So Iím gonna
talk briefly about how we will engage in individuals capitalizing on this individual variation
and point you in the future to our design report at some point down the line, weíll
be made available with all of the gruesome details that Iím passing and sharing with
you today.
But in brief, the approach that weíll take to capitalizing on this individual variation
that exists. Again, the, all of these analyses are aimed to capitalize on the fact that we
have an experimental evaluation. We have individuals randomized to a treatment and control group
and we want to be able to use that. We donít want to break the experiment and we want to
use the experiment to learn more about non-experimental questions. And so the approach that we will
take is an analysis of symmetrically predicted endogenous sub-groups, that is, weíll be
using baseline characteristics to predict program experience, and because those baseline
characteristics are exogenous to treatment, we know that the sub-groups that they can
support creating are experimental as well. And so if we can well predict program experience
be it participation in a particular component or combination of components, then the difference
in the treatment and control group outcomes remains experimental and unbiased by various
sources of influence and we can then interpret the results of that kind of analysis for those
who actually would have experienced part of the program that weíre modeling by assumption.
So our intent is to think about a variety of, of components for this analysis, and weíll
make the determination of which in consultation with OPRE and through the information that
we get from the NIE and implementation research, but we expect to think about kind of two main
categories of these that Iíve listed here. That individuals participate in particular
program components, that is, they used emergency assistance. They accessed child care support
or services, or, and thinking about a combination of services, theyíre sort of high service
individuals, they accessed a lot of services.
And those are things to the extent that we can model them with the baseline data convincingly
are candidates for this line of analysis, but not only program experience, but also
intermediate outcomes are possible candidates. That is those individuals who perhaps met
a particular program milestone they achieved a credential and we can look at the influence
of that as a mediator to their, to their outcomes and impacts. And so the analytic challenge
that we face in this is, of course, the predict membership into these subgroups, but as Alan
mentioned, luckily, we have a very rich baseline data that will, that includes the standard
information about individuals, demographic and employment and educational histories,
but then also, a variety of their other measures that they mentioned, self-efficacy, motivation
kinds of things that we think will be useful in predicting the pathways that they follow
through the program.
This is where, where, weíre extraordinarily grateful to Molly Erwin and the rest of our
colleagues at OPRE for allowing us to, to innovate in this. A lot of what weíre doing
is new and really exciting and weíre glad to have the HPOG grantees too, as part of
this project letting us use them as a testing ground for new methods that we hope will teach
us plenty about what works in Career Pathways Program. Thank you.
>> BRENDAN KELLY: Thank you Laura. We have five minutes or so for questions, comments.
Yes, if you can go to the mic.
>> AUDIENCE QUESTION: ED: Hi, my name is Ed. My question is about statistical power. So
to what extent have you thought about, particularly, for the induced variation thatís someone
you can control with a natural variation you canít so much. So have you thought about
that and if OPRE asks you if youíve got an analysis of some sort of, will you be able
to find positive effects, you know, even if variation is large?
>> LAURA PECK: We have done, yes, indeed, we have thought about it, and we have done,
you know, our own V package and design report includes estimates of the minimum detectable
effects that weíll be able to estimate given our sample size. Overall, the sample will
include just over 10,000. We donít know yet what the target sample will be for the selected
components, but part of what we, as a team, argued is that any information that we can
get from the experimental test will be of value to teasing out the relative effectiveness
of particular components and we do have the opportunity to be able to pool across many
sites, so that increases our sample size.
The, I mean, I think that our initial estimated suggests this is a worthwhile path to go down,
and then in the end we will see what, you know, weíll see what actually happens in
the field by inducing this particular kind of variation. The, the enhancements that we
chose to test in the field are ones that anecdotal we have evidence to suggests that where they
added to programs, or if theyíre part of program, suggestively imply that they would
have relatively greater effects. So with the exact magnitude of that, we donít know right
now because these are things that have not been studied. So weíre excited to have the
opportunity to learn something from that.
>> AUDIENCE QUESTION: ED: And see to womenís policy research. Could you briefly what sub-populations
youíre interested in and particularly, whether the, among the low-income individuals currently
with a, would it be possible to look at older women, so people who donít have younger children
anymore and hence, donít get TANF?
>> ALAN WERNER: Thank you. We do have information that would allow us to look at distinct sub-populations
by age and family status and all of that. The question is how large a group of people
would that include. We, we have some suggested, suggestions about sub-populations weíd look
at, for example, TANF recipients is important. Single parents with children is important,
and if, you know, women or individuals over a certain age are a big enough group, we will
be able to say some things about their experience and results, but at this point, itís unknown
and I donít, I just donít have in my head, you know, the frequencies by, by age group.
The average age, I think, for an HPOG participant is in the early to mid 30s, so there are some
older individuals involved.
>> HOWARD ROLSTON: I would add that weíre prepared to do something similar by sight
and because the populations vary in the sites, thereís some, at least one of the programs
for youths, so wonít have any older women. Other programs have sort of a broader distribution
and itíll come down again, to the size of the sub-group and whether we could pick it
up.
>> LAURA PECK: I would add really briefly on the NIE too, Alan, we, we will have a very
large sample size and while we wonít be able to look at the impacts on those sub-groups,
thereís lots of descriptive work that weíll be able to do just by having such a, much
larger sample size of those within the NIE and the program experience that they have
as documented in the PRS.
>> ALAN WERNER: Yeah, I did mention this, but the NIE will include since itís all HPOG
enrollees to include in 25,000 or more individuals, so thatís a big sample.
>> BRENDAN KELLY: Please join me in thanking our presenters and Iím sure we look forward
to seeing results at future acts. Enjoy your lunch.