Tip:
Highlight text to annotate it
X
>> I am going to start the broadcast now.
>> The broadcast is now starting. All attendees are in listen-only mode.
>> JAE'MIE: Good afternoon, everyone. I wanted to welcome everyone to the PREIS and Tribal
PREP Grantee Meeting. Before we begin the presentation, I just want to review a few
quick administrative items to let you know how everyone can participate in todayís web
event. And all participants should be able to hear the audio and view all the web slides
throughout the presentation.
If for any reason you are unable to view any of the information, please follow-up and send
an email to contact to let us know if youíre having any audio issues and/or any viewing
issues. Itís our expectation throughout this webinar that many of your questions or concerns
will be answered. And every attempt will be made to answer questions posed by any of the
participants throughout the webinar and at the conclusion of the webinar.
Quickly, I want everyone to turn their attention to the webinar attendee interface which is
made up of two parts. And on the viewer window on the left, that will allow for everyone
to see what the presenter is showing and sharing on their screen. And youíll also see the
control panel on the right. And within the control panel, thatís how you can participate
in todayís web event. By clicking the orange arrow at the top of the screen, you can open
and close your control panel. And to keep your control panel open and/or visible from
the view menu, please ensure that the ìAuto-Hide Control Panelî option is not selected on
the top of the file menu.
Questions may be typed in in the question box at any time during and at the end or the
conclusion of the webinar. Simply type in your questions and click send.
We donít anticipate any issues today with the event. But if for any reason technical
difficulties arise throughout this event and the audio and/or screen views are lost, please
attempt to dial in and log back into this webinar through the original invite that you
received. And if access is still unable to be regained for any reason, please check your
email inbox for updates regarding the rescheduling of this webinar event. But we donít anticipate
any issues today. So hopefully, that will not be the case.
Weíll now begin the webinar presentation. And Iíd like to introduce LeBretia White.
And thank you for attending.
>> MS. LeBRETIA WHITE: Thank you, JAE'MIE. And welcome everyone to the PREP, Personal
Responsibility Education Program, webinar. And this is actually a grantee meeting kind
of all rolled into one for our tribal PREP as well as for our PREP Innovative Strategy
grantees.
Just to review with you the flow for todayís meeting, Iíll begin with our agenda. There
we are. Welcome and introduction. We have several project officers who are participating
in todayís meeting and some of them will be presenting as well. Project officers for
both programs include myself, LeBretia White, Sarah Axelson, Joel Tinsley and Itege Bailey.
We also have another person who also interacts with the tribal PREP program frequently. Her
name is Deb Yatsko who is also participating in todayís call, as well as our programís
director, Marc Clark. Okay. Those are the federal introductions.
Also, just moving through our agenda, I wanted to share with you the first part of our meeting
on today. Weíll be talking primarily about non competing continuation applications and
how you should submit those. And Sarah Axelson will review most of the information for you
regarding those applications. I will then follow with information about our new system,
Grant Solutions, that is replacing grants.gov for the submission of your application.
Followed by that, weíll have some information sharing around performance measures reporting,
one of our favorite subjects for our PREIS grantees only. And so, with that being said,
after Sarah has completed her presentation around continuations and some other general
updates, we will excuse the tribal PREP grantees from the call. Youíre welcome to stay on,
but you may not find it advantageous for you as most of that information will be directed
to the PREIS grantees.
So again, after the performance measures reporting information for the PREP Innovative Strategies,
our PREIS grantees only will move into a presentation on evaluation reporting requirements. Again,
this segment also of our meeting on today is specifically for the PREIS grantees.
And Iíd also like to point out the evaluation reporting information sharing is not pertinent
to those PREIS grantees who are participating in the PPA study. And I think you know who
you are. Many of you are actually assigned to Itege Bailey. And so if youíre participating
in the pregnancy prevention approaches study, you do not have to remain on the call for
the evaluation reporting info. And it is our goal to end this meeting at 4:30 p.m.
All right. With that being said, weíll move on with Sarah Axelson sharing information
about non-competing continuation.
>> MS. SARAH AXELSON: Thank you, LeBretia. So you all are familiar already with our federal
staff. So weíre going to jump right into the content on your non-competing continuation
application. As you know, the continuation application is used to request funding for
the next fiscal year. And for our PREIS grantees, youíll be requesting funding for year three
of your five-year project. Our tribal PREP grantees will be requesting funding for year
two of their three-year grant project.
The continuation application also provides grantees with an opportunity to report on
their progress during the current budget year, also to provide a work plan including program
and evaluation activities for the upcoming budget year. And finally, we ask that grantees
provide a detailed budget and budget narrative justification for the upcoming year to allow
for planning.
There are a number of requested documents that are part of the non competing continuation
application. This list of forms is outlined in the non competing continuation guidance
that youíve already received previously from your project officer via email. And it is
requested as part of the package. Weíll go through each of these items briefly.
The federal forms that are required are relatively standard for many of you who have completed
this process before. You will submit an SF 424 which is the application for federal assistance,
as well as the SF 424A which is budget information for non construction program. The SF-424B
provides assurances for non construction program. SF-LLL, disclosure of lobbying activities
is also required. And finally, grantees are expected to complete the PSL which is the
project or performance site location form. All of these forms should be accessible in
Grant Solutions once you have entered into the system. And weíll discuss how to do that
again in just a little bit. But additionally, if you need to download a paper copy for any
reason, they are available on the FYSB Internet website.
Weíll talk a little bit about the requirements for each of the programs individually. So
the project narrative for PREIS grantees, and again this is specific to PREIS grantees,
should include an introduction with a brief summary or overview of the project design.
It will also include a six-month progress report, which they can see an example template
for in Exhibit A in the non-competing continuation guidance.
This progress report serves as the supplement or the replacement for your actual PPR program
report. And it includes the goals, objectives, activities and status of what youíve conducted
so far.
The Project Narrative for Tribal PREP grantees is quite similar. It will also include an
introduction and a brief summary or overview of the project design. However, tribal grantees
will use the project narrative to focus on their needs assessment activities that will
be conducted prior to August 1st, 2012, as well as to describe the target population
and models that are being evaluated for potential use during implementation. We understand at
this point grantees may not have solidified all of that information. But we would like
you to use the project narrative piece to describe what your plans are as broadly as
possible.
Youíll also include a six-month progress report. This is maybe a lot of similar information
that you were including in your performance progress report due April 30th. But it should
cover a project goal, needs assessment, objectives and activities. And again, thatís focusing
on needs assessment given that this first year is a planning year and also a status
update on activities, those that are in progress, met, unmet. What accomplishments and barriers
youíve encountered.
Both grantee programs will submit a work plan as part of their continuation application.
The PREIS grantees will submit one for, again, the next budget year, which is year three
of their five-year project. And that will include information on their goals, objectives
and activities for the coming year. There are examples, again, in the non competing
continuation guidance for what documents you could use to complete this work plan. Look
specifically to exhibits B and C.
The Tribal PREP work plan again is quite similar. However, keep in mind that this next year
will be the first implementation year for tribal prep grantees. So their work plan should
focus as much as possible, if not wholly, on implementation activities. Again, some
of those may not be completely solidified, but please include as detail as you can about
the things that you plan to do during this next year of the project. And Exhibits B and
C likewise in the tribal prep continuation guidance provide templates that you can use
if you so choose.
Both grant programs, PREIS and Tribal PREP, will include a budget and budget narrative
as part of their continuation application. And this is one of the final broader pieces
that have to be included. We expect grantees to provide a detailed budget. Youíll see
here some information about what that might look like, a spreadsheet with an itemized
budget that clearly delineates how each object class category of cost was derived.
Grantees should also submit a budget justification that clearly describes how the budget items
directly relate to the completion of the program activities. And finally, they have the option
of submitting an indirect cost rate agreement as applicable. And youíll see there a website
that provides additional information on indirect cost rates if you need that.
At this time, Iím going to turn it back over to LeBretia White whoís going to review the
online submission system and process for using Grant Solutions to submit your non competing
contingency.
>> MS. LeBRETIA WHITE: Thank you, Sarah. I just want to point out before walking through
the slides that you should have received an email with several slide presentations for
todayís meeting. Weíre currently operating under the first one which is the actual grantee
meeting. We have a set of webinar slides. And within those slides, there are some edits
that I want to bring to your attention before we walk through.
I wanted to give you as succinct information as possible related to how to utilize Grant
Solutions. So you will see some slides. They will be in the same order, but they may have
some additional detail as far as instructions. And please note that we will make those available
to you after the webinar, either this afternoon or first thing tomorrow morning. I definitely
wanted to share that with you.
And as Iíve said, Grant Solutions is a replacement system for grants.gov. And in order to access
the system, you would go to www.grantsolutions.gov. Many of you may already be familiar with the
system if you have other federal grants and those federal grant programs or offices have
already begun to use this new system for processing grant actions.
I do also want to bring your attention to one other thing. In the instructions, the
guidance document that you received on how to submit your non-competing continuation
application, we indicated definitely the federal governmentís desire at this point is to go
electronic to reduce paper.
And so to utilize Grant Solutions, however, if you do not have the appropriate technology
within your grantee site to submit the document, there is a mail in option. And you would want
to refer to the guidance document for instructions on how to do so and what justifications are
needed in that case.
First of all, what weíve requested on the federal level is that grantees would submit
a user request form. Weíve received quite a few of those over the past several days.
If you have not submitted those forms, please make certain that you submit those as soon
as possible. They were due on April 16th. And that form will give those persons within
your site who are authorized to make applications on behalf of your organization, it gives them
access to the system. So please again make certain that you submit those documents. If
you need another copy of the request form, please contact your individual project officer.
And this is what the system looks like when you actually enter the web address, the web
link, into your web browser. This is the actual home page or home screen. This is where you
will actually log into Grant Solutions. And what you see here is in the goal is where
you click onto to actually go into the log in page. But I also want to point out on this
screen youíll see a notation with an arrow that says contact information for technology
support. Thereís a phone number here that you can call if youíre having difficulty
once you receive your access information and password. Also, there is a web address that
you can contact, itís help@grantsolutions.gov.
This is the actual login page for Grant Solutions. On this page is where you will actually type
in your user name thatís provided through the system after you submit your user request
form as well as your password. If you run into difficulty or forgetting your password
or get locked out of the system because youíve entered it incorrectly several times, you
can actually obtain new log in information directly through the system. You do not need
to contact the help desk or your project officer. You would simply click on to ìreset password,
unlock account.î
Okay. And next, I just want to share with you accessing your application. Once you login
into the system, youíre going to actually click on ìmanage amendment.î And then youíll
select what type of action. For the purpose of this webinar and the instructions given
today, youíll be utilizing the system to upload your NCC or non competing continuation
application.
Once you click on ìmanage amendment,î the system will take you into a section where
you actually upload your application and those supporting documents related. Those documents
that Sarah has just reviewed with you, those are the ones that you will actually be completing,
both application documents within the system or uploading a copy. The preferences for your
SF 424 and your SF 424A forms that you actually complete those documents directly in the Grant
Solutions system.
And this is referred to on this page as your application control checklist. And youíll
see to the far right, there is like a triangle with an explanation and some checkboxes that
indicate those actual icons indicate those areas that you should be completing. And if
you have questions about how do you upload your narrative information, those non federal
forms, thatís your narrative that tells us how youíve progress over the past six months
and what you plan to do in the future six months. That information would be uploaded
under ìmiscellaneous informationî which is located on your screen at the very bottom.
>> MS. AMANDA PATT: LeBretia, this is Amanda. I just wanted to point out for the very first
link that the grantees need to click on. They actually donít need to click ìmanage amendment.î
They can click ìapply for non-competing continuation award.î Thatís the link under that one.
>> MS. LeBRETIA WHITE: Okay. Itís not shown here, but Amanda Patt who just chimed in,
and thank you Amanda, is our Grant Solutions liaison. This is a new system to FYSB as well
as to some of our grantees. So thank you, Amanda. And feel free to interject any other
comments as we move through.
>> MS. AMANDA PATT: Thanks.
>> MS. LeBRETIA WHITE: And what Amanda referred to is most recently the information that grantees
receive for non competing continuations, those hard copy documents or emailed electronic
copy documents such as the checklist and the guidance documents. That information is now
uploaded into the system. So at this point, Amandaís saying youíll actually be directed
to non-competing continuations versus amendments.
And weíve referred to already some of the information thatís stated here on the screen
in a previous slide. Again, the application control checklist is where you have options
to do the following. You can fill out forms, download documents and instructions that we
just discussed. You can upload files or documents and also submit and unsubmit your application.
Also here, entering and saving online form information. To enter online form information,
select the ìenter onlineî link. Thatís pretty self-explanatory. Associated with the
item you wish to fill out. For example, the SF 424A. And there is a box within that control
checklist where you can click onto to provide that information.
When youíre done entering the info, select the ìsaveî button at the bottom of the screen.
You donít have to fill in all of the information in one session. You can always save and go
back and return at a later time. However, the online forms must be validated before
you can submit the application.
In reference to validating online form data. Upon completing the online form information,
select the ìValidateî button at the bottom of the page. Any validation errors, such as
missing required data, will be presented to you. Usually thatís in red lettering. If
you are filling out the 424A form, the system will prompt you to copy the budget information
over to the 424 form. And you would select ìYesî if you wish to do so.
And here is the screen that actually gives you a visual of how to upload an attachment.
You simply click onto ìuploadî to load a file or a document. You will be prompted within
your browser of your computer to actually locate the desired file or document for uploading
to Grant Solutions. And thatís just general additional information regarding uploading
a file that weíve already reviewed.
And finally, youíll be able to view your full application once all of the documents
have been uploaded and submitted. To complete your application electronically, you enter
the information by using the online form or adding attachments. And then required items
are noted by an exclamation -- I want to point that out to you ñ- by an exclamation point
image. Youíll see that circle here on this screen. If an enclosure has not been verified,
a red X image will likely be displayed.
And submitting your application. Once youíve completed your application, click the ìVerify
Submissionî button at the bottom of the page. Then click ìFinal Submission.î Your application
has now been submitted to the grant staff for review. And this is just another screen
that gives you a visual for where to locate the ìFinal Submissionî button.
Just some reminders, and weíll continue to remind you throughout todayís meeting. Submission
of continuation applications. Those notifications were mailed on April 29th. So all grantees
should have received that information from LUCK which is our grant review and other grant
support contractor. If you did not receive that notification, please contact your project
officer immediately for a copy.
Grantees are to submit non-competing continuation applications via Grant Solutions not later
than May 29th. You can submit before May 29th, but the deadline again is as stated.
User Request Forms were to be completed by the Authorized Representative and Project
Director and submitted to the assigned project officer for access to Grant Solutions. A reminder,
if you have not received your access to the system, please forward those forms as soon
as possible, and I would say not later than tomorrow, to your assigned project officer.
At this time, are there any questions regarding non competing continuation or the grant solution
system. We do have Amanda Patt whoís on the line with us. So if you have any specific
grant solution questions, I would say to you ask those now. And you can type those in.
There is a question that states, ìWhat about the form SF 425 that is to be turned into
OGM? It says to send original and copy to OGM. Can this be done by email instead of
paper?î Yes, it can be submitted electronically. You can mail it in or submit it electronically.
Let me just clarify. For the PREIS grantees, your SF 425 will not be due until December
of 2012. But for the tribal PREP grantees, your SF 425s are due semi-annually. Tribal
PREP, semi annual. SF 425 PREIS, annually.
ìPlease confirm that the six month progress report covers October through March for PREIS.î
Yes, that is correct. Technically, September 30th, but October through the end of March.
ìCan we have additional users for grantsolutions.gov?î Yes, you can. However, right now weíre trying
to make certain that we at a minimum have the project directors and the authorized reps
in the system, but you can also make requests for other persons to have access, depending
upon what their role is with the project. They may have limited access to only view.
Some of the financial accounting persons, senior programs, may have the option to upload
some fiscal information. But everything has to be officially submitted to us by your authorized
representative.
Okay. Another question. ìCan Word or Excel documents be uploaded into Grant Solutions?
Or do uploads have to be in PDF format?î You can upload ... Amanda, Iíll ask you to
answer that question, but I think the answer is all types.
>> MS. AMANDA PATT: Yes, we accept all standard file types. So Excel, Word, even Images, things
like that. Everything you can upload. And if it does not upload, youíll actually get
an error message when you try to upload it, telling you that the upload has failed. And
then if itís successful, similarly you get a message saying ìUpload Successful.î
>> MS. LeBRETIA WHITE: Thank you, Amanda. So all documents, your Word, your Excel, as
well as your PDF document, can be uploaded. There is a question about due dates for the
tribal PREP as well as for the PREIS grantees. The due date for non ¨compete continuations
is May 29th. And someone has asked where you can obtain the user request form. Please contact
your project officer. You can email them now or after the meeting today and theyíll be
certain to forward a copy of that to you.
>> MS. AMANDA PATT: And while weíre waiting for other questions, I just had one other
thing to add, LeBretia. Similarly, how youíre contacting your project officer for the user
account forms, you should contact them if you have any questions about filling out your
form, how to fill out forms or how to do your budget narratives or project narrative. And
you should only contact the Grant Solutions help desk if youíre experiencing a technical
error. So, for instance, if you click something on the screen and the screen goes blank and
says ìSystem Error,î then you should contact the help desk. So I just wanted to throw that
in there.
>> MS. LeBRETIA WHITE: Amanda, we have another question that you probably can help us with
as well. This question is, ìWe have a Grant Solutions account from a previous grant process.
Do we need to initiate a new one for PREP?î
>> MS. AMANDA PATT: If you have a previous account for this same organization, which
means youíve registered with the same EIN and Dunnís number with CCR, then you should
not need to fill out a new form. When you log in, you should actually have access to
your current grant, all of them.
>> MS. LeBRETIA WHITE: And if they do not have access to their PREP grant, whatís the
next step?
>> MS. AMANDA PATT: The first step would be to contact their program specialist to make
sure that theyíre assigned to the grant. And the second step would be if theyíre still
not able to see it, then you should contact the help desk on their behalf.
>> MS. LeBRETIA WHITE: And does it hurt if grantees who already have an account also
request a new account based on the PREP program?
>> MS. AMANDA PATT: No.
>> MS. LeBRETIA WHITE: And let me just clarify for some grantees who said, well, you told
us we had to submit a form and yes, we did. And as Amanda said, thereís no harm in submitting
if youíve already been connected with the program. But we also want to verify and make
certain that the appropriate persons within your program are requesting access to the
PREP grant. So thatís just a point of clarification regarding that process. That appears to be
all of the questions. Thereís one more regarding performance progress reports. But weíre going
to clarify that in the next couple of slides. So I think that that will help. And if not,
we will come back and address that when we do questions at the end.
>> MS. AMANDA PATT: And just so everybody is aware, when you log into Grant Solutions,
thereís going to be a ìHelpî tab up on the top menu. And on that tab, youíll actually
find recorded webinars where you can watch how to submit a continuation application,
as well as manual.
>> MS. LeBRETIA WHITE: Okay. Thank you for sharing that, Amanda. And also I would ask
grantees to refer to the grantee guidance document that was submitted for non competing
continuations that also shows the web links for those training and resource materials
for using the system. Thank you so much.
And I think weíll move on to the next segment of our meeting for today. And Sarah is going
to give some additional updates. And after which weíll hear from Ina Wallace as well
as Amy Farb. Ina Wallace is with RCI International. And Amy Farb is with the Office of Adolescent
Health. And again, weíll move onto our fun topic for our PREIS grantee about performance
measures. But hang in there grantees, tribal PREPs, Sarah is going to share additional
information that both programs need to be aware of.
>> MS. SARAH AXELSON: Thank you, LeBretia. So weíre going to talk about performance
progress reports. And I think clarify a little bit of the confusion around these requirements.
Because they are a little bit different for each program. Tribal PREP grantees need to
complete performance progress reports on a semi-annual basis. This means that their reports
are due thirty days after the end of the second and fourth quarters of the budget period,
or every six months.
Youíll see here that this means that a performance progress report will be due April 30th of
this year and October 30th of this year. For purposes of the April 30th report, tribal
PREP granteesóand again only tribal PREP granteesówill complete this report using
the PPR template that was sent to them via email by their project officer and will return
that template again via email to their project officer by April 30th.
In contrast to the tribal PREP grantees, PREISí requirements for performance progress reporting
is a little bit different. Performance progress reports or PPRs are due on an annual basis.
However, as part of the continuation application, PREIS grantees also report on their progress
during the previous six months.
So youíll see here that it lists a performance progress report due May 30th with the non
competing continuation. Again, this is simply that section of your continuation application
that speaks to six-month progress reports. Itís not a separate form. The reporting period
for that foróand one slight correction is actually May 29th. We will correct that on
the final slide before we wend them out.
Again, the reporting period for that progress section of your continuation application is
September 30th, 2011 to March 31st, 2012. PREIS grantees will then also submit a performance
progress report using the PPR template in OLVC or thatís already been sent to your
project officer. That will be October 31st, 2012 covering the reporting period of April
1st through September 29th, 2012.
So again, for PREIS grantees, one six-month progress report that is folded into your continuation
application, due May 29th, reporting on the previous six months. And then one annual progress
report due October 31st using the PPR template.
Weíd also like to give you a brief update about some upcoming training just to be sure
that your calendars are marked. For PREIS, thereís a western regional training that
will be held June 12th through 13th in Minneapolis, Minnesota and an eastern regional training
to be held June 28th through the 29th in Philadelphia, Pennsylvania.
You should have received notification regarding which location you would be assigned to as
a grantee to attend via email. And if for any reason, you have a concern about that,
you can speak directly with your project officer. But we are asking you to attend the location
that you have been assigned to for planning purposes.
Tribal PREIS grantees will also have a webinar on adulthood preparation subjects. That will
be on May 22nd, 2012 at 3:00 p.m. Eastern Standard Time. And we will be announcing additional
trainings at a later date.
Just a reminder again if we havenít mentioned it enough about some critical due dates that
we want to make sure youíre aware of. The first slide is for PREIS grantees only. Your
continuation application is due May 29th. Your performance measures upload and evaluation
report, both of which we will discuss after this portion of the webinar. That will be
what LeBretia was suggesting with Ina Wallace from RTI and Jean Knab from Mathematica. Those
are both due May 31st.
On July 16th, we request that PREIS grantees submit earlier than that if possible, but
no later than that date, any budget revisions from year two that need to be processed. On
October 31st, PREIS grantees will need to submit their year two performance progress
report, again using the PPR template. And then on November 30th, theyíll need to do
their final performance measures upload and evaluation progress report for this budget
year. On December 31st, they will also have a year two financial status report due.
The Tribal PREP grantees also have a slide for critical due date. Again, as you all are
aware, you have a semi annual performance progress report due April 30th. And you also
have a semi annual financial status report due April 30th using that SF 425.
Like the PREIS grantees, your continuation application is due May 29th. We also request
that for tribal PREP grantees, any budget revision to their year one budget be submitted
by July 16th. And then as you should know already, your implementation plans will be
due August 1st. We will be sending out additional information about the implementation plan
shortly and we will also be holding sessions on the implementation plan at the upcoming
conference. So donít be concerned if you havenít received information about that yet.
On October 31st, tribal grantees will have another semi annual progress report due. And
on December 31st, theyíll have another semi annual financial status report. At this time,
weíll take a look and see if there are any other questions that came in during that last
portion of the webinar.
The only other question that we had again with regarding the progress report, but I
hope that we have answered that sufficiently. We did just get one question come in that
said, ìCan we use the PPR template for this reporting period? This is for a PREIS grantee.
And upload it in Grant Solutions as our project narrative.î And the answer to that is yes.
You can do that if that is your preferred method of reporting on your progress for this
reporting period. But again, always go back to the guidance if you have questions about
what kind of format you should use or how to upload your information.
Thereís another question that says, ìPlease clarify if we submit our PPR via OLVC, email
or grantsolutions.gov for this May 29th deadline and also for the end of the year report.î
This will differ based on which grant program you are in and the question comes from a PREIS
grantee. So four PREIS grantees for the May 29th report, you can fold that report into
the narrative of your continuation application and not have to submit a separate PPR form.
If you prefer to use the PPR template as the previous question was alluding to, you can
do that. And you would then upload that into Grant Solutions with the rest of your continuation
application package. So that we are clear that all of the required pieces for the continuation
package are there. For the end of year report, you will use the PPR template and youíll
submit that via OLDV.
I don't see any other questions at this time. So at this time, we would like to thank the
Tribal PREP grantees for participating. Weíre going to dismiss you all from the webinar
so that we can continue with information that is pertinent to our PREIS grantees. So tribal
PREP grantees, youíre free to go. Now weíd like to hand it off to Ina Wallace from RTI
International who will be presenting information for all of our PREIS grantees on the performance
measures report. Ina?
>> MS. INA WALLACE: I am here. Okay. Well, itís a pleasure to be here today and I have met many
of the PREIS grantees previously, met them at the annual conference. So itís real nice
to be in touch with you again. And some of you, I have actually spoken to before since
weíve been back in regard to this. So some of this maybe old hat for some of you and
some of you and for some of you, it may be more. So weíre going to review what it is
we need to do for reporting your performance measures.
Okay. The first thing I wanted to let you all know are the various resources that are
available for you to use in completing your performance measures and reporting them. First
of all, we have a website and I think that every grantee has already registered with
one or more individuals. So that is great. And I hope youíve had a chance to look at
it and get familiar with it. There is a manual for using the website that you can access
right on the website. I did want to let you know that we are in the process of revising
it. We had to make some changes to the website which were completed last week. And now that
the changes are complete, we are finishing the manual. And we will let you know by means
of an email when the changes to the website manual are complete.
There are also reportings and transcripts of the different webinar trainings that have
occurred this year. And you can find them on the performance measures website. Under
resources you will see them. Theyíre also on the OAH website. And weíve given you the
links here.
Weíre going to talk later about the perceived impact measure and we developed a script for
your usage. And that is on the website. We have a help desk request on the website. And
we hope that this is something that if you have questions, you will use. When you do
this, your question is logged. It is sent to our webmaster who either answers it himself
or forwards it to another member of the team.
And the advantage of using that as opposed to my email or somebody elseís email is that
he monitors it all the time. When I leave here at the end of the day, I rarely have
my laptop with me. So your question will have to wait for the next time Iím in the office.
So itís much better to use the help desk because your question will be addressed then
or sent onto who needs to address it. And then as we find out there are other resources
you need, if there are things that you want, you let us know and weíll post them.
Next slide, please. There are three different ways for you to report your data to us. And
I want to discuss them because itís important that you understand what the differences are
because we want to make sure that you will be selecting one and using it properly.
Option one is reporting your raw data directly into the web system. And we have setup fields
to make this as easy as possible. And this is a particularly good way for any of you
who donít have a large number of participants in your study.
Option two is including the same kind of raw information, but using spreadsheets with predefined
variables. And we have those spreadsheet templates right on the website for your use.
Option three is entering aggregated data into the system. And only grantees who have approval
will be able to do this and will see the field that they need to enter the aggregated data.
Now, to date, weíve given only a very few grantees permission to enter data in this
fashion. And those who have done so have submitted detailed plans as to how they will calculate
everything. And most of the ones who have been given permission are doing so because
their data is coming from another system. There are a few who were given permission
because theyíre collecting data from several thousand students. Most individuals have elected
to use either option one or two. And we strongly recommend that because you will not be the
one who will have to determine, calculate everything. And some of these calculations
can be rather complex.
And one thing I have at the bottom that you need to indicate your selective reporting
option to your project officer no later than Wednesday, April 25th which is this Wednesday.
And if you select option three, youíre going to have to indicate how youíre going to calculate
all of those different measures.
Okay. Weíre ready for the next slide, please. I wanted to go through the different measures
that we have and how it is weíll be able to report them. The first group of measures
are those that concern program structure. And the first one is our measure of reach.
And reach is how many participants the program is reaching. And for your purposes and our purposes, it will
include all program participants, regardless of whether or not they are in the evaluation.
Thatís the most important point to remember. So even if they donít have permission to
be in the evaluation, they should still be counted if they are receiving program services.
And we have several measures. We have the number of youth served by different demographic
characteristics and we have the number of other types of clients who are served, such
as parents.
The reporting. Option one and two grantees will report participant level demographic
information. And we will be able to calculate reach because we will have your attendance
data. Those who are using option three, or aggregated data, will need to do this classification
by demographic characteristics before entering it into the system. And there are a lot of
demographic characters.
Next slide, please. The next grantee level measure is that of partners. And what weíre
asking are the number of organizations partnering with the grantee with formal agreements, without
formal agreements or just currently assisting in implementing the program. And then weíre
also asking for retention of partners. That is the number with formal agreements who are
involved with you at the end of the reporting period.
All grantees will be entering aggregated data on the website. So thereís no option one,
two or three here. Everybody will be reporting aggregated data.
Next slide, please. Our next measure is training. And we are interested in you reporting any
type of training that improves the facilitatorís delivery of the program. Itís not just the
curriculum material.
So these are some examples of things that you might include. If your facilitators go
to a workshop on classroom management or on adolescent development, that would count.
If you have any questions about whether a training that your facilitators receive would
count, contact your project officer.
The measures that weíre collecting are the number of new facilitators trained during
the reporting period and the number of facilitators to receive follow-up training. And for reporting
all of your aggregated data on the website.
Next slide, please. Our next measure is dissemination. For this measure, we are interested in those
kinds of manuscripts and presentations that you disseminate to the public. But the materials
should be related to the funding through the PREIS grant. And they might include your experiences
in implementing your program, the kinds of lessons youíve learned, the results of your
evaluation, anything that concerns this particular grant.
And the measures that weíre asking you to report on are the number of manuscripts that
are accepted for publication or were published. And if theyíve been published, weíd like
you to give the publication information. Weíd like you to provide us with the number of
manuscripts submitted for publication, the number of presentations youíve made by level.
And by this, we mean by local, regional or national level. Along with the title and where
it was you presented it. And finally, the packaging of your program for replication.
And all grantees are to report aggregated data on the website.
Next slide, please. Our next kind of grantee level measures are those that are concerned
with program implementation. And the first measure is dosage. And by this we mean how
much of the program the participants received. And weíre using attendance to calculate dosage.
And just like reach, it will be collected on all participants even those not in the
evaluation. So again, individuals may not have permission to be in the evaluation, but
they still should be counted through dosage.
So the measures, we are interested in the mean and the median percentage of program
services that are received by youth. And we need to have this information calculated by
gender and age. Weíre also interested in the mean and median percentage of program
services received by all other types of participants. And then weíre interested in the percentage
of participants that received at least 75 percent of the program. And again, for youth,
we need this by gender and age.
Now, the reporting. Option 1 and Option 2 grantees will report attendance data for every
participant, for every session, on the website or by uploading spreadsheets. And Iíve spoken
with some of you and some of you have situations that are not straightforward. You might have
a couple of different kinds of types of programs, some that are individually based and some
that are group-based. And if you have questions like this, send them to us and we will work
with you and your project officer to determine how it is you should report this information.
For individuals who are using Option 3, the data will need to be aggregated by the combination
of gender and age again. And I know this can get a little bit confusing which is why weíre
urging you to choose Options 1 or 2.
Next slide, please. Our next measure of program implementation is fidelity. And this will
address how well the implementation adhered to the programís model. And we have a couple
of different ways of measuring this. Weíre measuring based on the facilitator reporting.
And so we will be asking you to report on the percentage of completed sessions for which
there is this fidelity monitoring log. And then the mean and median percentage of activities
that are completed.
Then weíre also asking for ten percent of the sessions to be observed. And for every
session thatís observed, weíd like the observer to complete the exact same fidelity monitoring
log that the facilitators use. And weíd like the mean and median percentage of activities
completed to be reported. And thereís an additional measure, one that is a measure
of quality assessment. And we will be asking you to report that as well.
Weíll also ask you to report the percentage of sessions that were completed and a score
on a fidelity process scale. That, however, will only be measured once a year and you
will not have to report on that until next November.
So Option 1 and 2 grantees will report the required data for each session on the website
or by uploading spreadsheets and aggregated data for the additional measures on the website.
Option 3 will report aggregated data on the website.
Again, for those of you who have unique situations in that you donít really have sessions, please
contact us and we will work with your project officer to help you figure out what it is
you need to report.
Next slide, please. We have some participant level measures. The first is our measure of
perceived impact. And this is to be collected we hope anonymously, either at the end of
the program or annually for longer programs. And I apologize that I don't know how many
of you have long or standing programs. But if you have a program that extends beyond
a year, we ask you just to collect this once every year, a minimum of once every year.
You also will need to collect demographic data along with the perceived impact measures
so that we can analyze it. As I mentioned, we have provided a sample script for you to
facilitate the administration of the question. Because we know there were some issues about
how the questions were worded and whether students would understand it. So we have tried
to provide some clarifications that you can use. And again, itís on the website under
the ìResourcesî tab.
And there are four measures that are listed. Reporting. All grantees must report non-aggregated
data, either on the website or by uploading spreadsheets. There is no Option 3 for these
data.
Next slide, please. The second group of participant level measures are the measures of behaviors
and intention. And these are to be reported by the evaluators, not the grantees. And youíre
asked to report baseline and follow up data for intervention and control groups if the
youth have parental permission to be in the evaluation. And thereís a variety of measures
of both behaviors and intentions.
And there are also some additional measures that are not listed here that are used in
order to route the questionnaire. So itís more than six questions. Itís more than these
questions that are listed here. But for some youth, there may not be a lot if they havenít
yet engaged in any *** activity.
Next slide, please. The comparison groups. Now, all of you are doing rigorous evaluation.
So you will have comparison groups. Your comparison groups need toÖ they are involved in the
evaluation. And we would like you to collect and report the behavioral and intention data
if theyíre participating in the evaluation.
If theyíre not participating in the evaluation, theyíre not part of the control group. So
you will report those data and you will report those data for pre-tests and any post test
time that you have. Do not administer the perceived impact on comparison youth. Because
theyíre not in the program. So they wouldnít be able to address this issue. Do not report
attendance data on comparison youth because theyíre not in the program. And do not report
fidelity for any comparison. All you need to report on is the behavioral and intention
data along with demographics.
Next slide, please. Program Youth. There will be some program youth who may be in the program,
but do not have parental permission to be in there. This might be a situation in which
everybody in a particular grade is being given a program. And so there automatically, if
itís a school situation, theyíre automatically going to receive the program. But they may
not have permission to be in the evaluation.
In these cases, you need to provide information regarding reach and dosage, but you would
not be administering them any of the other kinds of measures such as the perceived impact
questions or the behavioral data.
Next slide, please. The reporting dates. Weíve gone over this before, but just so you remember
all data are to be reported twice a year and theyíre to be uploaded no later than November
30 and May 31 of each year.
The May 31st report includes data from September 30 to March 31st. And the November 30 report
includes data from April 1 ñ September 29. We urge you to enter data early. You may have
questions. And weíre just not going to be able to address all your questions if you
all wait until May 27th to start entering the data.
And as we said before, those who wish to use Option 3 need to send their project officer
a detailed description of the system for approval. And that will be by this Wednesday.
Next slide, please. If you have questions, I want to remind you to use the Help Desk
request on the website or contact your Project Officer. And this is what the Help Desk screen
looks like on the website. And you can see you will be entering your contact information,
your telephone number, your name. And when you click it, I think that will go to Eric
who is our webmaster.
So, I will entertain some questions now and Amy Farb from OAH is on as well. So she might
be able to answer some questions also.
>> MS. LeBRETIA WHITE: Thank you, Ina. We do have two questions that came in regarding
performance measures. The card says, ìIna, I was hoping to do Option 2 because Mathematica
is doing the spreadsheets for us. I do need to use Option 1 for fidelity [inaud.] forms.
Is this okay?î
>> MS. INA WALLACE: Itís absolutely fine. You can mix and match. No problem.
>> MS. LeBRETIA WHITE: The second question say, ìSince we have not yet entered training
data on the website, should we report all training that has occurred to date? So just
for this reporting period, we would tell you about initial training as well as all follow-ups
that have occurred from September of 2010 through March of 2012.
>> MS. INA WALLACE: No, my understanding, and Iíll let Amy clarify this if Iím wrong,
because a lot of what you were doing previously was sort of piloting and getting things ready,
we want it to begin with September 30, 2011. Amy, have I misstated anything?
>> MS. AMY FARB: No, thatís fine. I mean, if they were doing a training the day before
or two days before that really was full blown, part of the program, then we donít want to
discourage anyone from entering that. But we donít want all the early stuff.
>> MS. INA WALLACE: Thank you. Okay. It doesnít appear that there are any other questions
coming in at this time. So now weíre going to turn it over to Jean Knab. And we would
like to dismiss the PPA study grantees at this time. If you are in the PPA study, you
do not need to stay for this final portion as you have different requirements.
>> MS. LeBRETIA WHITE: We want to definitely thank Ina Wallace a well as Amy Farb for joining
us for the performance measure portion of our meeting. Thank you so much.
>> MS. INA WALLACE: Thank you. Bye-bye, everyone.
>> MS. LeBRETIA WHITE: And now weíll hand it over to Jean.
>> MS. JEAN KNAB: Great. Hi, everyone. This is Jean Knab of the Eval TA team. And Iím
going to talk about your annual reporting requirements regarding evaluation progress.
And some of you have already gone through this process. You had data collected in the
fall and have already submitted your assessment of that data to us. Some of you were able
to attend a webinar in the fall on this, but werenít able to go through the process yet
on your own. So we just wanted to make sure that we presented a little refresher for you
that was a brief overview of the requirements. Some illustrative examples and then an opportunity
for you to ask questions as well.
Next slide. So, to assess your evaluation progress, weíre going to request specific
details on two key aspects of your evaluation and that sample intake and baseline equivalent.
And these are important aspects of the evaluation that are too detailed for us to cover on our
monthly phone calls. So weíre going to use the annual and maybe a reporting process to
collect this information from you. And you saw earlier in the presentation the deadlines
for those.
Once we get that data from you, weíre going to look at that data and then provide you
back a written assessment of where the evaluation currently stands with regards to the HHS evidence
standards. And then weíll also review that data with you on a monthly call to discuss
how you can use that to inform your evaluation implementation, your data collection, and
ultimately your analytic plan.
So one question people might ask is how is this data valuable to us now? So two things
that weíre going to ask for again are the sample intake which weíre going to have you
put in through a CONSORT diagram and then also the baseline equivalent.
So if we look at the CONSORT diagram now, itís really valuable for pinpointing sample
loss throughout the intake and data collection process and understanding where youíre losing
samples, such as not turning in consent forms or absent for surveys or just proportional
loss say from one particular site or demographic group is really important for improving implementation
processes for either the current cohort or for future cohorts you might enroll.
And examining baseline equivalents on a partial sample will allow you to determine whether
there are baseline differences on key measures so large that would not allow you to attribute
the impact to the intervention. And therefore the study would not meet HHS evidence standards.
So if you observe these group differences early on, you could potentially target data
collection efforts to rectify those differences, for instance, on demographic characteristics
such as age by spending additional resources on targeted follow-up with certain students.
Next slide. So first, I just want to start with some general guidance, some questions
that we get from people. The first thing is do not provide data on your evaluation pilot.
I know some people have still been implementing pilots recently. For our purposes, weíre
not interested in assessing the evaluation pilot data, only the final evaluation sample
that would go in your final report.
Next. We want you to provide data pooled across cohorts in place currently enrolled. Most
people donít have multiple cohorts enrolled yet. Although, some have some very short term
cohorts that they might actually have started on a second cohort. But essentially pool the
data across all the cohorts and all the sites that you have. However, we do encourage you
to look at data separately by cohort and site. Because you might find that your first cohort
of youth is not based on equivalent, simply by chance. And knowing that early on would
allow you to target your recruitment and data collection efforts for round two to attempt
to rectify this for the full sample.
Weíre asking everyone to provide equivalence data for up to two samples. And this is whatís
a little different from our last request. So the first request is the same. Weíre saying
everybody who is in your full sample at baseline please present baseline equivalence for those
groups. And again, that might be like say the full sample for your first cohort and
a partial sample for your second cohort aggregated together and thatís fine.
The new request is if you have started collecting any outcome data yet, we would like you to
provide baseline equivalence for that sample. Thatís effectively your current analytic
sample. Anybody who has an outcome data point. And then weíd like to compare the baseline
equivalence for that sample which is the sample responding to any follow up surveys.
And if youíve already provided data in the last round and you havenít collected any
new data, so there might be some people who did enrollment in the fall. Say itís a school
based design. You did enrollment in the fall. You havenít collected any outcome data yet.
Maybe youíre not going to collect it until the end of the year. So effectively you have
no new data to present to us.
In that case, you donít have to present anything in your May 31st reporting. You can just send
us a note to say we have not collected, you know, we have not brought any new sample into
the evaluation at this point and we havenít collected any new outcome data yet. I donít
expect many people will fall into that category. But if you do, you can just indicate a note
to us.
And the other thing is that this is due May 31st as has been indicated previously. And
some people ask at what point should we cut off the sample to include? Letís say youíre
actively enrolling sample right now. And the guidelines weíre giving people are to follow
the same reporting schedule that youíre using for the performance measures. So Iím not
sure if that cutoff is February 29th for the data that you need to provide by May 31st.
But essentially whatever that cutoff is, itís a good cutoff to use for us as well since
youíll have to process that data anyway.
Next slide. So letís talk through the CONSORT diagrams for a moment. I just wanted to mention
that there are examples available in the continuation application guidance. LeBretia sent that around
to you I believe. And there are also templates where templates available online on our share
point website and there will be a link to that later.
So in terms of the CONSORT diagrams, for clustered, randomized control trials, we need you to
do two CONSORT diagrams, the first one which is describing the cluster intake and then
the second one which is describing the youth intake. And then in that one, itís important
to note that we only need youth who are still in the remaining clusters. And weíll show
an example of that in a moment.
For individual level RCTs and QEDs, you only need to do the youth intake CONSORT diagram.
Next slide. This is just a brief screen shot of what the first CONSORT diagram should look
like. This is an example where thereís a grantee with clustered random assignments
and grantee X has a cluster randomized control trial, a sixth grade program within a single
school district. And now weíre going to take a look at each of these.
Next slide. So, the first thing that we want to see is a paragraph on the intake process
for clusters. And Iím going to read this one as an example.
ìThere were 30 schools in the district that were serving 6th grade youth, which is the
target population. MOUs were obtained from 22 of the schools. Ultimately, only 20 schools
were included in the study, due to constraints on staff time. The 20 schools were selected
based on school size (the 2 smallest schools with MOUs were not included).î
So the key items covered here were the eligibility for the evaluation and any exclusions that
were made prior to random assignments. It also includes the timing of consent relative
to random assignments. They got their MOUs first. And it talks about non consent. And
this is really important for understanding generalizability and representativeness of
the initial sample and is often sort of lost documentation later on if it isnít put down
early in the process.
Next slide. And this is the more traditional component of a consort diagram that youíre
used to seeing. Essentially, we start out with the number of clusters randomized to
treatment and to control. And then it documents some sample loss during the baseline data
collection process. On the right hand side, you can see there was one comparison school
that left the evaluation because the principal left the school and baseline data could not
be collected in time.
So one thing to note is that you want to make sure your documentation reflects the order
in which the activities occur if itís not apparent. For instance, the random assignment
of youth relative to the consent of youth or the assignment of youth to the condition
relative to the random assignment of clusters. So you can use our Word template and move
them around to really customize them to fit the design of your study.
Next slide. So now weíre going to continue with the youth CONSORT diagram for that same
evaluation. And you will note that here weíre only going to include youth from the 19 studies
that are still participating. And the reason for that is when we do the attrition calculations
for the HHS evidence standards, we donít want to double count the loss of students
in that one school. So weíll already have factored that in when we look at cluster level
attrition. We donít want to count those again when we look at the loss of youth.
So here again we start with an intake paragraph. This says, ìAll 6th grade youth enrolled
in the 19 randomly assigned schools as of September 15th were eligible for the evaluation,
with the exception of students in self contained special education classrooms.î And then again,
it talks about not including the school that dropped out.
And then we go through the number of youth eligible in those schools, the number who
consented, refused, didnít return forms. They spec out again the exact number of students
who are not eligible. And then we go through those who completed baseline as the baseline
of those who consented, listing the dates of data collection and the reasons for non-complete.
Breaking out the refusals of non-consents are very valuable for examining as you go
into the field for future data collection and also the dates are very important to look
for overlaps so that there are no compounds in data collection compounds between the two
groups.
Next slide. So here weíre going to construct a similar CONSORT diagram for youth enrolling
in an individual level assignment design or in a QED. And I wonít read the examples here.
One thing I did want to mention is that while HHS evidence standards donít require attrition
to be assessed for quasi experimental designs, weíre asking those who use those designs
to provide this because it is really important for determining whether thereís been intervention
induced loss and at what stages in the process that that has occurred. Which can be really
important when thinking about your future implementation and the representativeness
of your final analytic sample relative to the sample that you started with.
Next slide. Iím not going to cover this in detail. It looks much like the other one.
But you should absolutely refer back to this as youíre doing your examples. One thing
to note on the bottom is that the sample sizes in the last two boxes, some to the number
of students consented and randomly assigned, not the number of students with baseline data.
So in randomized control trials in particular, efforts should be made to collect follow-up
data on all students who consented, regardless of whether or not they had baseline data.
Letís move onto baseline equivalence. Again, we want to pool data across sites and cohorts.
And weíre going to focus on variables assessed under the HHS evidence standardóin particular
age, gender and race/ethnicity and measure the *** behavior. There is an Excel template
available on the evaluation TA website and there are two tabs in the workbook. Thereís
a blank template for you to use and thereís also a populated example. And if you need
to provide baseline equivalents for two samples, that is the whole sample with baseline data
and then the subset of those who also have outcome data, you can just make a copy of
that one worksheet and provide us two in the same workbook.
The other thing I wanted to mention is there are two different versions, depending on the
version of Excel that you have. And for those of you who have already provided data to us
in the fall, weíve revised this template to try to make more clear where you need to
enter data than what the spreadsheet calculates for you. So please use this new version, even
if you used the other version previously.
So what weíre going to ask you to provide is any binary measures or those measures that
are answered just as a yes/no variable. Weíre going to need the percent who responded yes
in decimal form. So, for example, 0.05. And thatís very important. If you just put five,
ten, fifteen, the statistics arenít going to calculate correctly. And then also the
sample size for each measure.
For continuous measures, we need the means, the standard deviations and the sample sizes.
And for the categorical measures, such as race, we only need the counts in each category
of treatment and control. And weíll look at an example of that in a minute. And then
also, you should provide any documentation of deviations from those basic statistics
such as if you calculate statistical tests accounting for clustering. You can just note
that to us in a column on the side.
Next slide. So in terms of the required data processing, we try to minimize that by trying
to overlap as much as we can with the performance measures. But we know that you will need to
construct a race variable, recoding those, selecting multiple races into a ìTwo or more
racesî category. So that we have mutually exclusive categories there. Youíll also need
to construct dummy or binary variables for any of the yes/no survey items for gender
and for Hispanicity.
And then finally, weíre going to ask you to construct full sample *** behavior variables.
So when we look at baseline equivalents, we want to see is the whole sample equivalent
on this measure. So not only the subset that might have answered based on the skip pattern.
So essentially, youíre going to be imputing zeroes or noís anytime that you know the
answer to the question based on the skip pattern. So, for instance, youth who did not have sex
should be in the questions later on as to how many times they had sex, et cetera, but
coded as zeroes.
So, a specific example is youth that did not have sex in the past 3 months should be coded
as zero in the number of times had sex in the past three months and included in the
denominator. And youíll go across all the variables and do that for us. And if you have
any questions on that, you can certainly follow-up with your TA liaison.
Next slide. I apologize that this is small. The numbers arenít important. Weíre really
just going to review the structure here and the pieces that require data entry. So this
is what the new spreadsheet looks like and this is the part that examines based on equivalence
for demographic measures. The yellow cells are the ones in which you need to enter data.
And the shaded and Xíd out cells are calculations performed by the spreadsheet or data that
isnít needed for you.
One thing that we want you to provide sample sizes for every measure because you might
have item non response, particularly for the behavioral measures. Or you may not have processed
all the data youíve collected and that was reflected on the CONSORT diagram. So we will
need you item by item to enter sample sizes here. And so again one thing that youíll
notice that the race/ethnicity variable is different than the others. We only need you
to enter the counts that are in each category and then the spreadsheet will calculate across
squares for you.
One thing some people have asked is whether or not they need to provide data for each
of those categories. Sometimes you may not have many people in some of those cells and
they arenít necessarily racial and ethnic groups of interest for your evaluation because
they are so small. And in that case, you should absolutely feel free to collapse some of those
categories into more meaningful categories for you. Just be sure that you re label the
spreadsheet so that we know the categories that weíre looking at.
Next slide. This is the section of the spreadsheet that looks at the *** behavior measures.
Again, youíre going to enter your binary data as decimals and no standard deviations
are required for those. So youíll notice that the sample sizes within each group here
are identical, even in the last measure which is only a small number of people who would
have had *** intercourse in the past three months without using effective contraception
and had a number of times variables. But again, we want you to impute zeroes for everybody
who did not make it as far as that question. So that we have a full sample measure. And
again maybe your sample sizes will vary by a few people because of some non responses.
But it shouldnít be, say, two or five kids when you really have about 150 kids who answered
the survey. I think those are the key points to hit on that. You can go to the next slide.
So the templates are located on the evaluation TA website under Shared Documents/Annual Progress
Report Requirements. There are Word templates for the CONSORT diagram and then the Excel
workbooks for baseline equivalence and the two different versions. Again, if you have
questions, you should absolutely follow-up with your TA liaison or your project officer.
And if there are any questions now, Iím happy to answer those as well.
>> MS. LeBRETIA WHITE: Thank you very much, Jean. We are going to take a look and I don't
believe that there are any questions that have come in at this timeÖ There are no questions
at this time. Weíll hold for just about fifteen, twenty seconds to see if anyone enters any
last minute questionsÖ I guess not. I don't see anything coming in. So I think at this
point, we would like to thank all of our presenters for taking the time to speak with our grantees
today. And we would like to thank our grantees for participating in todayís webinar. We
hope that it was informative and that you found the information useful. And we will
look forward to continuing to work with you on the continuation applications and on your
projects in general. Thank you so much for your time. And if you have any questions,
please contact your project officer. Have a great day.
>> Bye everyone.
>> Bye-bye.
>> Thank you. Bye-bye.