Tip:
Highlight text to annotate it
X
>> LAMBERT: The key issue is representativeness. The key issue is are you
representing your population of students? I don't think that there's anyone that would
defend a twenty percent response rate but can you get a representative sample of your
students particularly in a large class with sixty percent response rate? Sure. No question.
One way to think about it is, you could have a ninety-five response rate and have a bias
result. Right? Because if you have five percent of your students who are in one sub group
and none of them responded, a ninety-five percent response rate could give you a bias
result. On the other hand, if you had a sixty percent response rate but all the subgroups
in your class responded, you could have a very representative sample with a sixty percent
response rate. So that's an important thing to remember about it that the clearly higher
response rates tend to give more representative samples, no question, I mean that's just common
sense. But it is not the case to say that response rate directly equates to a representative
sample because it does not. And there are even empirical studies that in survey studies
that show that, you know, you can get a representative sample with a relatively lower response rate
and you can get a bias sample with a higher response rate. So that's an important thing
for people who are worried about this to think about. The issue is representativeness, this
issue is the validity of the responses, not the actual response rate. Although clearly
high response rates tend to be better and tend to lead more closely to representative
samples and more valid feedback. And so we should all do our due diligence in trying
to get the response rates up. Ensure students that it's confidential and anonymous just
like it was before and that we really want their feedback and that their feedback will
benefit future students because quality instructors tend to use that to improve their courses.
\ [cross fade]
Measurement error in these kinds of surveys tends to be a much bigger issue than sampling
error and so what we really need to be concerned about is considering it is not a closed case,
as we make this transition we can continue to evaluate the quality of information we
get and continue to improve the quality of the surveys. If we could move towards standardizing
the measures and using uniformly high quality items that are understood in the same way
by all students we'll get more liable and valid responses. And I think that's going
to be actually a more important place to focus on than to focus on the data collection modality
issue. Many, many years of research now has shown that general data collection modality
issues do come into play in the quality of evidence but for the most part we've transitioned
from phone and mail and face to face surveys to online surveys and we still get a lot of
high quality information and there's not any big movement to turn the clock back to going
back to paper surveys. I mean, that's not going to happen in our society. So then the
bigger issue then becomes the quality of the items themselves and is it giving us valid
feedback? Is it representing the voice of the students to the instructors the way we
want it to? So the great thing about the online method is that is continues to build a base
for us. Every time we use it, we have immediate access to a large amount of data that we can
use to continually use to improve the questionnaires themselves. And that will allow us to minimize
the measurement error in the questionaries which is probably a much bigger concern than
actually representativeness of samples or the data collection modalities. So, at least
in our center we look forward to continuing to partner with you to use these large data
sets that we collected every semester to improve the quality of the questionnaires themselves.
And I think what we saw in our pilot study and what we're beginning to see after this
first roll out of the online method is that we are getting at least a majority of the
students to respond and that the response rates will continue to increase as we all
get more comfortable with this method. As students get more comfortable with it and
as professors get more comfortable with how to remind their students in a professional
way that we do need their responses. }