Tip:
Highlight text to annotate it
X
So I'll just try to briefly go over the second study that I did and this is the one where I'm comparing
what students learn from discussing with their peers and what they learn from me during that whole...you know,
I want to know if I'm still useful to the students' discussion.
So this is just a schematic of how peer instruction works.
So I pose a question, the students respond individually, then they discuss in the group discussion there,
and then the students do another vote after the group discussion, and I'm calling this...I'm calling these "Q1's",
so, question 1, and then I'm calling the second vote "Q1AD", so question 1 after discussion,
alright, you know, so we're taking 2 votes on the same...on the same question here and then after the Q1AD vote,
then the instructor does the...the classroom discussion,
the kind of conclusion to the question and explaining all the distractors, discussing the concepts,
discussing the misconceptions and so on. And that's my interest, right, what am I adding as an instructor?
When I do the...the discussion at the end, right, the classroom discussion, the sort of mini lecture you could call it,
what am I adding to the learning gains that the students are exhibiting on these questions?
So I...I use something in my, in my next research study, called isomorphic questions, and these are questions that
target the same concept, but are different questions and the reason I need to use these is that and you'll see in a couple of slides,
but I'm going to ask one of these questions, and I'm going to let students discuss, and then I'm going to ask the
second of these questions and my interest is what is their performance on the second of these compared to the first, right?
If they're learning anything, then the performance on the second question should be much higher
than it was on the first question, right?
And so I use these isomorphic questions to here to try to test the exact same concept and do it twice and then
compare...compare the percentages correct that students get.
So, so for the CS people in the audience, they're both complexity questions, the only difference is that one of them has a constant
in the J loop and one of them has a variable in the J loop.
But with no reference to computer science at all, the structure of these 2 questions is exactly the same,
and that doesn't mean that they're isomorphic; it's very, very difficult to create 2 questions that test exactly the same concept
but I hope that, that I've succeeded in doing that because the study kind of relies on the fact that I'm asking 2 questions
in a row and they are testing the same concept.
So, what I did for this study is I have 2 question modes that I used in my lecture.
So, I'm only doing these modes for the purposes of research here.
This is not something that if you want to use peer instruction that you should...you shouldn't really use these modes
but I used these just in order to measure learning gains from peer discussion, which is the P-mode here
and peer discussion plus the effect of the instructor's explanation, the C-mode.
So what I'm doing in both of these modes is in the P-mode,
I'm having students answer question 1, talk to their peers,
and then immediately answer question 2 and the rationale here is question 2,
if they're learning from their peers, question 2 should be higher than question 1.
And then in the combined mode before question 2, I'm also giving students my explanation of the question,
so before they answer question 2 in the combined mode, they're not only getting their peer influence,
but they're also getting my instructor influence.
And the question is -- what additional benefits are the students getting from me compared to only discussing with their peers?
So, hopefully, for most of our sakes here, hopefully the C-mode has way higher learning gains than the
P...the peer-only mode, then that would suggest that the instructor is adding significant benefit to what the students are learning.
So how I've worked, how I designed the study is based on some earlier studies, that also used isomorphic questions,
and these earlier studies are sort of a subset of the study that I'm going to describe next.
So I'm not going to go into these earlier ones but their results that I present will confirm what was seen in these earlier ones as well.
And then my study also adds the instructor explanation piece.
So this study was run at UTM, in the fall. It was a CS1 course, it was 3 lectures a week, I had reading quizzes before lecture,
and I had the students discussing using peer instruction in lecture so quizzes were worth marks
and clicker responding was worth marks and in both cases it was just responding, not having to respond correctly so, just participation only.
And what I did was in each lecture, I used one of the...one of the two modes, the peer mode or the combined mode,
and I asked the isomorphic question, and I compared the performance on Q1 and Q2.
So there's a lot that can go wrong in studies like this, so I want to try to address some of it, one question is --
are questions actually isomorphic, right, can we claim that two questions are of equal difficulty and test the same concept?
I still don't fully know the answer to that question but what I did do was I sent all my questions to one of my colleagues
who also teaches CS1 and I had him tell me, just looking at the two questions,
if he thought that the two questions were testing exactly the same concept and if they were of comparable difficulty.
I mean there's a bit of a flaw here, in that I don't care if another instructor thinks they're isomorphic,
I really care if students think that the questions are isomorphic and that I didn't do, so, one follow-up I should do is look at whether the
students believe that the two questions are asking the same concept.
What I did was I also randomized everything, so I randomized the mode that the questions would be asked in,
the peer or the combined mode and by doing that I found that questions were equally difficult, so there was no mode bias here.
So if I find an improvement, then I can say that it wasn't because questions were easier or harder than other questions.
And what I also wanted to know was what are the effects of discussion and instructor explanation on subgroups of students?
So I also divided students into weak, medium, and strong students based on their performance on question 1;
the question is, I want to know overall if the instructor has an effect, but I also want to know within specific subgroups of students
if the instructor has an effect. I don't want to only help, for example,
the strong students or the weak students; hopefully the instructor helps all types of students.
So, show some of my results here. So the y- axis is showing percentage correct on these questions
and I've broken the questions down...sorry, I've broken the responses down based on the subgroup of students;
so I've got weak students, I've got average students, and I've got strong students here, and then
I've got overall students on the extreme right of the graph and what I'm doing within each set of bars
is I'm comparing the peer-only mode and the combined peer and instructor mode.
So you can see for example, let's focus on the left side of bars here,
so for the Q1's, I've got 29 and 23% so there's a bit of a difference there, not statistically different, but,
the Q1's are not exactly equal and that's probably just because I didn't have enough questions to make the percentages closer there.
But more interesting in this graph is looking at the Q2's.
So in the peer mode, students answered correctly 54% of the time and in the combined mode,
students answer 64% of the time, right, if we look at the next group of bars, the medium students,
we see that students go from 66% to 79%, so another big increase when the instructor is allowed to explain the question
before students answer and then even for strong students, there's still an improvement of the instructor,
so from 83% to 93% correct so on all subgroups of students and overall here, the instructor has a pretty large benefit,
10% at least benefit, on the percentage of students that answer Q2 correctly, the second of the isomorphs correctly.
So what I found pretty interesting in these results is I don't really think I did a great job of explaining the question as part of my instructor
explanation there and the problem was that I knew that the Q2 was coming up next and so I had to be really careful not to give away Q2,
right, because like, you know, a good strategy is after each question, you say things like what if I change this to that
or what if I replace this with this or what if I, you know, delete that or like make...doing all of these little changes to the question,
and I couldn't do that because if I did, then I would've given away the question that was coming up, the isomorphic question.
So I think...I think I could have done better than what the previous graph shows
if I could have explained the question in the full generality that I wanted to explain the question in.
So I -- one other hypothesis that I wanted to run by you and that one is - okay so I know the instructor has a pretty large effect for all
subtypes...for all subgroups of students, right, weak, medium, strong, and all students, but I've had this idea
that maybe I'm even more important on difficult questions, right?
Maybe the idea is that if a question's easy, right, students can discuss it in their groups and make some progress,
but maybe if the question is difficult, they can't do anything, right? They're just, you know, we can't make progress, the question's too tough, right?
And so I thought maybe on those difficult, really, really, difficult questions, the instructor would have an even larger effect
than what it showed on the previous graph.
So there was some conservation involved here, but I made some nasty, nasty questions for students,
I mean I felt really bad making them and giving them...the students did really, really poorly on these questions.
I don't think they were unfair, I just think that they were maybe a little more difficult than I should have given in a CS1,
but I did and I wanted to know what happens now --
so I give these really, really tough questions where students get below 50% correct on the Q1 and now what happens, right?
What's the effect of the instructor now when I give students these extremely difficult questions?
So, another graph; I've got again, on the y-axis of percentage correct and I'm showing the performance on only question 2 here,
so only on the isomorphic vote here and I've broken it down based on the difficulty of the question
and also whether students get Q1 correct or incorrect.
So, if we focus on the left set of bars here, we've got from 54% to 73%,
the instructor adds almost 20% improvement there on difficult questions that students get wrong initially,
so if the question is really, really tough and students get it wrong initially then when asked a second question,
if I gave instructor explanation, there's a huge 20% improvement there, okay.
The instructor also helps everywhere else in the graph, you can see that Q2 is always higher for the combined mode and for the instructor mode,
but the biggest effects of the instructor, are when students get the first question wrong,
so - sorry - 64% to 78% is the gain for easy questions when students get them wrong.
So it's almost as if the real benefit of the instructor here shows up when students don't know how to answer the first question.
When they can make some progress, then the peer discussion is extremely productive
and the instructor explanation is less defined than it is when students don't know how to answer the first question correctly.
I really think that this explains exactly the purpose of an instructor right, to help students with difficult material that they don't understand.
If the material's easy then, we're not really going to add much, but, conceptionally difficult material,
really, really challenging, core course concepts, this is where the instructor really makes a huge difference.
So, just to conclude a little, peer discussion is very, very valuable; it always increases the percentage on the,
on the isomorphic question there, but the instructor explanation is -- on top of peer discussion -- is,
is even more valuable than peer discussion on itself...on its own.
Right, and this is especially true when questions are difficult, on the most difficult questions,
the instructor explanation has that 20% improvement that I, I just showed on the previous slide right there.
So, in conclusion, I think that we have, we have Two really, really important roles, as an instructor:
One is to prepare students for class, that means giving them these reading quizzes, giving them readings that are appropriate before lecture,
and then using their responses from those reading quizzes to shape our lecture,
but the second role that we have is our explanations in class even when the students are using peer discussion,
we still have a...very important role to play in that our explanations to students are very valuable
especially when the material is difficult. So we have a lot of resources posted,
some of them are computer science-specific and everything I've done is computer science- specific
but there's also a website where there is materials for other courses as well
and I really enjoy discussing this kind of stuff if you wanna discuss this further, I'd be very happy to.