Tip:
Highlight text to annotate it
X
Dan is currently a PhD candidate from OISE who will finish hopefully this year,
I think everybody wants to finish their PhD.
He does research on student learning, types of computer science assessment,
and peer instruction. He's also interested in online learning, um but prefers computer science.
And this year he taught two courses for us at UTM, Introduction to Programming and Software Tools and Systems Programming.
And he's also taught a number of other computer science courses.
He has awarded himself the most entertaining CSC108 instructor.
I'm really disappointed there were no computer science students here today to verify this but we'll let him know at the end of this research
talk what we think. Dan, thank you very much.
So thanks everyone for joining us today and thanks to Sue and Simone and Andrew and everyone else for getting this scheduled
and getting me here today. I'm going to go through a couple of research projects that I've done over the past year.
One of them was at St. George and one of them was at UTM. And they both involved this pedagogy
that I've been using in computer science called peer instruction and it's interactive pedagogy that I've been using
in computer science, called peer instruction. And it's interactive pedagogy where students are discussing a lot and voting on questions, responding to questions and
we'll get into all details of the pedagogy. But before that, I wanted to first discuss why we're trying to do pedagogy, maybe lecture's okay,
we'll see. And then after that, I'll discuss what peer instruction is, what I mean by that,
it's a specific type of pedagogy, it's got a well- defined set of steps that you carry out as a lecturer. And then I'm going to go through
two studies that I've done and try to give you a flavor of the results.
So one is how to prepare students properly for these types of classes where there's very little lecturing and a lot of talking
and then I started to get a little self-conscious, right, so, you know students are talking so much...what's my role as the instructor?
And so...then I tried out another study where I validated that I actually still do something, which is nice.
So even in this interactive pedagogy, the instructor's still useful. So a bit of a spoiler there, but I think our topic's safe.
So I think this talk will...will help you to realize that.
So just to kind of discuss the effectiveness of traditional lecturing for a little bit...This is a quote from Carl Wieman,
He's a physics lecturer. He has a Nobel Prize, I don't, and he has studied how to improve lecture for many, many years
and he comes to this conclusion here that he's doing these wonderful lectures and students are not really understanding very much of what he's
trying to say in his lectures. He's actually carried out a few studies that I think are pretty interesting
and it kind of made me a little depressed but... so what he did in one study was he told his students he said
"okay students, listen up, I'm going to teach you something that's really important and then in 15 minutes
I'm going to quiz you on what I just taught you, do you understand that?"
And students said "yeah, yeah, it's fine, it's easy, you're going to tell us something and then you're going to quiz us on it, no problem."
So he did that and 10% of students got the quiz correct and maybe even more concerning than that
is that in physics there's this concept inventory, it's called the force concept inventory, FCI,
and it tests apparently basic understanding of force and motion but I --
it didn't seem all that basic to me when I took it -- so students are supposed to do well on this when they've finished their first physics course,
and so if you give students this concept inventory after the first physics course, they get 30% on it, which is bad.
But it's not as bad as what else happens, which is that they actually do worse after the first physics course
than before the first physics course so the lecture is having a negative effect on the student's performance on this.
This is just Carl Wieman's results; I don't think that this would hold with...with all lectures.
But the fact that that this lecture, he's trying really hard and who has studied education for so long has got these negative results,
is really concerning to me. So there's this proposal that he gives and that others give that in order to improve education,
we've got to take a scientific approach to it, right?
It works for science, we've had considerable scientific advances and so maybe we should apply the same kind of thinking to education
right? And so that involves using techniques that have observable outcomes and then telling, you know,
communicating our results to others and treating education as a science.
It is really hard to do, right, because students are not exactly as predictable as bacteria and things like that
so it's not quite as easy but I think the notion makes sense. And so we've been studying peer instruction,
this alternate pedagogy, this alternate to lecture, for a few years in computer science and you can be the judge,
but I think we have some evidence that I'm going to consider evidence-based,
as scientific as I think we can get in education and so I'm going to try to show that we've demonstrated things using peer instruction
that are, that have a basis in objective data so.
So what is peer instruction?
So like I said, it's this active, interactive learning pedagogy that, that students - where students discuss a lot instead of the instructor lecturing
and how it works is we replace lecture by this, this sequence of steps
and it's always the same sequence of steps, so there's no creativity involved,
which is good, because I think it would be a lot more difficult to start using this pedagogy if you had to be creative every time you used it.
So how it works is the teacher poses this multiple-choice question and then students have clickers
and they vote on the question once on their own without talking to their neighbors.
It can be difficult to have the students not talk but after a while it becomes, you know, second nature,
the students know that they first have to think about the question on their own and then respond and
then we have students discuss the question in groups and then the students vote again.
So on each question we've got two votes, we have the vote where they haven't talked to their partners
and the vote where they have talked and they're voting twice on the same question here
right and then after the second vote, the teacher carries out this overall kind of discussion on the question right --
okay, what are the misconceptions here, what's the major concept we're trying to convey using this question?
And then we move on to the next question. And we start again by posing another question and having students vote on it and stuff.
You can usually get through like 3 of these in a 50 minute class. I've tried to do more than that,
but at some point students are just like "no, no, no, like that's it, like I've worked way too hard today, it's too early for me,
you already gave us 3 questions, that's enough right."
So, but we are though, we're asking students to do a lot more work in this lecture right,
they can't just, you know, show up and sit there for an hour and assume they learn something, right
they have to actually be active the whole time, right.
And, and my students still do what you expect, like there's still some students on Facebook and things like that but it's
more difficult not to be paying attention if everyone else around you is discussing these questions and so on,
so, in my experience it really gets students involved. But there's an interesting kind of body of research
that we've got using peer instruction in computer science and just briefly to go through some of this prior evidence that we have.
So you know that we have 2 votes for each question right, we have the vote where students vote on their own
and then they discuss and vote again and we see immediate gains between these 2 votes,
but we don't know necessarily if the students are actually learning things or if they're just copying off someone
who they think has the answer right, like, just, you know, just tell me what the answer is and I'll click that one;
and we'll get to that in a sec, but you'll notice that in the table here, right, students are very, very positive to this pedagogue
and this to me is extremely encouraging, asking students to do this much work in lecture and then having like 97% of them
saying to you that "yeah, I wish everyone would use this", this tells me that students are really understanding
how useful this is for their learning. We have other evidence that it's effective as well, in this one,
the instructor taught two sections of the same course -- and same book, same lab, same exam, same instructor, same everything --
except that one section was peer instruction and one section was lecture and peer instruction did considerably better than the other section.
We also found that peer instruction decreases fail rates by quite a bit,
so fewer students are failing these CS courses, and it also - it also reduces fail rates within the instructors.
So if we have an instructor that switches from lecturing to peer instruction, their fail rates also decrease too,
so it's not just an artifact of good lecturers using peer instruction,
it actually happens for everyone who switches from lecture to peer instruction.
The research that I just went through quickly is showing that peer instruction as a whole is effective
and I started to be interested in one of the individual pieces of peer instruction that are effective,
and my rationale for this is I want to work on the little pieces of this pedagogy in order to make improvements
and try to improve PI further than what we've already managed to do.
And my focus in the rest of this talk is going to be on the instructor's role in peer instruction,
so I've described how students are discussing with their peers and such,
but I wanted to know what the purpose of the instructor is with all of the shift to this student centered kind of learning.
So, I'm going to go through a couple of studies where I try to assess the importance of the instructor.
Alright, so, there's this interesting catch-22 we have where in a lecture,
we want students to be discussing with their peers on these difficult questions that we're posing, right,
but one of the first times I tried to use this, I was asking students questions, and they were saying,
"sorry Dan, you know, I don't know what we're discussing today, I don't know anything related to programming,
how are we supposed to discuss with anyone with any sort of productivity if I don't know anything at all?"
So I started to realize that okay, so we can't have students showing up unprepared for class anymore, right?
You know you can't ask them to show up and then...and throw a question out there, right,
they're just not going to discuss it, they're just going to say that I'm crazy and then discuss their weekend or something right?
So what we started doing is we had students prepare for every lecture by having them read a little piece of the textbook
and complete a quiz before class. Okay, so we're all probably thinking, yeah, yeah we all ask students to read before class
and they never do it, right, and so there's a reason though in peer instruction that they have to do it
and the reason is that if they don't and they show up to class, they're going to be completely clueless, right,
and you do that to students enough times and they're going to realize okay, this is serious, I really have to do the reading before lecture
or I'm not going to know what's going on, right?
And so what we do on these reading quizzes before lecture, I have a little software thing that I wrote, it's like a web interface where I post
some questions and then students respond to them and then they submit their responses to these questions before each class.
Okay, so if we have, what is it? We have 12 weeks of lecture and 3 lectures a week, that's 35 or 36 quizzes
that students have to do and they don't love it, because that's a lot, right?
That's a lot to ask students, like 36 quizzes and so keep that in mind,
you'll see that maybe students aren't as negative to it as you might expect.
So, in addition to those content questions, we also have to ask students this really important --
it's called the confusion question -- and so on each quiz, we ask students: okay, tell me what you found difficult,
what you found confusing, what you found interesting, what was challenging, right, help me make the upcoming lecture more useful right?
And then the idea is the instructor tries to read these responses quickly before the lecture and have an understanding
of what students know and don't know.
So, they have a lot of questions related to these reading quizzes,
perhaps the first one and the most obvious one is do students do them? I mean are they going to do these quizzes at all and if they do --
are students going to give me reasonable responses?
Or are they going to just give me junk and submit the quizzes for the sake of submitting them?
And then also, what kinds of information are students going to give me from these reading quizzes --
are they useful to the instructor of the course?
So this current study, I ran it at St George campus, it was an engineering course and it was a course where it was
the -- it's called the T program -- so all of the students were unsuccessful and they took the same course again
in a remedial offering so they'd all failed the course the previous semester.
And so one reason I used peer instruction in this course was that I didn't want to be lecturing again, it didn't work the first time, they didn't
pass, so I wanted to try to change things. So it was a reasonably small class,
it was 40 students, and I used reading quizzes before each lecture and I made them worth 6% of the student's grade,
and maybe this is interesting or surprising but I decided to mark them based on submission
so I told the students, "you can submit whatever you want, it doesn't matter, you'll still get the marks, for those...for those quizzes"
and in the back of my head I was like, I don't know, I don't know what's going to happen here right like what am I going to get,
what am I setting myself up for here right, are students just going to, you know, submit like space, space, space, space, space,
and then try to get the marks for that or are they actually going to put some effort into their reading quiz submission?
So they're worth 6% and it's not surprising that a lot of students submitted these things, right,
I mean these are free marks, you might as well submit. So almost 80% of quizzes were submitted, and I was happy with that,
the students on average missed a few quizzes, but there's 30-something of them so it's expected that students are going to miss them
but the question is okay so they're submitting these things but what are they submitting right?
They're getting the marks for submitting only so are they actually putting in some effort for this or
are they just sending you know whatever they want to send before bed or whatever?
So it didn't count for marks, but after the course was over,
I looked at the types of responses I was getting, so I marked it as I would've marked a test,
and so I had like a little marking scheme where I marked each question as empty or incomplete or incorrect or correct,
and I was really pleased with what I got so the empty responses, the 1.8% responses,
were the ones that I was petrified that I was going to get like "acdefg", you know, I got a few of those, but only 1.8% so not bad.
And then the incomplete ones were responses that made some progress towards the solution, but they just stopped, right,
maybe they were working on it, and then they got a phone call or something and they just said,
you know what I don't care anyway, I get the mark so I'm going to submit it regardless --
so 10.5% of those.
But what was encouraging to me was the incorrect and correct responses were all complete
and I'm totally okay with students being incorrect on these quizzes because the lecture coming up is going to clarify all
this...well, I think it will clarify all this stuff for them. So almost 90% of responses that I got from students were I think,
evidence that students were engaging with the reading to the level that I wanted them to engage.
So I had only 1.8% of students taking advantage of the fact that I was giving marks for nothing, so I was really pleased with this.
Looking now at the confusion question, the one where I asked students to tell me what was confusing and difficult and things like that,
I coded all the responses, there was tons of them, it took me forever, but I think I have some interesting results to convey here
and one of the results that I got is that students are asking a lot of questions, so, 26.6% of the time,
students asked me an explicit question on the quiz, alright, these are some examples from my computer science course here,
but, I want to stop here for a second and just mention that I never have students asking me questions unless there is an assignment due,
right? Like right before an assignment deadline and the office hours are crazy...so, that's the usual pattern,
right, you've got students asking tons of questions right before the assignment deadlines but what I have here is 3 times a week,
right, with all 3 lectures, I had students asking enormous number of questions,
and of course, the downside to this is that I can't answer all these questions, right, I've got way too many of these.
But it was encouraging to me that students were asking these questions with such frequency
and throughout the entire semester instead of just before the assignment deadlines.
So, not to dwell too much but this table just shows that students are giving me quite a bit of information from these reading quizzes.
So tons of questions; the other category right there is kind of unfortunate, the other category is 16%, was like
"Dan, is this going to be on the exam?" "Dan, I'm scared, your course is tough".
Right, all of sorts of you know things that probably aren't that useful. But the other categories are pretty well represented too, right?
Students acknowledged a lot of confusion, a lot of problems, 8.3% of them lied to me and told me that my information was interesting;
they told me that things were difficult, so I really have a lot of useful information from these...from these reading quizzes from students.
So, based on this fact, I think that it's...I've somewhat succeeded in preparing students for...for lecture.
There's many of these quizzes; they are taking them seriously, I think;
they're asking a lot of questions; they're telling me what's confusing, and so the next question,
the second study I want to discuss today, is now that we have been...
kind of prepared for a class, what do we expect from them in the lecture?
Right, so we're going to do this peer instruction now and what can we expect of students in the lecture?
What is the quality of their discussions? What are the learning gains they're exhibiting and what is the role of the instructor?
I'm going to go back to that again -- what's the role of the instructor in the peer instruction lecture section itself?
So I have one more study to go through...but I wanna restart my laptop first, so let's take like a 2 minute break here, okay.