Tip:
Highlight text to annotate it
X
>> From the Library of Congress in Washington, DC.
[ Silence ]
>> Thank you so much for joining us.
This is the first even online conference for the Library of Congress and for educators.
We have a fabulous program lined up for you right now.
During this session, you'll learn how to use free online assessments designed
by the Stanford History Education Group that incorporate documents
from the Library of Congress's archives.
Participants will examine assessments, rubrics, and sample students' responses.
Leading us through this session is Joel Breakstone.
Joel co-directs the Teaching With Primary Sources Program at Stanford University.
And directs the Stanford History Education Group.
Along with Mark Smith and Sam Wineberg, he led the development
of Stanford History Education Group's assessment website, Beyond the Bubble.
He received his PhD from Stanford Graduate School of Education.
And before coming to Stanford, he taught high school history in Vermont.
So I'm going to turn things over to Joel.
>> Thanks so much, Kathy.
I really appreciate the introduction.
I'm really pleased to be with all of you this evening.
And I want to also just reiterate my thanks to the Library
of Congress for organizing this conference.
It's a really excellent opportunity for teachers,
and we're really excited to be able to participate in it.
My colleague, Sam Wineberg, just finished up a session that perhaps some
of you had a chance to hear him speak as well.
Over the course of the next 40 minutes or so, I hope to give you an introduction
to the materials that we've created and have posted online on beyondthebubble.stanford.edu.
And to not only provide introduction to the materials, but also to provide a bit of a window
into the process by which we designed them and what we hope they are useful for.
But before I get to that, I want to just say a little bit about how I ended up doing this work
and being at the Stanford History Education Group.
I was a -- as Kathy mentioned -- a high school history teacher in Vermont.
But I came to that position after being a history major in college.
And I graduated from college and I didn't have a job.
And I ended up getting offered a position in Vermont, even though I had only spent a day
in Vermont before getting the job.
I moved there from Boston -- I had grown up in Boston.
And I took over for a gentleman who had been teaching for a number of years.
And the main things that I inherited were a classroom set of the American Pageant.
And I knew that although the textbook would be a great starting place for my students,
it wasn't the only thing that I wanted my students to read in their history class.
And so I wanted them to read primary sources.
That is what I read as a history major.
And so I gave my students a document.
One of the first ones I gave them was this great speech by William Lloyd Garrison
in which he describes the hypocrisy of celebrating the fourth of July in independence
when there are thousands and thousands of people enslaved in the United States.
And I found quickly that giving a student a document that is 850 words long
and in 12-point font at 7:37 in the morning is a bad idea.
And it was that terrible feeling that I had too often as a young teacher
where the lesson's going poorly and you're trying to figure out how you can salvage it
so that it goes a little better when you have to teach it three more times the rest of the day.
And so I figured out some things over the course of my time teaching in Vermont.
But I also had a lot of questions about how I could do the work better.
And how you could provide more materials for teachers.
Because it's simply unreasonable to expect that each teacher is going to be able
to create a variety of document-based materials on their own.
It's an incredibly time-consuming process to track down materials and to build lessons.
Certainly you can do some, but it's really difficult to create an entire curriculum.
And so that's what led me ultimately to pack up my life's belongings into a Penske truck
and drive across I-80 to California, where I began a doctoral program at Stanford working
with folks who had similar interests at the Stanford History Education Group.
Around, how do you create materials that are available
for teachers to use with their students?
And make them in an accessible fashion so that you don't have the looks
that my students gave me that first morning at 7:37
where they simply had no interest in reading.
And in many cases the materials were simply inaccessible for them.
And when I arrived at Stanford, we had just released the Reading
Like a Historian curriculum online.
And we had a lot of interest in it.
There was a lot of excitement about the materials.
But there was also simultaneously a lot of talk about assessment.
And it came from the highest levels of government.
Barack Obama, in March of 2009 said, "I'm calling on our nation's governors
and state education chiefs to develop standards and assessments
that don't simply measure whether students can fill in a bubble on a test,
but whether they possess 21st Century skills like problem solving and critical thinking."
And we agreed.
We said, yes, absolutely.
We would love to see those types of assessments.
But where are they?
That's an excellent goal, but if you're a teacher,
where do you get those types of materials?
And simultaneously we were releasing, as I mentioned,
the Reading Like a Historian curriculum.
This is our old website.
And when we first released the curriculum, we got a lot of emails
from folks who began to use the materials.
And the most common query was, where are the World History materials?
Which was a fair critique.
We've since adjusted and have begun to make World History materials.
But the second most common response that we heard
from people was, where are the assessments?
If you guys are going to make all these materials based
around teaching historical inquiry and having students engage in disciplinary reading
and thinking, how do I know whether or not my students can do it?
How do I know whether or not they can consider source information?
Where did the document come from?
What was the context in which it was created?
How does the context in which it was created shape the content of the document?
These historical things need scales of sourcing, contextualization, and corroboration.
Which our curriculum was based on.
And then at the same time the Common Core standards were released.
And they called for students to do things like analyze primary and secondary sources.
Consider authors' perspectives.
Corroborate sources.
And develop written historical arguments.
And we said, you know, these are some interesting skills.
But how are we going to assess whether or not our students can do that?
These various fairly complex ways of thinking and reading, where are the assessments
that would help students to engage in -- to give teachers feedback about whether
or not their students can engage in these ways of thinking and reading.
And so we continue to ask the question, where are the assessments?
But we didn't have a good answer for where to find them.
And when you look around, you can find an enormous number
of materials that talk about assessments.
And particularly issues of formative assessment.
How do you create short tasks?
If you go on Amazon you can find hundreds and hundreds of resources
that you can buy about formative assessment.
But there are not many for history teachers, for guidance for how do you create short tasks
that give us quick feedback that we can use to quite literally form our next steps in teaching.
There wasn't much guidance.
And moreover, there weren't many resources in the world of formative assessment.
So we weren't sure what these types of assessments could look like.
But we knew where we could find the materials to make those assessments.
We had just joined the Library of Congress's Teaching With Primary Sources Consortium.
And as a result, we suddenly had access to the world's greatest library.
And all the resources that the Library's archives offer.
And so we knew we wanted to construct some short tasks
that would provide teachers meaningful feedback about these ways of historical reading.
But we still weren't sure exactly what form they would take.
And so we wanted to be clear first about what are the resources
that are immediately available to history teachers?
And we see them as existing on a continuum of complexity.
At one end of this spectrum are multiple choice questions,
which are quite familiar to us as teachers.
But also as citizens of the United States.
The United States likes multiple choice questions more than most places.
In the U.K. They call multiple choice questions "the American form."
So not everywhere has the same fascination with multiple choice questions.
And so before we -- we knew that multiple choice questions were available,
but we wanted to also be very clear about,
what is it that a multiple choice question requires our students to do?
And it was an important first step in the design process.
We wanted to be very explicit about the skills and knowledge that are required for students
to answer a multiple choice question.
And so here's an example drawn from the California State Test
for tenth graders, World History.
And what I'd like you to do is to take a moment, read the question, and just think about all
of the things that a student would need to know and be able to do
in order to answer this question.
So when we give a student this multiple choice question,
what is it that we're really assessing?
And so as you think about that, take a moment and just type in the chat box some of the skills
that are required in order to answer this question.
We'll take a minute or two to do that.
And then we'll go on as people begin to fill in some of the things
that are required in order to answer the question.
Think of sort of the most basic thing.
What's the first thing students need to do?
And then what are some of the knowledge that they need to know?
[ Silence ]
Great. So looking through this list, you know, I see a lot of --
there are a lot of things that are required in order to answer this.
You know, one of the very basic things is it requires reading comprehension.
Students need to be able to read and decode the question.
So somebody wrote down academic vocabulary, right?
Students need to know what an outcome is.
And then beyond that, there are some crucial vocabulary words.
Like "colonial possessions."
You need to know what a Communist is.
"Dominant military powers."
"International superpowers."
So there's some just reading comprehension.
There's also content knowledge, right?
Issues of, how does World War II end?
Right? Students need to know something in order to answer the question.
And also how to read a test, right?
That there's also a basic move of test taking.
Of eliminating, somebody said, the wrong answers, right?
It's that, you know, by the time a student's in tenth grade,
they've taken a lot of multiple choice questions.
They have a pretty good idea of how to answer the question.
And so that's about doing a process of elimination.
And so, you know, when we think about multiple choice questions, we see them as providing,
you know, immediate feedback about students' reasoning.
But it's not necessarily the way to measure all of the things
that we want our students to know and be able to do.
Because it's also measuring some things that are not particularly historical.
Like process of elimination.
That's not a move of a historian.
That's a test-taking move.
Similarly, it doesn't give us a lot of feedback about students' more complex ways of thinking.
But it's quick and we can do it fairly efficiently and get some immediate feedback.
But it may not be the best tool for measuring all of the things
that we want our students to know and be able to do.
At the other end of the spectrum are DBQ's -- a much more complex, rich task.
One that has gained increasing popularity as the Advanced Placement program has grown.
And the State of New York has used them in their Regents Exams.
And other groups have provided materials around DBQ's.
And it's a much richer task, one that requires students to engage in a variety of skills.
And again, what we did before we moved on to develop the assessments was,
we wanted to be really clear about, what do some
of these most common types of tasks require students to do?
So here's an example; this is pulled from the 2010 AP U.S. History exam.
And it was, "In what ways did ideas and values held by Puritans influence the political,
economic, and social development of the New England colonies from 1630 through the 1660's?"
Students then receive a series of primary sources.
In this case they received ten documents.
They have then an hour to read through the documents and formulate an analytic essay,
and to compose that analytic essay.
And so again, I'd like us to take a moment and do the same thing that we did
when we were initially surveying the field.
And take a moment and list in the chat box, what are the things that students have to do
from start to finish in order to answer a DBQ?
So that we can have some clarity about, what are the skills
that we're really assessing with this type of question?
[ Silence ]
Yes, as this chat box indicates, it's a lot of things, right?
I think that's indicative of the complexity of the task.
Students have to, at a basic level, be able to have some strong reading comprehension.
They need to first unpack this slide.
This is not a small thing.
You know, you need to understand this question.
What are the different pieces of the question that need to be addressed in a response?
And then they need to engage with the documents.
And as people's comments have noted, there are a variety of things the students have to do.
They have to skim, they have to look for context clues.
They need to look for evidence, right?
They need to figure out what is the argument that they're going to be making,
and then seek it out in the documents.
And as several people noted, it's also about a process of multi-tasking.
You need to be reading at the same time as thinking how the documents connect
to the broader argument that they're seeking to make.
And then they need to read a map.
They need to understand archaic English.
So they're -- and the ultimately, after they've done all this reading, and analyzing,
and organizing, they need to write.
And write pretty furiously.
In the process, hope their wrist doesn't cramp up, they can mop some sweat off their brow,
and maybe do a quick proofread before they're done in about an hour,
within the framework of the traditional AP DBQ.
It's a rich task.
And it is testing the types of things we want our students ultimately to be able to do.
To analyze a series of primary documents, and to write an analytic argument.
That's what we want our students to be able to do.
However, it may be a little too challenging for some students.
And moreover, its complexity can make it a little difficult
to discern what a student is doing well at,
and the things that students might be struggling with.
If a student doesn't do well on a DBQ,
is it because that they don't have the reading ability?
Or is it perhaps because they struggle to orchestrate these various skills all at once?
Or perhaps they don't do well with a timed task?
So as a result, if I'm a teacher and I receive the DBQ back, it can be a little more difficult
to figure out, what are my next steps to help my student do better?
In addition, though, is the issue of time.
If I'm a teacher and I assign a DBQ, I then have to do what?
I have to grade it.
And that's really time consuming.
And unless you want to spend all of your waking hours reading student writing, it's not --
it's simply unrealistic to expect that we can use DBQ's for frequent classroom assessment.
They're a great task, but they're not ideally suited
for quick feedback about student understanding.
And so as we thought about the spectrum of assessments that were available
to history/social studies teachers, we were struck by this gap
between the more basic multiple choice questions and the complex DBQ's.
And we hope to be able to provide something
that could inhabit this middle ground between the two.
And so in a world of acronyms, we decided to make our own to fill the middle space,
what we call "history assessments of thinkings, or HATs."
And so rather than talk a lot about what HATs are, I'd like to show you one.
To give you an example of one of the tasks that we created in an effort
to create short document-based assessments
that give us some quick feedback about specific skills.
This is an assessment that we created
to target very specifically the historical thinking skill of sourcing.
That is, can students pay attention to where a document came from?
Who made it?
When? Why?
And for what purpose?
And specifically with this task, we were really interested in whether
or not students will notice the date of the document.
So this assessment gives students this image from the Library of Congress's archives.
The title of the document is, "First Thanksgiving, 1921."
The document is from 1932, made by a guy named J.L.G. Ferris.
And the question is, "This painting helps historians understand the relationship
between the Wampanoag and the Pilgrim settlers in 1621.
Do you agree or disagree?
Briefly support your answer."
What I'd like you to do is take a minute or two and just think about,
what do you think a common student mistake would be?
Not what do you think the idea answer is?
But rather, what do you think a common mistake would be if you were to give this to students?
What might they answer that might trip them up?
So think about that and then jot down,
what do you think some common student mistakes with this task would be?
And write those in the comment section.
[ Silence ]
So people have really hit on the issues of what the most common mistakes are.
Without a doubt.
And we piloted this task now with thousands of students.
And the types of mistakes that you are identifying are spot on.
And so I'd like to show you a few of the responses.
This response is by far the most common one that we see.
The student wrote, "True."
True as in, it would be a useful source.
"Because you can see how they're interacting with each other.
Without any picture, you couldn't really see how Wampanoag Indians
and the Puritans interacted with each other."
So this student is essentially saying, yes, it's a great source.
Of course it would be useful.
If you didn't show me this picture, how could I know anything about the First Thanksgiving?
Essentially, a picture's worth a thousand words.
Which is, as several people noted in the comments, it seems like more
than anything the student is matching the image of Thanksgiving
to their own beliefs about Thanksgiving.
This matches what I think Thanksgiving looked like, and as a result it's a useful source.
The second most common type of response looked more like this one.
The student in this case disagreed.
And they said that it would not be a useful source for historians.
The student wrote, "As soon as the settlers arrived, there was mass curiosity
which turned into violence and hatred.
There was never such a "party" between the two peoples.
They couldn't even understand each other."
And so at first, you know, the student said something good, right?
They're trying to bring some background knowledge to bear on the source.
And they're trying to have a critical examination of the document.
But there are also some errors here.
You know, the students say that there never such a party.
But we have documentary evidence that there was a meal held between the two groups.
We can certainly argue about the nature of the meal, but there's certainly evidence
that the two groups understood each other and even could understand each other
with Squanto having some exposure to English and being able to speak.
But regardless, the bigger issue is that in both the cases,
the students have totally ignored the date of the document.
And have not addressed the problems that are associated with using a document
that was created 311 years after the event, as a source of information about that event.
And to acknowledge the problems that result from such a gap in time between the date
of production and the event that it seeks to depict.
And so as a teacher, from those quick responses I get immediate feedback about whether
or not my students can attend to one aspect of historical thinking, which is sourcing.
At this point, though, some people say great,
why don't you just make this into a multiple choice question?
And get rid of all this writing?
It's a lot easier to score multiple choice questions than it is
to have even short constructed responses.
But what we think is that there is an enormous amount
that we can learn from short student responses.
So for instance, this student disagreed and said that it would not be a good source.
So if it was a multiple choice question, this student would have gotten it right
because they're saying no, it's not a useful source.
But their reasoning is interesting.
The student wrote, "Painted by a white person from Europe,
of course they will be extremely biased in order to bring to light a highly positive image
of the Europeans as to excite more foreign expeditions.
Because the Indians look sooo friendly and thankful."
So, although they might have gotten it right in terms of arguing
that the document would not be a useful one, their reasoning is clearly pretty suspect.
First, they assume that J.L.G. Ferris must be white and must be from Europe.
But moreover, they seem to have ignored the date as well.
Because they seem to think that in 1932 the Europeans were still trying
to excite more foreign expeditions to the new world.
Deeply problematic.
But as a teacher, there's something good going on here as well.
The student is acknowledging that the author of a document -- the artist in this case --
influences the content of the document.
That's what we want students to do.
We want them to think about the way in which the author carefully shapes the content
of a document.
So if this is my student, I can say, "I'm really glad that you're paying attention to the way
in which the author influences the document.
But we also really need to pay attention to the date of a document.
And that's going to be a crucial piece of information to think about as well."
We also in some cases see really polished responses.
This student wrote -- and it was an eighth grader, as a matter of fact --
wrote, "This painting was drawn 311 years after the actual event happened.
There's no evidence of historical accuracy,
as we do not know if the artist did research before painting this,
or if he just drew what is a stereotypical Pilgrim and Indian painting."
Really impressive for an eighth grader.
I did not have a lot of tenth graders when I was teaching who could formulate this response.
And the student has nicer handwriting than I do.
But what is striking is, the student has both identified the danger of using a document
so far removed from the event that it depicts.
But they've also acknowledged that if we knew more about what the artist did in advance
of creating the document, it might change the way that we think about it.
The document -- this answer also shows something that we've seen over
and over again with these responses.
Which is, there isn't a clear developmental trajectory in terms of student thinking.
In other words, it's not as though seniors can engage,
only high school seniors can engage in this type of thinking.
In fact, as this eighth grader demonstrates,
many students can engage in this type of thinking.
Perhaps not surprisingly, what we found is it's the students who've been taught to engage
in historical thinking who do it.
And it's the students who have not who struggle.
So for instance, we've given this task to high school seniors who have gotten fives
on the AP U.S. History exam, and they've missed the date of the Thanksgiving painting.
And I don't mean that in any way as a knock against those students.
I'm very confident that with a few minutes
of instruction they would see the importance of the date.
They just haven't been taught to look carefully at the dates of documents.
That, especially in the past, has not been heavily emphasized in AP testing.
In contrast, we've seen fifth graders who have come up with really strong responses.
Not quite as polished, and the handwriting's not quite as good as this one.
But we've seen fifth graders do it because they've been
in classrooms where this has been emphasized.
However, none of these tasks are based on only a single document.
Instead, they are -- we intend them to be flexible
so that we can make multiple versions of them.
This is the exact same assessment.
But instead of a Thanksgiving painting, we have an engraving
of the signing of the Mayflower Compact.
And again, there's a big gap in time between the date
of the document and the event that it depicts.
And so if I'm a teacher I could use the Thanksgiving assessment
at the beginning of the school year.
And then give it back to students and give them some feedback
about what strong responses look like.
And then I could give them another one a few weeks later, like this one, to see whether
or not they've really internalized those ideas and they have been able
to track the crucial nature of considering the source when evaluating documents.
So when we think about HATs, we have made them to be short,
so that they can be done quickly in five or ten minutes.
And they target specific content and skills.
So in this case it's very much about sourcing,
and one aspect of sourcing, the date of the document.
So that we can get quick feedback about students' ability with them.
Similarly, it gives evidence of student thinking.
So as teachers we get some immediate information that we can use to shape our next steps
and it doesn't take us hours to get it.
We can quickly even flip through, even if we don't give students detailed written feedback,
we can get a temperature for our classroom.
Which will shape our next steps as teachers.
And hopefully they're more accessible, right?
That if some of our students are not yet ready for the really great,
rich challenged of a document-based question,
that we can have them complete these shorter tasks in the building block towards some
of those more complex types of assessments.
Often at this point after looking at that assessment, some people will say,
so do you guys just care about skills and you don't care about content at all?
And they couldn't be further from the truth.
When we constructed our domain of what we were seeking to assess, historical thinking,
we place historical knowledge as one of the three main pillars of historical thinking.
Students need to know content in order to engage with historical documents.
And then there's evaluation of evidence.
Students need to consider the source and place it in context.
And corroborate it, to consider multiple documents,
and to look for points of similarity and difference.
In order to build a historical argument.
And so other assessments that we've created address historical content very directly.
This is an assessment that's about students' knowledge of the Civil Rights Movement.
And it would be an assessment you'd give
after you had taught students about the Civil Rights Movement.
And it draws on two letters from the archives of the NAACP.
And the first letter, both written to the executive director of NAACP.
The first letter says, "Before I received your letter today, I had been in to the President.
And he said the difficulty is that it is unconstitutional, apparently,
for the federal government to step into the lynching situation.
The President feels that lynching is a question of education in the states.
I'm deeply troubled about the whole situation, as it seems to be a terrible thing
to stand by and let it continue."
Students then read a second document, also from the NAACP about conditions
at a previously all-white school.
The letter reads, "Conditions are yet pretty rough in the school for the children.
The treatment of the children had been getting steadily worse for the last two weeks
in the form of kicking, spitting, and general abuse.
As a result of our visit, stronger measures are being taken against the white students
who are guilty of committing these offenses.
The President of the United States was very much concerned about the crisis."
And instead of asking students a question like, what was the year of Brown v. Board?
Or when did lynching reach its peak in the United States?
We asked a very basic question, which is, which letter came first?
We didn't give them the dates of the documents in this case.
Instead we wanted them to use their knowledge of the Civil Rights Movement in order
to place these documents in context.
Which one came first?
And what from the document provided them the necessary information to answer the question?
So if students don't know about the Civil Rights Movement, they might answer it this way.
Student wrote, "Letter B was likely written first,
because letter B is addressing the unsegregation of schools,
and letter A is about the response to the issue -- lynching."
And there's a certain cause-and-effect logic going on here.
But it's simply historically incorrect that following the integration of schools,
although there's some violence, there is not widespread lynching that happens as a result.
The student does not have a grasp of the Civil Rights Movement and the way in which,
over the course of the 20th Century, the federal government became increasingly involved
in acting on the behalf of African-Americans.
And intervening in the Civil Rights Movement.
Moreover, that the lynching was much more a problem in the first part of the 20th Century
when it reached a sort of critical mass.
In contrast, the integration of schools happened later.
And so as a teacher we can get some quick feedback about,
do my students have the broad narrative of the Civil Rights Movement?
Do they understand the broad contours of the Movement rather than,
do they know about specific names or dates?
Although those are important, this is a different type of task
that assesses their broader knowledge of the movement as a whole.
As with the Thanksgiving assessment, we can make parallels that focus
on other important moments and events in history.
This one is about immigration.
And so this shows an excerpt from a newspaper article that says, "It's my firm belief
that an exclusion act even more stringent than the present one should be passed
in the matter of the Japanese immigrant.
The Chinese are dangerous enough,
but the Japanese would drive all competition out of business.
It is the stern duty of the American citizen, particularly of those
of us upon this western coast, to scrutinize this even
and then suppress it with appropriate legislation."
And the second document says, "There had been 4700 emigrants landed on the previous Thursday,
and we learned on Monday that 10, 035 had arrived on the Saturday and Sunday,
principally Germans, each possessed of a small capital, and all ready to proceed instantly
to the various German settlements in the interior without staying
to waste their means in New York.
The Custom-house officer's duty was a mere form."
So again, the students have an understanding of when did exclusion acts take place.
And in what place and time did different waves of immigration occur in the United States.
All of these assessments are available on beyondthebubble.stanford.edu.
And there is a very short video that you can watch right now at this link
that gives you a little taste of why we've constructed materials the way we have.
And then after you watch that, we'll take a very quick tour through the website.
And then we will take some questions.
>> Imagine if real-life decisions were like a multiple choice exam.
>> First question, "To surprise the British troops in Trenton, should I A),
dress my troops up as a marching band and offer to play a free concert?
B), Hide my troops in a giant horse and present it as a victory gift to the British?
C), Cross the icy Delaware in a rickety rowboat?
D) Stay in my tent because it's cold outside, and let someone else deal with them?
>> Seems ridiculous, right?
Essentially, that's what multiple choice tests do.
They penalize kids for not knowing facts that are ripped out of context.
On the opposite end of the spectrum are document-based questions requiring students
to analyze and write a college-level essay based on ten documents.
But what if your students haven't learned to analyze one document, let alone ten?
As an educator, do you have any other options?
Now you do.
Beyond the Bubble was developed by the Stanford History Education Group.
And unlocks the vast digital archive of the Library of Congress
to create a new generation of history assessments.
Our easy to use assessments capture students' knowledge in action.
Rather than their recall of disconnected facts.
We call our exercises History Assessments of Thinking, or HATs.
HATs ask students to engage in historical reasoning
as they critically examine primary sources.
Most HATs take just a few minutes to complete.
And they're easy to score.
They also come with interactive scoring rubrics, samples of student work, and Going Deeper videos
that extend your understanding of these assessments.
Your students' short written responses to these HATs will give you access
to what they are really thinking.
Of course, there's always the alternative.
>> Are you sure we weren't supposed to dress up as a marching band?
>> No, I picked A last time; it has to be C.
>> I'd have preferred D; I'm freezing.
I think I got some ice in my knickers.
>> Everyone be quiet.
I picked C, and that's the answer.
[ Water noises ]
>> So all the materials are on beyondthebubble@stanford.edu.
On the top of the home page, it says Assessments.
That's where you'll find all of the materials.
And so you'll see they are organized by assessment type.
So for instance, you'll see the First Thanksgiving here.
And then below that there are the parallel versions of it.
So there's the signing of the Mayflower Compact.
There's a brief description of the assessment.
There are words that are underlined in red; if you hover your cursor
over that you'll get a brief definition of what that word means.
There's a link to the Common Core standard that goes along
with those materials highlighted for the History/Social Studies.
It certain addresses lots of ELA ones as well.
On the right-hand side of the page once you go into a particular assessment,
you'll see that there is a link to the PDF of the document.
So if you want to hand it out to students --
I will note that you need to be logged into the website in order to download it.
Logging into the website is completely free; you just need to make an account on the website.
Or if you've already made an account on sheg.stanford.edu, it will work there as well.
You can see everything without being logged in,
but we require people to log in to download materials.
Just so we can track who's using the resources.
And then there's a link to the document in the Library of Congress's archive.
So you can see the original document as well.
Then going on the right-hand side of the page,
you'll see that there is a rubric for each of the tasks.
And so the rubric is a basis three-level rubric.
Proficient: Student has gotten it dead on.
Emergent is, student's done something good but it's not all the way there.
And then Basic is, the student's missed the boat;
they haven't really understood the question just yet.
And each of the rubric levels is linked to student work.
And so all of the same student responses that we showed before are on the website.
And there are these little blue thumb tacks on the answers.
And if you roll over them, they're just prefanatations [assumed]
about what we see going on with the assessment.
And that is, an example of the responses that we've chosen to include are examples
of the common responses we've seen in piloting these with hundreds and thousands of students.
So as a teacher it will give you some really quick feedback about the kinds
of responses you might expect to see.
You can also show it to students to demonstrate what exemplary responses might look like.
and then finally, each type of assessment -- assessment types, so Thanksgiving for instance,
has a short streaming video that goes with it that talks
about how you might use these materials in your class
or how you might create your own that you can watch.
And then finally, actually, there's also a comment section so you can --
if you have some thoughts about the task,
or you'd like to see what other teachers have said, you can go there as well.
But I also want to point out that we certainly don't have any monopoly
on the creation of assessment materials.
And in fact we hope the teachers will create their own materials.
We see them as templates for other people to create materials.
And the Library of Congress offers an incredible set of resources in order to do so.
And so some of our favorite Library of Congress resources are available
on the main page of the Library, loc.gov.
A really great, really a wealth of materials are available
on the teachers' page, loc.gov/teachers.
Great primary source sets.
Blog quotes as to how you might use these materials.
The ability to search by standard or by state, all available there.
So although loc.gov is incredible, it can also sometimes feel a little overwhelming
because there's so much.
And loc.gov/teachers has really great curated sets of materials.
And then chroniclingamerica.loc.gov, I think, is one of the coolest websites around.
And somebody who spent far too much time in the basements of libraries going through microfilm,
the chroniclingamerica has thousands and thousands of newspapers totally digitized.
And you can search by keyword, by date, by location.
And you get the full pdf of the pages of the newspapers,
even after they've been fully searchable by key word.
So if you want to know how the sinking of the Titanic was covered in Missoula, Montana,
you can go look for that and then compare it to how it was covered in Washington, DC.
It's a really fabulous resource.
And then also see all of the ads that go alongside the newspaper stories.
And then finally, World Digital Library is an incredible resource for world history teachers.
And it is a project that the Library of Congress has undertaken in partnership
with cultural institutions around the world.
And it offers world history materials and documents
from various museums and libraries around the globe.
And you can search by time period, you can search by geographic location as well.
And they continue to add materials all the time;
it's a really great resource that continues to grow.
And so finally I'll just leave you with a few thoughts
about how you might create your own assessments.
You know, the first thing we always do is to choose what we're trying to measure.
Then make a rough draft.
And then take a moment and do what we did when we looked
at those multiple choice questions and DBQ's.
What is this actually measuring?
Do the analysis before you even give it to students.
What are we actually measuring?
If I'm trying to make an assessment about sourcing and my student has
to write a long paragraph, then maybe it's more assessment of writing than sourcing.
And pilot it; try it out with your students and see how it goes.
And then revise it.
We've never made an assessment that works perfectly the first time we do it;
we need to adjust it after we've done it.
And then tweak it as you go.
And see how it works.
And if you make something great, please share it with us.
We love to hear from teachers who are using our materials.
And that's everything from, I'm wondering how to get logged onto your website,
to I made this cool assessment; what do you think?
Or, I'm trying to use these materials in my classroom.
Please do reach out to us; we really do want to be in touch with teachers and support you
in any way we can as you're using our materials or you're navigating the Library's archives.
And I would just point out the last two items
on this list are the SHEG's twitter handle and the Facebook page.
That we're adding new materials all the time to our website.
And we don't send out emails to our email list when we do that.
But we do post on twitter and Facebook when new resources go up.
And then I would also just point out, as Kathy put it in the chat section just now is
that the Library also offers an incredible set of professional development resources
on the teacher's page, and she put the link there.
So that's a great resource too.
So thank you so much for spending some time with me
on a school night; I know that's not easy to do.
And I look forward to some questions now.
>> Oh, Joel, we have lots of questions.
Thank you so much for the wonderful presentation.
So I'm just going to dive right into them so we can get in as many as we can.
Jodi -- and I'm not sure I'm going to pronounce her name correctly,
Montebiano [phonetic] asked the question, "How do you correct errors in student thinking?"
And then Chrissy Jacobs chimed in, "You can't really correct a student's thinking.
You can help them construct knowledge by giving them opportunities
to notice inconsistencies, etcetera."
And then Amber, who was in the last session with Sam, noted that that was basically her question.
She's new to her district and it's been a struggle.
So how can you address the errors when looking at the students' thinking?
>> Really, really good questions.
You know, and I think our hope is that these types of tasks provide opportunities to surface.
What are some of the misconceptions or errors in student thinking?
So that we can address them as a whole class.
You know, we work with a teacher in San Francisco Unified School District.
And she gave the Thanksgiving task on the first day of school.
Because she knows that almost none of her students will get it right.
She doesn't even grade the task; she just has everybody do it.
And she knows that probably 90% of the students will say it's a great resource.
And then instead of even spending the time to mark up all of them, she goes over,
"What does a strong response look like?"
So that it's an opportunity to build student understanding of,
it's really important to look at the date.
And in this class we're going to care about where documents came from.
And so I think that's one way.
And then the other is, you know, of the idea of formative assessment is to be able
to provide students with a goal of what we're trying to do.
Ultimately this is what we want to be able to do in terms
of our thinking, we want to source documents.
And that the tasks are a way for us to identify the gap
between where we ultimately want our students to be and where they currently are.
And it's a way for students to see where they currently are,
and have a sense of where they can go next.
And so I think there are ways in which we can do some with individual feedback.
But also given, you know, the constraints of if you have 150 kids in your classes,
you can't write even short responses on all of these types of tasks all the time.
So figuring out other ways to provide constructive, meaningful feedback that's based
on sort of the overall place your class is in.
And then if there are a few students who are, you know, way above or way below,
that you can work with them individually but have tried to figure
out how we can help move the whole class forward in general.
>> That's great.
Thank you.
Rita had sent a question, but I think you've already answered it,
saying, "Where can we find the HATs?"
And we provided the link to Beyond the Bubble.
And you explained some of the functionality of that site.
So I thank you for that.
Chas Chastain asks a question, "Does the way the opening statement
in the DBQ is worded influence students to believe the document is more credible?"
>> Yes, you know, I think we always are trying to be really cognizant of the way
in which the wording of tasks may influence students.
And we try certainly quite hard not to do that.
And one way that we do that is, we do think-aloud interviews with students
where we sit down and ask them to think out loud as they complete our tasks.
And sometimes we find that, you know, the students read the tasks
in a very different way than we intended it.
And then we tweak it as a result.
So, you know, our hope is certainly that we don't want
to be influencing students one way or the other.
But it's an ongoing process.
So it's possible, but our hope is they're not unduly being influenced.
>> Okay, I have a question.
I'm going to make it a two-fer.
But they come from two different folks.
Veronica Zermino asks, "Will Stanford move into creating something like this for --
I mean Beyond the Bubble -- for other subjects other than Social Studies or History?"
And Crissy Jacobs asked a question, "What kind of issues do you see come
up in a pilot that would need to be tweaked?"
I don't know if they belong in the same bucket, but they're not far from each other.
>> Yes, those are really two excellent questions.
We are currently working on creating tasks
that assess how students interpret information online.
That, for instance, are students able to recognize
that a new story is sponsored content, that it's essentially an ad?
Or will they ask, where did the Facebook post that makes a claim
about that Denali means Black Power in Kenyan, for instance?
Will they ask whether or not they trust that Facebook post?
So we're trying to create some short tasks that move us into the realm
of assessing students' ability to interpret information online.
So that's a new effort that's beyond just History.
>> That's great.
And I think we have one last question.
I'm going to leave it with Jody's question, "Is it effective to teach by direct instruction?
Or do we need to veer away from this?"
>> I mean, I think you could do some of both.
You know, I think that it's quite possible to do some direct instruction, you know,
we're going to give this task out and we will all answer the question and then I'm going
to provide some instruction about how would I answer this question?
What does an ideal response look like?
You know, that too often our expert thinking as teachers remains invisible to our students.
And so more opportunities for us to make our thinking visible, of thinking aloud
and doing some cognitive modeling of what strong responses would look like.
Or how do we interrogate historical sources certainly can be useful.
That's not an argument to say that it should all be direct instruction.
We also want our students to wrestle with these documents, ultimately.
And that any cognitive modeling should be followed up with opportunities
for students to practice on their own.
And to begin to develop their ability to engage in this type of historical thinking,
and reasoning and writing independently.
So I think it's going to be a mix of both.
>> Alright, thank you, Joel.
And I see more questions coming in.
And they're excellent, but we are past our time.
So I am going to invite you to make sure that you get Joel's contact information there.
It's breakstone@stanford.edu.
His twitter feed is @joelbreakstone.
You can contact him through the SHEG website, sheg.stanford.edu.
The url beyondthebubble is there as well.
And the Stanford History Education Group has a page on Facebook.
I will make sure to get Joel the chat from today.
And so he will have your questions.
A couple of items before we break for the evening.
Please give us your feedback.
We have a very short three-question survey.
It helps us program for the future.
We will send you an email within the week about how
to get a certificate for completing this session.
And you'll receive an email within five business days.
And for those who asked the question about, where can I get some information about how
to get students to analyze them, please go visit us at loc.gov/teachers.
We have tons of information to help you in getting students to analyze primary sources.
Joel, thank you so much for this great presentation.
This will be something that I know will be treasured in the recordings.
I know that our folks got a lot out of this tonight.
So thank you all for staying with us at the end of a long day.
We do appreciate your time.
Joel, thank you very much.
>> You're very welcome.
Thanks so much.
And thanks to all of you who came out and participated.
It was a pleasure.
>> Terrific.
I'm going to end the recording, and I'm going to end the meeting.
Thank you very much.
>> Goodnight.
[ Silence ]
>> This has been a presentation of the Library of Congress.
Visit us at loc.gov.