Tip:
Highlight text to annotate it
X
So out of curiosity, I mean, you, you guys are clearly a biased sample of the class,
because you're here. And, the other people are not here, and so your behavior might
not be representative of theirs, at least with regard to this. But how many of you
guys find the webcast useful? Do you go back and look at them again? Anybody?
Really? Okay, cool. and, and it's, it's I mean. Obviously it, it, it's valuable or
you wouldn't keep doing it I suppose. So the, you, you, you're finding, it's useful
with you? It's, or is it just on those days that you don't come in that you.
Yeah? . . Okay, we were talking about we're starting to talk about sampling. We
talked about parameters and statistics. And, when I to talk a little bit about why
anybody would want a sample instead of taking a complete census of a population.
it can save time and money, you can get a very accurate answer if you do it right.
There are situations where you're basically forced to sample of for a
variety of reasons. For instance, if you're doing destructive testing. Now
you're a company, you make light bulbs. You want to know on average how long your
light bulbs last so that you can offer a warranty, right? You can't run all your
light bulbs until they run out to figure out what the average lifetime of your
light bulbs are, or you wouldn't have enough to sell, right? So destructive
testing, when the kind of evaluation you do ends up destroying the thing that your
trying to do it too. Your kind of stuck with using a sample. they're are actually
many situations where you can get a more accurate answer based on a sample than you
can trying to do a census of the whole population. And, where that can happen in
particular is if you're, you have to train people to collect the data or something
like that. Then you can have a higher level of training for a smaller number of
people that can result in collecting more accurate data. Where somehow your ability
to collect accurate data is resource limited as well. And so, you can by
allowing some statistical uncert ainty, coming from the fact that you're taking a
random sample, you can end up decreasing the uncertainty in individual measurements
because you can, you can make those measurements either using better trained
personnel, using a more expensive, more accurate instrument, making your
observation over a longer period of time something like that. So there are
situations, where you can actually get a, get a more accurate estimate based on the
sample than you can, based on the population. In order for that to work, in
order for that to be true, you have to draw your sample in a particular way. You
have to use random sampling, you have to use probability sampling. And we're going
to talk about some of the ways in which things go wrong if you don't use a
probability sample. all right. So I want to talk a little bit about the Hite Report
on Women and Love. And this discussion is largely stolen from a book by Chamont
***, came out on 1993 called, Sense and Nonsense Of Statistical Inference:
Controversy: Misuse and Subtlety. it's your Hite She did a study in the 1980s.
She sent out over a 100,000 questionnaires to ask women how they feel about their
relationships with men? And the, the findings were reported in a book called
Women and Love. she came up with results that were in. they, they contrasted very
strongly with what the previous wisdom had been. What, what, what other studies have
found. So, in particular Kinsey's Study from 1953 was the definitive work on, on
sexuality in the United States up, at that time. And the Kinsey report found that 26%
of women had extramarital affairs in contrast Hite you know, some years later,
but you know it is what 30 years later, 34 years later found that 70% of women had,
had, a married women had extramarital affairs. Okay, just a, a huge difference .
Now Hite claims that Kinsey was wrong because of how Kinsey did his surveys that
the problem was that Kinsey was using face-to-face interviews, whereas Hite was
allowing people to fill out questionnaires anonymously in some sense. A nd so, her
feeling was that people were reporting more accurately what their behaviour
actually was, whereas face-to-face they'd be embarrassed to admit what they were up
to. So Hite argued that Kinsey's figures were, were biased downwards. Hite found a
flood" of unhappiness". in the responses that she received. But, what makes this,
kind of curious is that there were other studies that were done roughly at the same
time as Hite's. There was a Redbook survey, and there was a, a, a random
sample, you know survey of 2000 people done by Patterson and Kim. And they both
found that roughly 31% of women had, had, had extramarital affairs. Okay? So 31% is
a lot, you know, is somewhat bigger than the 26% that Kinsey had found in the
Fifties, but a whole lot smaller than 70%, that you know, for, which was the, the
contemporaneous estimate by Hite. So Hite also claimed that you know this feminist
have raised outcry against the many injustices of marriage, exploitation of
women, financially, physically, sexually and emotionally. this outcry has been just
and inaccurate, okay. But again, a poll from roughly the same time, a Harris poll
of 3,000 people found that family life gives great satisfaction above men and
women, and 89% of the respondents said that their relationship was satisfying.
Okay, so again we have this tension between Hite on the one hand saying, its a
mess and other people saying, well not that big a mess. so what, what's going on?
All right, so the, all of these results are based on sample surveys, some used
face to face interviews or telephone interviews, some used questionnaires. we
talked about the difference between parameters and statistics. So the, the
percentage of women in 1991 who had, had extramarital affairs is a parameter,
right? That's some property of the population. We don't get to know the
parameter. We only get to try to estimate the parameter based on the statistic,
which is what fraction in some sample of, of, of women said, that they had, had
extramarital affairs. So we've got two steps removed, one is a sample instead of
the population, and the other is, there's a reporting issue. It's what people say
about themselves, which might or might not accurately reflect what they actually did.
Hite was very focused on the issue that what they said was likely to be more
truthful if they said it anonymously through a questionnaire than if what they
said, they said either face to face or over the telephone to somebody who was,
who was talking to them. All right, so Hite says that her study was
representative of what was going on. Largely, because the demographics of her
respondents match the demographics of the country really, really well. So, let's,
let's look at that. So, here is we've got, this is, this is looking at income and
nineteen percent of the, the, the women who woman responded to a survey where made
less than $2,000 a year. Whereas the, pop, in the US at large at that time, the
percentage was about 18.3. income between 2000 and 4012 versus thirteen. If you look
at all of these numbers, they match remarkably well. there's a, there's kind
of a gap here, I'm not sure what's going on there. I should really check to make
sure that's not a typo on my part. but, but the demographics, the income
distribution matches the US population really, really well. Similarly, if you
look at where they live. So, you look at people who were urban city, urban dwellers
60% versus 62, rural 27 versus 26, smalltown right demographics match really
well. Look at what part of the country they were from. Again Northeast, all, all
of these things match remarkably well. Look at their ethnicity and race. Again
the demographics match remarkably well in, in each of these slices, each of these
strata each of these, these slices through the culture to our society. So you know,
does that mean that Hite is right? The fact that, that the demographics of her
group match the demographics of the country. Does that mean that the reported
behavior of her group matches the behavior of the country, okay? And the answer is
not necessarily, right? the re, there other explanations and we're going to talk
about them. And the, the biggest thing that comes up in this particular survey is
non-response. so who is likely to send back a questionnaire who receives it?
Somebody who is single unhappy, has a lot of time on their hands, so forth and so
on. Or somebody who is happily married, busy with family and kids. You know, so
forth and so on, has other things to do. Right, I mean, it's just, it, it, the,
the, the people you would expect to return the survey, you would expect them to be
disproportionately people who are angry and had time on their hands, right? As,
the anger is a much stronger motivator than happiness. You, you never, you don't
see big demonstrations out on the street, hundreds of thousands of people saying
we're happy, we're happy, everything's great. Right, I mean, you, [LAUGH] people
go out to complain about what they don't like, right, about what they're unhappy
about. okay, so it would be, you know it, it, it, it certainly the case that people
might be inhibited about answering truthfully and answering face to face
versus answering anonymously in a questionaire. And that, that's some amount
of that is probably going on. But what I think the larger effect is that is going
on here is simply selection bias it's, it's who choose to response, non-response
bias, so let's talk about that So, Hite sent out a hundred thousand questionaires.
She only got forty-five hundred back. Okay, so the, the level of non-response or
the number of people who didn't return the questionnaire, divided by the number of
questionnaire's she sent out is 95,500 divided by 100,000. This many didn't come
back. So her non-response is 95.5%,. okay. Now, if she wants to claim that the
results of her survey match what things would be for the population of large. She
needs to believe that the 95.5% who didn't respond would have responded the same way
the 4.5% who responded yet, if their answers would have been similar, similar
mix of things. Now, we've given one reason that, that is unlikely to be the case.
That is, who is motivated to respond? Who has time to respond, who spends their time
that way? It's more likely to be people who are unhappy, and angry, and whatever,
than people who are satisfied. Yes? Okay, so the question is, is it unlikely for the
demographics of the sample to match the demographics of the population so well if
there was non-response bias? And for, for my eye or for my ear, what I think that
that's actually saying is that misery and unhappiness don't obey economic and
demographic boundaries. That they're spread kind of uniformly across the
country, that you're not rather, rather than, rather than saying that the opinions
of these people match the opinions of the other, the other 95.5% I think it's more
likely that, that the, just, the people who are unhappy are unhappy in all walks
of life. . So, but, but I have no data to, to confirm that, right? That's just that's
just another hypothesis that explains the same observations. Okay all right. So we
could ask, well what could have happened as, if, if these other 95 and a half
percent of people had actually responded? Well, suppose that, you know, her estimate
was that 70% of women had, had extramarital affairs, okay? What would
have happened if the other 95.5% of people had responded. Well suppose that they had
all reported that they had extramarital affairs, right? If that were the case then
the estimate that she would have gotten, based on what she did get and what she was
missing, would have been that 98.7% of women had, had extramarital affairs, okay?
Well, what if none of the women who didn't return their questionnaires, had reported
that they had extramarital affairs. Well in that case, the 70% would have been 70%
of four and a half percent would have been exponent zero percent of 90, 95 and a half
percent and the estimate would have been 3.2%,. okay. So, depending on how those
other 95 and a half percent of women might have responded, the estimated rate of
extramarital affairs could have been anything from 3.2% to 98.7%, all right?
Her estimate that it's 70% is completely riding on the assumption that those 95 and
a half percent would have responded the same way that the four one-half percent
did, who did respond. Does this makes sense everybody? Okay so, we can,
non-response introduces a bias, we don't know how big that bias is but we can put a
bound on the bias. The effect of that bias could have decrease the estimate down to
3.2 or buff the estimate up to 98.7. Is there a question, yup? if you multiply it
by 70% you get 70% out, that's essentially what she did. She pretended that everybody
who didn't respond would have responded the same way that those who did respond.
So, she got 70%. Yeah, cause if you take 70% times, times four and a half plus 70%
times 95 and a half divided by 170%. So that's, that's an essence the assumption
that she made. Okay, all right, so you get the idea that we don't really know what
the story is for the non responders. We don't really have a reason to believe that
they would have responded the same way the people who did respond, responded. If we
assume that they did, then non-response doesn't introduce a bias, but we don't
really have a justification for that. And depending on how the non-responders would
have responded, the estimate could have been washed enormously in one direction or
the other. Okay, all right. in general, when you are making a sample survey,
taking a sample survey, there are a variety of things that can introduce bias,
non-response is one of them. But there, there are a number of others. So one of
the most important is frame bias. So the frame, so typically there's a population.
You wanted to learn something about population. But the, you can't really draw
a sample from the population because you don't have like a list that has everybody
in the population on it. So you don't, you don't have a way to sort of ensure that,
that you could, that you could pick anybody you wanted. Instead excuse me.
instead what you typically have is a frame, which tries to match the pop
ulation as closely as possible, but, but won't match it exactly. So for instance,
if I were trying to well here's, here's a, here's a, here's an example. in 1936
Literary Digest took a poll of 10,000,000 people trying to predict the outcome of
the presidential election. Alf Landon was running against Roosevelt. Uh,, the
prediction based on their sample was that Landon would win by a margin of more than
two to one, okay instead Roosevelt won. and the you know, what was, what was going
on? What was, what their frame? They, they weren't able to sort of draw a sample of
people from the population as a whole. They made a list of people and drew a,
drew a sample from that list. What were their list of people? Well they were using
phone directories, so you had to have a phone number or you had to be a subscriber
to Literary Digest. Okay, well it turned out that their list of subscribe, people,
telephones were still pretty expensive back then, not everybody had one. You
tended to be wealthier than average if you had one, similar to subscribed to Liberty
Digest. You tended to be wealthy, or you tended to be Republican. Okay, so their
frame, the sort of set of units, the set of people from whom they were drawing
their sample was not representative of the population. So if your frame, if, if the
list that you're drawing your sample from doesn't represent the population as a
whole, then the sample is not going to have a very good chance of representing
the population as a whole. Okay, so that's an issue, a case of frame bias. So it
happens, this kind of thing happens a lot. You know, you know there's a list of
things from which you can draw your sample, and that list is not, does not
exactly match the population. Now, it might be that, that mismatch doesn't
matter. and it might be that, that mismatch matters a lot. The closer the
frame matches the population, the less the issue of frame bias. Okay, now If you're
talking to human beings. So that issue of frame bias comes up whether you're
talking, no matter what kind of sam pling you're talking about. It comes up for
survey sampling, where you're trying to solicit individuals opinions and so forth.
It also comes up for you know sampling which is coming off the assembly line. If
the frame doesn't match the population, you have an issue of frame bias. Question
wording comes up when you're talking about people, or talking to people. And here,
it's just very clear that you can get any answer you want by asking the question
that's worded appropriately. So imagine the difference in responses you would get
to the question, should a woman have control over her own body including the
reproductive system? And should a doctor be allowed to *** unborn children who
can't defend themselves, right? Those are two questions asking for people's opinion
about abortion, but they're going to get answers at a very different rate, right?
They frame things differently, not in this, not in this sense of frame. Is there
any questions? Does that, this makes sense for everybody? Okay, so you can, you can
really monkey around with that and people do people do to manipulate public opinion
by taking opinion polls. That say you know, this fraction of people support
abortion and if they had asked a different question they would have gotten a
different answer, even if the content was you know about abortion for the sake of
argument. Okay sensitive topics. You ask people questions about sensitive topics
you are, you might not get a truthful answer, right? I mean, as like an example
that's given in How to Lie With Statistics is you know how often do you bathe? Right?
If, if you bathe once a week, you might not answer that way. If you bathe four
times a day, you might not answer that way, right? you could be embarrassed by
the truth one way or the other. Okay similarly, if you're asking people about
their use of illegal drugs, you know did, did they pay their income taxes or cheat?
I mean, there's a lots of things where people might not answer truthfully. Now
there are techniques for trying to get around this like randomized response.
There the idea's the following you, you have the person. It, it, it's something
like this. So, question one. Toss a coin, if the coin lands heads answer whether you
ever had, ever had an extramarital affair? If the coin lands tails, answer whether
you're male or female, right? So the idea, the idea there is that the person
answering the survey knows that the person who's analysing your results, will not
know which questions they were answering. And so, they will answer they, they have
an enmity built in because of the coin toss. Okay, but if I know that I
administered the survey to a 50% men and 50% women. I can undo sort of the expected
number of responses to this, the second question answering whether your a male or
females. So, I can somehow statistically subtract out the number of yes's orno's,
that would have been contributed by the coin landing tails, right? And then, and
then use, use what's leftover to figure out what fraction of people reported if
they had extramarital affairs. So this, does makes sense? Okay, all right. So
there are ways of trying to get around to the sensitivity issue. All right,
interviewer bias interviewer's can, can biased the results in a variety of ways
if, if the interviewer has any discretion over whom to pick, whom to have in the
sample then you might pick people who were just sort of look more pure or look more
open. Don't look angry, don't look like they are going to stab you if you stop
them on the street to ask them a question. I mean, you know what, whatever it is
there's also a well-known effect that the interviewee once you stop to talk to
somebody they're very likely to, to want to please the interviewer in someway to
say kind of what they think the person wants to hear us. So, It's not even a
concious things. It's not like deliver it misstatment of the truth, but it just sort
of it, it does affect responses. in the book How to Lie With Statistics by Huff
there's a lot of great examples of this and encourage people to read that if they,
they haven 't all ready. How many of you picked up a copy of How to Lie With
Statistics? Almost nobody, it is like you know it's this thick it's like ten bucks
and it's fabulous I really, really, really recommend it okay let's see. All right, so
questions about this, before we go on. Okay we're now going to, have taxonomy of
samples designs. There's lots and lots and lots of different ways, you can draw a
samples. I'm going to talked about some of the main ways and then someways that they
can be blend it in hybrid ways of drawing sample. And you know what the, you know if
you want to try to keep your eye on a ball. Here's, here's the two big divisions
are these, is it a probability examples or is there some random element being
deliberately injected to the selection of the sample or not. Is it a non probability
sample, that's, that's a huge divide That's a probability sample there are ways
that kind of dealing with it to get error bars is out the end and figure out what
happened. If it's not a probability sample all bets are off. You just you have no
idea of what relationship the, the results from the sample bearer to what results
would have been for the whole population. Unless, the sample is so big, if they
basically gives you everybody in the population. It's almost to census, then
you can figure out what might have gone wrong? But if its far from the census if
the samples only a small fraction of population and that's sample wasn't drawn
at random, you can't just say anything or you can't say reliably. you can't quantify
the extent to which your answers like to be off. All right, so that's the big
divide probabilty sample or non probability sample. Among probability
samples, then we start thinking things like well, in this way of drawing a sample
was every unit in the population equally likely to be selected or were some units
more likely to be selected than others If some units were likely to be, more likely
to be selected than others then that raises the issue of bias and we need to
make sure that we take that in to a ccount and coming up with our estimates of other
end. Okay, so the, were the terminology here we have a population of units. Units
are the elements of the population, units could be people, units could be something
else. We typically can't draw directly from the population, we're drawing from a
frame. We don't have an exhaustive list of everybody in the population, instead the
list that we have, the sort of, the, the, the things that we're drawing from are the
sampling frame. alright. If the value of the parameter for the frame is different
from the value of the parameter for the population that's going to introduce bias
no matter what we do. . Alright, we're gonna talk about little n is going to be a
sample size, big n is gonna be the number of units in the frame from which we're
drawing the sample. Alright, so I could draw a sample of people from this room by
picking individuals, right? I could say you, you, you okay that's a way, that's a
way of drawing a sample of individual units. Okay? Alternatively, I could draw a
cluster sample, where instead of taking an individual I take a group of individuals.
I could say everybody in the fourth row you're the sample. Okay, so that's a
cluster. I am picking, a cluster is everyone sitting in the given row. Does,
does that, make sense? Okay, so. If the ultimate, thing that I'm pulling is the
unit that I'm interested in. It's not a cluster sample. If the thing that I pull,
is a collection. You know, one, one draw gives me a group of people. That's a
cluster. You'll get a cluster sample . okay stratified sampling. So a stratum,
what it means to stratify population, is to divide it into a bunch of
non-overlapping groups that are a partition of the population; in this case
a partition of the frame. So, we can think of the class is being stratified by year.
So they're, they're... We can divide everyone in the class into freshmen,
sophomores, juniors and seniors - those would be the four strata and then I could
separately draw a sample of freshmen, a sample of sophomores, a s ample of juniors
and a sample of seniors. That will be a stratified sample. That make sense? Okay,
divided to subgroups, draw sample from each sub-group, okay. Cluster sample that
sounds almost the same is I divide people into freshmen, sophomores, juniors and
seniors and then I let my sample be, okay everybody who is a freshman you're my
sample. That's a cluster sample. Okay, so get the difference. In one case, I'm
sampling from the stratum and the other case I'm treating that group as a group,
as a cluster and taking it as a whole. Okay All right. Multi-stage sampling. This
happens a lot when we're, in things like economic surveys and whatnot. It happens
as part of checking the census for coverage, for undercount. So if I wanted
to, to draw a sample of people in the US, I don't have an exhaustive list of
everybody who lives in the country. Nobody has a list like that, okay. The IRS
probably comes closest. But, you know. All right. So, how could I do that. Well, I
could do it in stages. I could start by making a list of states. Okay. Then I
could pick two states from the 50. I can make a list of the counties of those
states. That wouldn't be very hard, it's easy to make a list of the 50 states. Yes?
If I pick two states it's not that hard to make a list of the counties in those two
states. Okay. If I pull two counties from each of those states from each, in each
county, I can probably make a list of the blocks, the residential blocks in those,
in those counties, that wouldn't be that hard. Then I can select three blocks from
the counties and now if I'm just down to blocks it wouldn't be that hard to make a
list of all the houses, all the dwellings, that there are on those blocks. Right? And
then I could the so pick housing units at random from the blocks, make a list of
everybody who lived in those housing units. Okay, so you pick my address. Who
lives at my house? Make a list of them and then pick one of those people, one of the
people who lives at that house. That would be a multistage way of drawing a sampl e
of people. And the idea would be that, that at the end, you're drawing individual
people. But you've done this in such a way that you don't need a list of everybody in
the country to do it, so this is a convenience to, to make it, to make it
possible to, to do that. Now this is multi stage, if I'd taken at this last step
instead of taking one person from each of the household units, if I took everybody
who lived there that would be a multistage cluster sample, because at the last stage
I'm pulling a cluster of people. That make sense? Okay. If I made a list of states
and then in each state, I selected two counties. And, then in those. So forth.
Then I would have a stratified sample. Right? Because I'm treating the states as
strata. I'm drawing a sample separately from each of them. Okay? If, if I made a
list of all the states, and took, took two counties from each of the states, and then
at the end, took, everybody who lived in the household that, that I selected, I
would then have a multi-stage stratified cluster sample. Okay, so you cobble these
things together into all kinds of Baroque, , structures. Alright, So far, so good.
This is all fairly plain language stuff, I think Okay now, how you actually decide
which two of the fifty states to take, which counties in those states to take.
Which, which block, so forth. That matters, that matters a lot, and it really
controls the, the reliability of the sample that you get. The ability to
extrapolate from the sample to the population reliably. And your ability to
assess the uncertainty in the result that you get, though a convenient sample, , is
not a good way to do it. this is, this is kind of what happened in the Hite report.
Here it's, you, you, you take whatever's convenient. So here an example's given
here. I mean, suppose I want to know, what fraction of the Berk, of the Berkley
faculty are registered Republicans. Okay? I could just start in the faculty phone
directory and, and take the first 100 names and call them. That's a sample of
convenience. There's no reason that those first 100 should be representative of all
faculty. You know, especially if. your last name says something about your
ethnicity, and possible about your, your political affiliation then that would not
be a very good way to, to do it. I might go to the faculty club at lunch and
interview the first 100 faculty who agreed to talk to me. Okay. Well who belongs to
the faculty club? You think that that's a representative sample of faculty. I don't
know, it might be, I doubt it. But, you know, anyway, okay. So those are both
convenient samples. The problem is that you have no idea how representative your
sample is once you get it. And you have no way to sort of quantify the error that
you're likely to make in extrapolating from that sample to the whole. Alright,
quota samples are sort of saying okay I'm going to deliberately match the
demographics of my sample or some features of my sample to some features of the
population that I know. So, in the height report, she didn't deliberately set out
to. have of those samples end up matching on those demographic things. So it's not
really a quota sample but it's, it's kind of like a quota sample. So if you sort of
say okay this fraction the student body are, is, is female so I want to make sure
my sample has that fraction of women. This fraction of student body are freshman so I
want to make sure my, my sample has this fraction of freshman. This fraction of
students are on financial aid. I want to make sure that my sample has this fraction
of students on financial aid. We could match on a bunch of characteristics of the
population... The problem is that matching on those characteristics which you know
about doesn't ensure that the sample matches the population on the thing that
you don't know about, which is the reason you're carrying out the survey in the
first place, okay? It could well be that, that even you know that some parameter
differs enormously between groups that have, that have the same demographic
characteristics and so, doing quotas is not a good way to ensure the accuracy of
the result. Alright, systematic samples, here this is what a lot of people think of
when they think of sampling. So the, the image is that you have a file drawer, and
you have a thousand files in the drawer, I wanna end up with a hundred files in my
sample. So what do I do? I take, the first and the tenth. Sorry. First and the
eleventh, the twenty-first, and the thirty-first, etceterea. So I take like,
every tenth file out of the drawer. And that gives me a 100 files that are fairly
spread out among the 1,000. Okay. Seems like a reasonable thing to do. There are
situations where it works pretty well, but there are situations where it doesn't work
that well. And in particular you can't really figure out, whether you're in a
situation where it works well, or where it doesn't, unless you either know the
results for everybody, or you have a random sample to compare it to. So
systematic samples one of the advantages they have is that they don't really leave
latitude for the data collector, to kind of pick and choose individuals that seem
you know, there's less room for deliberately introducing bias, or having
little subjective things come up. But, but the order of the list, where you're taking
every eighth element of the list, will matter. And if that order has some
association with a property you're trying to learn something about, then you're
going to get bias. Okay. Probability samples are the gold standard. This is
what we're looking for. You, probability samples are drawn using some random
mechanism to deliberately introduce randomness. And the analogy is like , look
I'll give you three or four analogies. So one is, you're playing cards. You're
playing high stakes poker. Okay. There's a deck of cards on, on the desk. You would
probably not just walk up to that deck of cards and start dealing a hand of cards.
and you know, and in high-stake game, you would insist on shuffling the cards first,
right? Okay. Even though you don't know what order the cards are in. They might
have bee n shuffled, they might not have been shuffled, they might be in their
original sorted order. You know, who knows what they are. You want to deliberately
introduce randomness so that the result ends up being fair, right? What's our
notion of fair? . The cards were shuffled well before you dealt the hand, right.
Every, every five card hand was equally likely to come up, okay. So that's taking
random sample is like shuffling the cards. What's another example? . Suppose you want
to know whether a, a, a pot of soup has the right amount of salt in it. Okay? You
want to stir the soup before you taste it. Because if you don't stir the soup, there
could be a whole bunch salt sitting in a puddle on the bottom that had not gotten
mixed in well. Right? If you stir the soup well, you can tell from a teaspoon whether
it's about right, too salty or needs salt. Right? It doesn't take a lot. And for that
matter, if you stir the soup well, it could be a little pot of soup, it could be
a cauldron of soup. It could be the entire Pacific Ocean. Right? If it's stirred
well, one teaspoon will give you a very good idea of how salty it is. If it's not
stirred well. Then you know, a gallon won't tell you very much right? It's just
sort of, okay, so stirring it really matter. Same idea like making, making a
batch of cookies, chocolate chip cookies, do it, you know are there enough chocolate
chips in the cookies? Is it about right? Well if you haven't stirred the batter
really well the chips could all come together somewhere, right? If you want to
know that, that your sample the dough that you get in one cookie is representative of
the batch of dough. You need to stir stuff up. So that's what random sampling does,
it ensures that stuff is stirred up. Okay. so not only, do, do probability samples
stir stuff up, so that what you get tends to be more representative of the whole
but, If you do it right, when you're done, because we have some idea how random
sampling, how drawing tickets from a box of numbered tickets behaves, we can
quantify h ow far off the answer is likely to be from the truth. So, remember when we
were talking about a simple random sample, from a box of numbered tickets. So you
pull a bunch of tickets without replacement out of the box. What's the
expected value of the sample mean? It's the mean of the numbers in the box, right?
So, what does that mean? On average if we did it over and over again, our sample
mean would be equal to the population mean. On average. Not, I mean, it's never
gonna ma, it, it might never match it exactly. But if we average the results of
a bunch of trials, we're going to get answers that sort of oscillate or you
know, they, you know, if this is the true population mean, then sometimes our sample
mean is going to be over here, sometimes it's going to be here, right? We're gonna
be off in the typical case, but on average we're, we're going to be, we're. The
average of a bunch of rep, repetitions would give us the right answer. But
moreover, what else do we know about the, about the sample mean? We know it's
standard error. What's the standard error of the sample mean? . So it's the SD of
the box, divided by the square root of the sample size. Okay. So what does that mean?
Well it means that, a big chunk of the time. We're going, our sample mean is
going to be within a few standard errors of the true population mean. But for most
samples, most of the time that we do this, our estimate is going to be pretty close
to the truth. Occasionally, we might get an answer out there. But we know, from
inequality if nothing else, that you know, we're not going to be five standard errors
away from the expected value, very often, not more than one over five square root of
the time, not more than four percent of the time, 1/25th, right? But so, if we're
drawing our sample at random, we know that sort of on average we'll get the right
answer, and more over, that typically we can say how far we're likely to be from
the right answer. And if we wanted to get close to the right answer, we could
increase the sample size. That would decrease the standard error, and say oh,
now there's a, now there's a big chance that instead of being within this distance
of the truth, we're within some smaller distance of the truth. We can control that
through the sample size. Now, if we're taking a sample of convenience, if we're
taking a quota sample, if we're taking something like that. We have no idea
what's going on here. Our, our, our estimate could have any relationship
whatsoever to the truth. And we can't quantify what's the chance that the
estimate is gonna be within this distance of the truth or within that distance of
the or something else. Because there is no such thing as a standard error for a
sample that isn't random. Right? That you need randomness. We have to be talking
about a random variable. Okay. So, deliberately introducing randomness let's
us quantify the uncertainty when we're done. In a way that we can't do for other
sampling designs. . All right. Not every sam, not every probability sample gives
you a unbiased result. we do know that if you take a simple random sample or random
sample with replacement than the sample mean, the expected value of the sample
mean is the population mean. All right, so we know that. So the that way to say that.
Another way to say that is, that the sample mean is an unbiased estimate of the
population mean in those circumstance. So the, the bias is the difference between.
The value of the parameter you're trying to estimate, that is the population mean.
And the value of your and, and, and your estimator, in this case the sample mean.
So, on average, the difference between those is zero. If you're using a simple
random sample or a random sample with replacement. We're gonna talk about that a
lot more in the next chapter. But let's, not every probability sample is like that.
Sometimes you sample with different items with different probabilities, and there
are reasons for doing that and we talk about that later. All right, so a simple
random sample is the canonical method. What is that? We've talk ed about it
already, it's a random sample without replacement. It's drawn in such a way that
if we're taking, little n units from a population of big N units. A simple random
sample is one that's drawn in such a way, that every subset of size little n of the
n objects in the population is equally likely to be the result. So how many such
subsets are there? There are this many possible subsets of size little n drawn
from a collection of big N. Right? Okay? Simple random sample, one way of defining
it, is it's a sample drawn in such a way that every single one of those subsets is
equally likely to be the result, equally likely to be what you get. So that's like,
you got the box of number of tickets . You stir them up really well. You, you grab
five, you pull them out. That's a samp-, no, but you need more than five in the box
when we're drawing a simple random sample. But okay. Alright Okay. That's not how
they're typically drawn in practice. This talks a little bit about how people tend
to do it. Typically, the way that you draw a simple random sample is to assign a
random number to every element of the population. And then, sort of sort them
and take the little n of them that had the largest rand, random numbers associated
with them. Or that had the smallest random numbers associated with them or something
like that, . We're not gonna worry too much in this class about the difference
between random and pseudo random. Computers don't really generate random
numbers. They generate pseudo random numbers. So there's actually some
algorithm that produces them. If you know the seed, you know the whole sequence of
random numbers. They're not really random, but we're not going worry about that here.
If you're doing elaborate simulations, it can matter. Okay, systematic random
samples. So, remember we mentioned systematic samples before. It's like you
take the first file in the drawer and then the, the K plus first and then, and then
the two K plus first, like taking every tenth file, something like that. Well,
systemat ic random samples are a variant that where, instead of starting with the
first file, you start with one picked at random and then you take every tenth file
after that. Does this make sense? So you start with a random one between one and
nine and then, starting with that, you then take every tenth file. Okay , so
that, is a probability sample, right? Every unit has a known chance of ending up
in the sample, but not every subset has the same chance of being the sample, when
you're done, okay? So let, here's an example, of drawing a systematic random
sample, of ten from a population of 100 things. So you would randomly pick the
starting point between one and ten. And then if one, if, if, if the number that
comes up is one, your sample would be the first unit, the eleventh unit, the
twenty-first unit, on, and so on. If the random number that comes up is ten, then
your sample would be the tenth unit, the twentieth, et cetera, on up to the
hundredth. Okay? So if you look at this sort of list, every, every possible value
is in here once, every, every possible unit is in here exactly once and, if this
row gets pulled, unit one gets pulled. So if this, so unit one gets pulled with
probability ten%. Because there's a ten percent chance that this is the row.
There's a ten percent chance that this is the row. In this case, number eight gets
pulled, number eighteen gets pulled, etcetera. So there's a ten percent chance
that any given unit is in the population. All, all, is in the sample. Every unit has
the same chance of being in the sample. But, what's the probability that the
sample consists of the units, two, seven, 23, etcetera? Zero, right? Because if it
has two in it, it's not going to have seven This is, this one has seven This one
has two Right? You can't get them both in the same sample. So, there are only ten
possible samples in this stratified. I'm sorry in this systematic random sample.
There are ten. There are 100 choose ten possible samples if you were doing simple
random sampling. Okay, much, much, much bigger number of possible samples. You can
actually think of systematic random sampling as a random cluster sample. I've
divided things into ten clusters. One cluster is the units one, eleven, 21, 31,
etc. And I'm going to at random pick one of those ten clusters, and my sample is
going to be that. Okay, does this, this make sense? Okay, alright, so, this is
random. Everything has the same probability being a sample, but it is not
a simple random sample and you need special techniques for dealing with the
uncertainty. if you do things this way. alright. So lets talk about some of these
ideas. So suppose you want to estimate the average size of classes at Berkeley,
undergraduate classes at Berkeley this semester, and we're going to think about
seven different approaches. So, one is take a random sample of the 100 from the
course schedule, average the enrollments of those classes. Okay, what kind of
sample is that? Is it some kind of probability sample? Or sample of
convenience. We're taking a random sample. It's a probability sample. What's the
sample frame there? And the courses in the catalog, and the schedule of classes, does
the frame match the population? Sorry? For that semester? Every, every course is
listed in the course catalog, right, I mean is listed in the course schedule,
right? And there aren't any things listed that, aren't courses, right? So that the,
the frame matches the population there, there's not an issue of frame bias. That
make sense to everybody? Okay, we're taking a random sample of size 100 from
the population that we're interested in. That's just a simple random sample. That's
somehow the, an, an easy way, clean way to do it. Alright, What if you take a random
sample of 50 students who are enrolled in the current semester, and make a list of
the courses that they're each taking, and average those lists, average, average that
list? Sorry? Multi-stage, cluster. Some. Okay. So. . What's the frame there? Does
every course have a chance of being selected or not? Let's assume that every
student has the, has the same chance of being selected that this is a nice simple
random sample of students. Some classes have more students, okay, okay. So that's,
that's very, very important. let's, let's first ask though, whether, every class,
is, is even represented, right? Is, does the frame match? Does the set of courses
that we can get, include every course that's offered or not. No, yes. Okay. So
if you have a class with no students, and it's actually in the course schedule, that
zero is not gonna end up getting averaged in. Okay. But that's kind of like not
having a class at all. So, I think we, we have a little technical point here to
worry about, but let's assume that every course, that if, that if the course
doesn't have, has an enrollment of zero it gets struck from the list. Okay. All
right. So now we get to the issue of, does every class have the same chance of being
included, and that's where your comment matters. Take a class that has. 700
students enrolled in it, and take a class that has, a similar class that has five
students enrolled in it. If we think of, a box of tickets. Where, the, the tickets
are the students in this case, right? A student is representing classes, yes? Then
how many tickets go in this box were the class that had 700 students in it? 700.
How many tickets go in the box with a class that had five students in it? Five,
okay so, we're much more likely to pull the larger class then the smaller class if
we do our sampling this way. What's that going to tend to do to the sample mean
that we get if we start averaging the class sizes on this list. Is it going to
be an unbiased estimate of the average class size or a biased estimate of the
average class size? Bias. Okay is it going to be, tend to be too big or tend to be
too small? Too big, right? Okay, we're going to get, get the big classes
represented more often. So if you think about the difference between asking. Do I,
do I ask this later? Yeah, okay. So yeah the difference between, if, if you ask 50
instructors how big the classes are that they're teaching, versus asking 50
students how big the classes are that they're taking. You're going to get a
wildly different answer, right? Assuming that every class has one instructor. You
know, some have more than one. But if you put instructors in the box, you've got one
ticket for each class. If you have students in the box, you have a lot of
students for the bigger classes, and very few, very few tickets for the smaller
classes. Does that make sense? That's going, going to buy us the results. .
Okay, So this is a, a random, cluster sample. Because well if we pick a student
we pick all the classes that student's taking, right. So what we're picking is a
cluster of classes. Does that make sense? Okay. And it's going, to be tend to be
bias as an estimate of the average class size because we're going to attend more
students representing the larger classes. Right here. you take a random sample of
fifteen structures. Neglecting courses that are team taught by more then one
instructor. You have one ticket in the box per class. Right? So this is really a,
this is still a cluster sample of courses, because some instructors teach more then
one course. Right? But, it's going to be an un, give you an unbiased estimate of,
of the size of the classes. That make sense? Okay, alright. separate random
samples of five courses from each department average the sizes of the
courses in those samples. What kind of sample is that? We're starting by dividing
the course catalog into, departments and we're sampling separately from the
departments, so each department represents stratum, right? We've stratified, okay
we're ignoring cross listed courses for the moment but okay so we've, we've
divided the courses into strata, and now in each stratum, we're drawing a simple
random sample. Okay, is the average that I get, is every course equally likely to be
in the sample? Head shakes? Okay, so what's, what goes wrong? Why, why isn't.
Yeah. Okay, so some. Okay so, so some departments could have larger classes but
why does that introduce. Why. The, the question right now is, is every cl ***
equally likely to be in the sample. And that. You're, you're onto something but
that's not, that's not quite there. Another. Yeah? Okay. Some departments
offer more courses, okay, yep. Okay. So the, there's two ideas are a kind of, that
they're related but they're not quite the same. So what she said. If you're, if you
have a class in a department that offers a lot of classes, right? You have a class,
you have a department that has 30 courses and you take five of those. Each of the
courses offered by that department is less likely to be in the sample than if you
have a department that only offers five courses. Because every single one of those
five is going to be in the sample. Does that make sense? So if a course that's
offered in a department that, doesn't offer very many courses has a bigger
chance of being selected than a course in a department that offers a lot of courses
because you're only taking five of however many there are, okay. Now if it happens
that the, the, so there, so there, there isn't an equal probability of selection,
right? Courses don't have the same probability of being selected. Now, if it
happens that as you said, er as you said that. The department that offers a lot of
lower division courses, offers a lot of courses, then you're going to get a
downward bias in the estimate because you're going to take, you're going to take
too few of the big courses, because you're only taking five of them. Where as in
another department that offers, you know, only seminar courses, you're going to get
all five of those nice small seminar courses. Okay. So that's going to
introduce bias into the estimate. The reason that you get the bias in the
estimate is that not every course is equally likely to be picked. Does that
make sense? Okay. Alright, yeah. , no. It's, it's a problem if So I Let-, let's
look at a department that has ten, ten classes. Okay? So here's one that has ten
and here's one that has 50. Okay. What, here-, here's, here's, you know, blah 101.
Okay? That's one of the classes in this dep artment. And, and here's, Quh you
know, 101. Okay? This is this department. What's the chance that this, class ends up
in the sample. Okay. It's five out of ten It's a 50, 50 chance, right. What's the
chance that this ends up in the sample? one out of ten, right? You're taking five
of these 50 in this department. So this class only has a five out of 50 chance of
ending up in the sample, a one in ten chance. But here you're taking five out of
ten so this class has a 50% chance of ending up in the sample. Okay so, if you,
you, different, the probability that the course is selected, depends on which
department it's in, if the course, if the departments have different number of
classes. Is that, that make sense? Okay. Alright, and then if, if this course that
offers, if this department that offers 50 courses, has a disproportionate number of
large classes, right? They offer a lot of large lower-division classes, then, your
estimate based on the sample mean is going to be biased downwards, because you aren't
getting a proportionate number of those classes. So to get rid of this bias, what
you would need to do is, you, you do something like decide on a sampling
fraction, F, which says, I'm going take ten percent of the classes in each
department. Okay. So then, every class would have the same chance of being
selected. Okay. If I take. So, so for this class here, I'm going take one class,
you're going take five et cetera. Then I'd have the same probability of selecting
every class. Of course, that's not going to work out perfectly, because, not every
department has a multiple of ten classes offered, so but that's the idea. So if
you're having, if you have a stratified sample and your sampling fraction varies
across strata, so you're not taking the same fraction of items from each stratum,
then your probabilities are going to vary, your sampling probabilities will vary
across strata and not every item has the same chance of ending up in the sample.
Typically when that happens you end up with bias in the estimator. Okay. What is
going on in number five? T ake a random sample of five science and engineering
departments. A random sample of five humanities professional departments. For
each of the departments that we get, we list all the classes, and we average those
sizes. So, first of all, what kind of sample is this? So we're starting by
dividing the university or the courses into, into two kinds right, engineering
versus other stuff. Right, so that's two strata yes. Within the stratum we're then
taking a random sample of five departments and what are we doing with that? We're
going to take every course in each of those five departments. So if we take a
department, we've in effect taken a cluster of courses. We've taken all the,
all the courses that department offers. So this is a stratified cluster sample.
Right? Two strata, draw five clusters from each stratum. Alright. Are all the courses
equally likely to be listed? Let's Suppose that, all right. So but, the answer is no,
but, let's figure out how we could fix it so that it would be true. Suppose that
there were ten science and engineering departments and ten humanities and
professional departments. Then I will be taking. five from each, right? The chance
that I pick any given department would be the same, right? 50% chance that I take,
that I get any particular science department, 50% chance I get any
particular humanities department, right? If I do get that department, I get all the
courses taught in that department. That make sense? So the, the, the chance that a
course gets selected is the chance that its department gets selected. Because if
the department gets selected, the course does. Okay? The chance that the department
gets selected would be the same for every department. If there were an equal number
of departments like this and departments like that. Alright. So the reason that
this doesn't give you the same probability of selecting every course is that you
don't have the same probability of selecting every department. You would, if
there happened to be the same number of science and engineering departments as
there are h umanities and professional departments. Without knowing that that's
true. If that isn't true then this doesn't give equal probability of selecting every
course. Is that work for everybody? Alright Okay. This one. Random sample, of
five departments of one kind, five departments of another kind. Then a random
sample of instructors, and then you average the sizes of the courses those
instructors are teaching. Now what do we have? We still have Strata. No. Now.
What's the next thing? Okay. Ultimately, we're pulling instructors. Instructors
could teach one or more classes. Right? So, what we're pulling at the bottom is a
cluster. So, we have strata, and we have clusters, but we have something else in
between. Its a second stage right, so this is a stratified multi stage cluster
sample. Yep. you're taking a sample of size five but it's not a cluster of five
courses. It's, it's from, it's, it's not like I've divided the, courses that the
department teaches. So if the, let's say the department teaches 30 classes and I,
I, I could do something where I have okay the first five the second five the third
five on up to the sixth five right? And if my strategy was, that I you know I'm going
to roll the di and if the di lands one I take the first five. If the di lands two I
take the second five. That would be a cluster sample but that's not what's going
on here. Your taking a, a simple random sample of five classes from that
department if you get the department. Okay. I'm sorry that's from every
department. So, there it's, what you are ultimately sampling is five individual
classes form the department not one a number of prespecified sets of five, okay.
okay. Alright. So this is a, stratified, multistage, cluster sample. The extra
stage is, having picked a department, you then pick instructors from the department.
And then, the instructors that you get, you just get all the courses that, that,
that their, that their teaching. . pick a random number from one to ten starting
with that class and the course schedule you average of every tenth you take every
ten th class you average the sizes. So this is just a cononicle, random,
systematic random sample. It's systematic because you are taking every case item.
It's random because you're starting at a random place. Is every class equally
likely to be, in the sample? Okay. Yeah, assuming it's a multiple attempt that, in
all, every class equally likely, but not every one of the number of classes choose,
whatever is equally likely, right, because some subsets are impossible. Alright,
Okay. In, the examples we've been talking about so far there, there are sort of
really is a population. There is a, a, a population of, courses and they, they have
certain number of students enrolled and the, you know it makes sense to talk about
the average number of students enrolled in those courses or , there's a population of
women, and some of them have had extra marital affairs and there really is a
number that have had extra marital affairs and we're trying to estimate that. There
are a number of situations where it's useful to think about drawing samples
from, populations that don't actually exist. and we'll talk about a couple of
examples of that. So first one is suppose I want know what the, the average test
score would be for some group of students and I'm going to do that by taking a
random sample of students and administering the test to them. Okay. Use
that to estimate, what am I estimating? Well I'm not estimating the test scores of
all the students, because not all the students have test scores. Because not all
the students took the test. Right? I'm thinking about a hypothetical population
of what would the test scores have been had all the students taken the test, but
in fact they did not. Okay. Something about drawing a random sample from a
hypothetical population that doesn't exist because I didn't do the work. Right? Okay.
Now that's, that's one direction. something that, that, that happens even
more often is when you start thinking about the effect of a treatment. When
you're trying to look at, you know, I, I, I give some people a drug. I want to know
what's the effect of this drug. And typically what I want to know is the
difference between, how many people would have gotten better if everybody had gotten
the drug? How many people would have gotten better if nobody had gotten the
drug, okay? The example I've got written down here is like SAT training SAT
coaching, right? You co, coach some students. You want to, you want to show
that SAT coaching works, right? I work for an SAT coaching company. I want to show
that I have this great service and that I'm going to improve your SAT scores by 75
points, okay? So, how do I do that. Well, I got to have, I, I would start with some
collection of students. I would take a random sample of them. I would coach some
of them, and I would not coach the others and I would compare the average test score
when we're done, okay. That would be a reasonable thing to do. Okay, how do I
know what the effect was on any particular student? I've, I've no way of knowing,
right because I, I can't simultaneously coach you and not coach you. So I can't
compare your score with coaching to your score without coaching. Yes? Somehow what
I'm thinking about is what would the average score have been had everybody been
coached? And what would the average score have been had nobody been coached? And I'm
trying to figure out what the difference is between those two averages. But that's
the difference between the average of two lists, neither of which exist. Because any
given student either was or wasn't coached. You don't get a list of numbers
that has that students test score if they were coached and that students test score
if they were not coached. You don't get both right? Okay. So there's a
hypothetical counter factual. Yep. You can. The problem with taking the test,
coaching, taking it again is that there's actually some learning associated with
taking the test. You have more practice taking that kind of test. You're, you're,
you're also three months older. Your other, it's just hard to, hard to tease it
apart. So. Okay, so this notion of hypothetical counterfactual sampling from
hypothetical populations is really useful. one of the ways people think about this,
which we'll talk about a little later when we get into the analysis of experiments
is, that every student is represented by a ticket. And that ticket has two numbers on
it. One number is, what would that student's score be if the student were
coached? The other is, what would that's student's score be, if that student
weren't coached? Then we can think about, well, is there any difference between
those two numbers? And one way to formulate things is, if coaching makes no
difference whatsoever. Then, for each student. The first number and the second
number are equal. My number might not be the same as your number, but my left
number and my right number are the same, your left number and your right number are
the same, right. So if, if coaching makes no difference whatsoever, that's a good
model of what's going on. Okay. questions about this. Do this example. Okay so
here's a city, it has 800 blocks. Blocks have an average of 50 dwellings units.
Dwelling units have an average of three people. But, not all blocks have the same
number of dwelling units, not all dwelling units have the same number of people.
every block has at least ten dwelling units. There's no homeless people. It must
be Southern California multistage cluster sample of city residence is drawn as
follows. You pick 80 blocks at random for each block. You pick ten dwellings at
random and the people on those dwellings comprise the sample, okay. So that's, you
got a random number of people when you're done and one question is does every
resident have the same chance of being in the sample? How do you end up in the
sample? Okay. So if your dwelling gets picked you end up in the sample. For your
dwelling to get picked. You're block has to get picked, okay. every block has the
same chance at being drawn. Every dwelling in a given block has the same chance of
being drawn. But does every dwelling have the same chance of being drawn? Okay, if
you're in a block that has a lot of dwellings, the dwellings in that block are
less likely to get selected, the sampling fraction is smaller. Right? Okay, so the
same thing we were talking about for departments over there. If every block had
the same number of dwelling units, then every person would have the same chance of
being in the sample, because you get sampled if your dwelling gets sampled.
Right? Okay, but because the number of dwelling units isn't the same in every
block. The chan, the individuals chance of being in the sample isn't the same. It
doesn't matter that there are different number of people in each dwelling. Right?
Because you get selected if your dwelling gets selected. So if the dwellings have
the same chance of being selected, the people would have the same chance of
getting selected. But they don't. Okay. So, what kind of sample is this?
Multi-stage cluster? Yeah. It actually probably says that, doesn't it? Yeah.
Okay. Wow. I'm, I'm consistent every once in a while. How about that? Okay. Alright.
So if you took the average number of people per dwelling in the sample, is
that. It is the expected value of that average, equal to the average number of
people in the dwelling? Overall. Not, not necessarily. Right? Where would you have a
problem in particular? Suppose that. Sir. Okay, so it, it could happen that You have
blocks that have sort of, a small number of dwelling units with, with a small
number of people in them. Okay, that would be like single family dwelling versus
tenements or something like that right? If you have high density and a large number
of people per dwelling combined, then the estimate is going to tend to be biased,
downwards, right. Because you have a smaller chance of selecting those dwelling
units where the dwelling units have more people in them. Okay? In the other
direction, if you're out in some suburban area, where, where there's, you know,
families of four, five, six people, and only ten houses per block, versus 150, you
know, dwellings in an apartment complex. Okay. Then it's going. Okay. So. You get
the idea that, that could introduce bias. Other, question. Yes. it's not stratified
if you like divided the city into quadrants. Like, north, south, east, west.
And then, you said, from this quadrant I'm going to draw 30 blocks from this
quadrant. But instead you're drawing blocks from the city as a whole. So it's,
it's multi-staged. Because first you draw a block, and then you draw a dwelling. But
it's not, it's not stratified. Because you're not sort of dividing the city into
groups of blocks, and then drawing separately from each of them. Okay?
Alright, I'm going to take, just like, two minutes to tell you a little bit more
about the, the election auditing I was doing yesterday. . So, The, the, the
method for drawing the sample, is actually a, a weighted random sample, where the
probability. So, what I did was, was, there were two experiments. One was,
looking at measure P and Davis, which, It, it, there were 31 precincts that voted in,
in the contest and I, I counted by hand, the number of votes that were cast in six
precincts drawn at random from those, from those 31. Actually there's, the 31 doubled
into, 62 because there were votes cast by mail and votes, votes cast in person. But,
the goal of this, this auditing is to figure out whether. There was enough error
in the counting of the votes to cause an outcome to change. So if you're doing
something like that, it makes sense to focus your attention where the errors
might be biggest. So instead of drawing precincts with the same probability, I
drew precincts that could hold more error with larger probability than precincts
that could hold less error. This is something that comes up in financial
auditing a lot. it, suppose that I've got. There's, there's a ledger with a bunch of.
How many of you have taken some accounting classes or, okay some, some of you at
least. Alright, so. You got a ledger, you have a bunch of line items in a ledger,
they have a book value. Alright? That's wh at it's reported to be. You know, a-, a-,
according to the book. And there's an issue of, well, was that overs, let's
suppose that this is a list of assets. Okay? There's a, an, an issue that you
might want to audit for is to determine whether the, the actual value of that set
of assets is very different from the listed value of that set of assets, the
book value of that set of assets. Have the book value been overstated? Alright? Well,
if you're trying to figure out how much stuff has been overstated, it makes sense
to look at the biggest items preferentially because they could have the
biggest overstatements. An item that is listed for a dollar could of been
overstated by at most a dollar. An item that is listed for a million dollars might
of been overstated a million dollars. And so if you want to figure out if there is a
lot of overstatement at all you want to pay a lot more attention to the million
dollar items than the one dollar items. So you're going to take, take a probability
sample but instead of taking them with equal chance of selecting every item you
take them in a way that it's a million times more likely to select a million
dollar item than to select a one dollar item. And it lets you end up doing
something much, much, much more efficient. You know, that's, that's the strategy that
we use in the election audit. And the difference was between looking at
something like, six precincts and probably looking at something like 30. In order to
be able to conclude that if there were no errors, there wasn't enough in the sample,
then it was very, that would be very unlikely to happen if there were enough
error in the aggregate than the whole population of votes to have changed the
outcome. the, the other, so that was one thing, the other pilot was to pull
individual ballots at random and look at the machine interpretations of those
ballots and make sure that a human interpretations of the ballot matched the
machine interpretations of the ballot and that let us audit another contest of Davis
which was a vote for two of, it wasn't Davis it was in Yellow County vote for two
or four candidates. and you know the margin between the, the, the second place
or the third place which is the margin between the, the, the last winner and the
first loser was only sixteen votes, so it was a relatively narrow contest but only
spanned one, one precinct. Then again the idea is, you're trying to make, draw, to
come to a conclusion that if the outcome were wrong, you'll be very likely to see
more error than you in fact saw in the sample. So if you see very little error in
the sample, you can conclude with high confidence, you know, small P value, well
get to all this stuff later. That the outcome of the contest is correct. Okay.
I'll let you guys go. have a good weekend.