Tip:
Highlight text to annotate it
X
There's a problem with the "F" ratio and that is the bigger your "N", the larger your sample,
the bigger your "F" obtained, in general, all else being equal. If you have larger and
larger sample, the "F" obtained will also get larger and larger. Now in a way you're
thinking wait, don't we want a large "F" obtained, yes, that's a good thing because we do what
our "F" obtained to fall over here where we can claim an effect. Here's the difficulty
that arises with language and interpretation. There is language I want you to avoid and
language I want you to use when discussing a "P" value.
The problem that happens is the computer often does the analysis for you and it says you
have a result with a "P" of, instead of saying less than .05 if you have chosen .05 as your
criterion, it will give you a more specific level. Even if the "P" were .0018 for example
your "F" obtained would be in the right region and you get to claim an effect. The problem
that happens when computers spit out a specific "P" value is that people think that this means
that I have a large result or a large difference between groups but that is wrong. It doesn't
necessarily mean that. When you think of the "P" value I want you
to only think of it as the level of doubt, your chance of being wrong in claiming an
effect. The "P" value doesn't necessarily say anything about the meaningfulness or the
size of the effect. For example, studying bodybuilders, you give them some treatment
and you measure the girth of a muscle for example. So you could come up with a highly
large "F" obtained, way out here, only through having a huge sample, even though the change
in the size of the muscle is so miniscule it could be meaningless. The computer could
spit out a "P" value that is way less than .05 so you'll like it but you do not get to
say the following words. I want you to avoid the following phrase, "this "P" value is a
highly significant result". Never say that and here is why. The difference for example
in the bodybuilder's muscle might be a millimeter so small that no one would notice. If you
have a big enough sample size you can wind up with a huge "F" obtained, which is good,
but that does not mean you can interpret the difference that you found as really being
large and meaningful. When you use the words, "this is a highly significant result", based
just on the "P" value you are implying that the result means there is a meaningful difference
between your groups, that the treatment made a meaningful difference or that it is a large
difference. But we don't know that only looking at a "P" value. All we know from a "P" value
is the chance we are wrong to claim that the treatment changes something.
So how do you talk about a "P" value? Simply say "P" equals .05 or "P" is less than .0018
and if anybody asks you, yeah but what does "P" mean? Say it's the probability that I
am making a Type 1 error when I find an effect. The probability of a false positive. The probability
that I am wrong when I say I found something. That's all that "P" is.