Tip:
Highlight text to annotate it
X
Some of you will recall the story a few months ago that cell phones had been linked to a
type of brain cancer. It made the rounds for several days, was a popular topic of news
blogs and discussion forums. But why hasn't there been any followup? Well, actually there
has been, but only in research circles, and it's not too surprisingly reversed the previous
discovery. It appears that the prior evidence was overstated. Cell phones may yet be demonstrated
to increase the risk of certain cancers, but the key point is that no research has reliably
demonstrated this.
This is so typical and expected, at least expected by me, that it can serve as a useful
lesson in skepticism of media reporting of science stories. It can also serve as an example
of the difference between evidence based medicine and science based medicine.
When the initial story hit, the coverage was very focused on authority figures and agency
statements, which is typical of this kind of reporting. For more discussion of this
phenomenon, I refer you to my video version of Ben Goldacre's essay on Bad Science journalism.
The World Health Organization put cell phones on the list of possible carcinogens, alongside
such things as dry cleaning and artificial sweeteners. The level of evidence was ranked
very low, which to a scientist puts it on a sort of global "to-do" list of promising
research leads, but it was hardly a definitive identification of risk. That takes verification
and a different type of analysis.
Let's go over the types of studies that scientists use to establish correlations. My goal is
that the next time you're presented with these kinds of shocking discoveries, you'll be able
to determine what kind of evidence it's based on.
The first two are a type of epidemiology, retrospective and prospective studies.
1. The retrospective cohort study In vastly simplified terms, grab 1000 people
with disease A and compare them to 1000 people with no disease A. Identify factors that may
be correlated with a higher risk for disease. The biggest problem with a retrospective study
is what is called self-selection bias. Your experimental, "sick" group are already diagnosed
with a disease, and because they already have the disease, whatever is the risk factor has
already happened. It's very easy to misidentify confounding factors. If you look at the 1000
people in the cancer group and find that 721 of them are heavy cell phone users, while
only 614 of the non-cancer group are, how strong is that evidence, considering all the
other possible reasons for the differences?
2. The prospective cohort study Take 1000 people, and follow them for a defined
period of time. Document disease outcomes (did the people get brain cancers?), try to
identify which risk factors were most predictive of the outcome. This is the retrospective
study, but done in advance of the condition being studied. Did the highest quartile of
meat-eaters get more gastric cancer, did the lowest exercisers develop more diabetes? The
populations are often heterogenous (or complex), and what looks like a simple association can
often be a linked series of categories, like finding people who use cell phones to be at
higher risk of neural-derived cancer, when the actual association may be with something
about their income or profession or other environmental factor or genetic factor. It
could be a multifactorial linkage where stress and urbanization plus better medical care
is responsible for the increase, and cell phone use is really just a proxy.
3. Case-control trials In a case-control RCT, or randomized control
trial, the researcher gets to control a lot more about the comparisons being made. You
can take humans or non-human animals and mix them up until the populations are evenly randomized,
then expose half the group to the experimental factor... that eliminates the self-selection
and the confounding bias, because the researchers are intervening artificially. This type of
study is not always an option for ethical reasons. We choose not to expose humans to
excessive risk if we can help it. This is a big improvement over purely epidemiological
studies, but it also has some weaknesses. We still have the multifactorial problem,
where multiple behaviors can be linked, and it's very difficult to untangle all the variables
of genetics, environment, nutrition, case history, and sheer luck. There could still
be a crazy mixture of lots of risks. Which is why we need...
4. Mechanistic research This is the kind of work that I've mostly
done. The scientist, guided by less precise observational or epidemiological data, and
complementing the experimentally rigorous case-control data, look for ways that the
experimental factor could cause the correlated effect. This is usually done in some format
that allows for almost complete control. Identical, clonal cells grown in identical flasks, transgenic
mice, or sibling animals. Even within this category there are shades of resolution. Molecular
biology is usually pretty good at sensitive and reproducible results. Very small margins
of error, and results that can be reproduced by different labs or different techniques.
We focus on something called method concordance, which is where I take my username. Arriving
at the same conclusion by multiple independent lines of inquiry... In the case of cell phones,
we'd need to identify a plausible and testable way that the radiation or heat or emotional
cues associated with cell phones can interefere with normal cellular development. We'd attempt
to replicate that finding in the simplified model of genetically identical animals or
cell culture. If the mechanistic outcomes contradict the prospective study, then the
smart money is on the more rigorous lab research, less on the epidemiology.
Notice how we've gone from the very complex and poorly controlled to the very tightly
controlled and rigorous design. That doesn't mean the mechanistic research is better than
a prospective study. After all, it would take a very long time to test all the possible
hypotheses, and prospective studies provide great clues about what's worth further investigation.
The problem arises when epidemiological data is presented as confirmed fact, rather than
the first step in a long process. I also often see a sort of reverse: mechanistic data on
some new drug that shows that it can cure cancer or other disease in a flask of cells,
or some poor mouse with a gutful of injected cancer cells, is touted as a major breakthrough
in cancer research before it goes out to a control trial and then on to prospective studies.
None of these studies is without merit, and they add up to a total picture of the real
underlying causality.
How do we differentiate between Evidence based medicine and Science Based Medicine?
Evidence based medicine is any medical practice which is based on a preponderance of evidence
for its efficacy and safety. It's a pretty broad umbrella, though, and it includes things
that we don't understand well, or possibly applications of practices that seem to work,
but shouldn't. There's nothing inherently wrong with EBM, but it's not on the same footing
as science based medicine.
Science based medicine is a subclass of EBM. It includes only those practices that we can
understand to a finer degree. We know that they're effective, but we also understand
WHY they're effective. The mechanisms are clear enough to us that we can understand
how best to use them in the clinic, when they might not be effective, and what improvements
can be made to improve efficacy and safety. Key to distinguishing the two is that we have
more than just evidence that something causes an improvement, we can evaluate that epidemiological
data in light of a plausible mechanism.
For example, laetrile, a popular alternative cancer treatment derived from the cyanide-like
compounds found in peach pits, is well characterized in terms of chemical structure and bioactivity.
It possesses no properties that would make it a good cancer drug. We can predict that
with some confidence. However, it has a long and storied history with quack medicine of
the mid-1920's, and so when evaluated in randomized control trials, sometimes it show evidence
of effectiveness, and sometimes it doesn't. It depends on the power of the test and who's
running it. Laetrile might be considered evidence based medicine, albeit very, very weak evidence.
It is not, however science based medicine, because there is absolutely nothing about
how it acts in the body that would make it effective. Cases like laetrile, chelation
therapy for heart disease or autism, or vitamin supplements are sometimes indicated by evidence,
but not by a deep evidence of mechanisms.
So, returning to the example of cell phones, a big challenge is the lack of mechanism.
The amount of radiation given off by modern cell phones is too small to reliably damage
DNA or cross-link proteins. There's never been a good, well controlled study on cells
that was able to replicate the effect suggested by prospective or retrospective epidemiological
studies. This makes scientists a little hesitant to jump behind a single study of a type that
is notorious for false associations and lacks rigorous experimental control.
Now, there is no process to show that something does NOT cause cancer, that's simply an untestable
hypothesis in practical terms. However, how does a proper skeptic view the outcomes of
these contradictory studies? Simple. With an open mind, and a rational estimate of risks.
We don't have to cast our vote for guilty or not guilty before the trial is finished.
Like good jurors, we can wait until all the evidence is in. That's what rational skepticism
is all about: withholding judgment until sufficient evidence is presented to accept or reject
a given hypothesis. We understand this when we're in the jury box, but forget it the moment
we step out of the courtroom.
It's more than a coincidence that so many scientists are skeptics and rationalists.
The process becomes a way of life, a way of thinking, and a way of viewing the world.
Turning those gears off on the weekends is just something some of us can't do. We care
about what's true, and we know a reliable process for determining this.
Thanks for watching,