Tip:
Highlight text to annotate it
X
we recently wrote an article in the journal of economic methodology about
something we think may be of wider interest
Think of it as a bug report for economics
a possible explanation
why economists who do technically brilliant work that captures systematic
regularities in the data
all the same can - repeatedly - reach conclusions about the real world that are
unfounded
and just plain wrong
the main idea is that when you evaluate research you need to focus on the
specific claims that are made
and assess how well each of them is justified
that doesn't sound particularly interesting but let's take an example
Here is an academic article
such an article will typically make a lot of different claims and they will be of very
different types
some have to deal with the formal properties of model
is the mathematics correct?
Are the concepts fully worked out and consistently applied?
is the model consistent?
Is it elegant or messy?
other claims have to do with a model`s ability to predict. How well does your
model fit the data? How well does it predict out of sample? Is it consistent
with stylized facts?
and then there are claims about the reasons why things happen
what factors cause X to happen in the real world?
How can we empirically identify,
establish and quantify their causal impact?
what's mechanism involved? what's the evidence that the mechanism that you
have just been describing
is actually out there in the real world?
and is that mechanism enough to make X happen?
What are alternative explanations of why x happens?
Do these other explanations supplement or counteract your mechanism?
or maybe they`re just plain wrong
and finally there are claims about the welfare
how will policies A, B or C affect the welfare of different groups?
and what you mean by welfare?
Is it income, wealth,
happiness, health,
something else?
is is something we can measure?
how well you actually able to predict changes in this measure of welfare?
and how do you separate preferences in the sense of
"what makes you better off"
from preferences in the sense of "what you choose"?
this is important because specific claims
made
require different types of evidence
depending on the type of claims that we are looking at
say that you've written an article that contains the sophisticated elegant and
innovative formal model
and you claim that this is a major step forward in pure theory
to examine this claim we read through your article to see if you`ve provided or
cited the evidence necessary to support that claim
and you have
however at the end of the article you discuss policy implications of your work
you claim that your model shows that some policy which is being discussed on
the news
would actually cause harm to consumers
the evidence required to support this claim however is missing. There's no
evidence that the mechanism in your model exists or is important in the real world.
there's no evidence that your model captures welfare in any measurable, credible sense
of the term and so on
so an article may be really good in supporting an important claim X while
making other claims Y not backed up by evidence or good arguments
if an article that provides good evidence for an important claim is a good article,
then a good article can still contain other claims that are wrong, perhaps even absurd
and that are presented without even an attempt to provide relevant evidence
this may be obvious in principle but we don't think it`s obvious in practice
we took the literature on rational addiction which is a growing field in
economics
regularly published in top journals
it's the standard approach to understanding addictive consumption in
economics
and all these four types of claims are made in the literature
in terms of formal theory these models were able to generate unstable
consumption
from rational, forward-looking choice
made from stable preferences
this claim is pretty well supported
the resulting models are also able to reproduce some stylized facts of addictive
consumption, especially if you use aggregate data
in our view the evidence that these models are a good way to capture
regularities and patterns market data
is there, but it's mixed
turning to causal claims, however
the claim that real world junkies, smokers and binge eaters
are just implementing the forward-looking plan
that rationally takes into account how today's cigarettes
needle, milkshake will alter tomorrow's health and tastes
that seems highly questionable and very likely wrong
there`s both evidence and arguments speaking against the claim
that beginning smokers smoke more and more
because they maximize discounted lifetime welfare given accurate
information about the effects of tobacco
and that they have found that to seemingly get hooked
is the optimal way to reconfigure their tastes
into that of a chain-smoking adults
similarly
we would argue that these researchers
have so far failed to provide evidence for the claim that addicts are best off
being addicts
you might say that you could care less about rational addiction theory but we
believe it tells us something important
about the economics discipline
think about the food industry and how it turns cattle
into hamburgers
if we have reason to think people are getting ill from hamburgers, the food
industry will test the food product for the bacteria E. coli
not because all strains of e_ coli a highly dangerous but because they're
relatively easy to find
and if e_ coli is present then this indicates
fecal contamination of the food
and with fecal contamination comes a host of nasty things that are more difficult
time-consuming and costly to test for directly
e_-coli bacteria thus functions as an indicator bacteria
it tells us that the control processes we are using are not good enough
in the same way you can think about research
researchers write articles that are fed into the discipline of economics
and out comes peer-reviewed, accepted claims in established journals
if you have reason to think that the claims coming through this process are
systematically flawed
we may look for absurd claims
not because these are necessarily super important in and of themselves
but because they're relatively easy to spot if they are there
if absurd claims such as those of the rational addiction field are found
this indicates, well, at least flawed control processes in economics
they tell us that we are currently unable to weed out even strikingly wrong
claims from even top journals
If such claims make it through
than there will likely be a host of other causal and welfare claims that
are less obviously absurd
but equally poorly supported
and these may be taken far more seriously than they deserve and may
have
quite negative real-world consequences
our hypothesis is that this is caused by a blind spot in the culture of economics
Even if economics start out
making all errors equally, their errors may be spotted and caught more often by their peers
when they consern formal errors in models or incompetent dramatic fitting and
prediction exercises, than when they concern claims about the real world
and welfare effects
they may also pick up a feeling
that uh... claims of causal and welfare insight require far less scrutiny
for instance there are incantations, that in some seminars settings function
like magic spells that can be used to deflect criticism
against your claims of causal and welfare insight
"These are standard assumptions"
"This is just as-if theory"
"The realism of the assumptions is irrelevant"
are examples
over time, this allows such claims to piggy-back on elegant, formally
consistent models that fit stylized facts
we freely admit that this is just a hunch
However, we tried to find a way to examine the hunch, or hypothesis if you will, by
sending a survey to all published researchers in the rational
addiction field
The answers, while no hard conclusive evidence, seem indicative of, or at least
consistent with, such a blind spot
when asked whether the literature on rational addiction theory supported a
specific claim about pure theory
seventy four percent agreed that this was the case
at other points in the survey, we asked whether the theories should be consistent
with various kinds of evidence
and whether they actually were consistent with such evidence
If we bring in the answers to these questions
thirty-one percent both supported the pure theory claim
agreed to the need for the relevant evidence in principle
and believed that it was there in practice
this means that fifty eight percent seem to be overconfident
in that they accept the claim
without believing the required evidence is there
or without realizing the type of evidence needed for this kind of claim
moving on, forty four percent believe that the models reproduce regularities
and data patterns
and twenty percent believe this is supported, which gives you fiftyfour
percent overconfidence
however
although thirty nine percent believe the models contain practical insights relevant to
those who wish to treat addictions
there was, according to our admittedly crude measures,
seventy two percent overconfidence on this matter
and while fifty six percent agreed the literature i could tell us about the
welfare of addicts under various policies
eighty percent of them
seem to be over confident in this
it thus seems that the evidence required for claims of pure theory
and as-if prediction
may be better understood than the evidence required for claims of causal
or
welfare insight
we realize fully that this is not the final word
and that the survey in some ways is more like a pilot than a conclusive piece of evidence
but we would claim that our hypothesis at least merits further research
and that, in any case, we should take care that claims about the real world
are not allowed to piggyback on well supported claims about pure theory
or a model`s ability to fit regularities in data
when you evaluate research, specify the exact claims made, and then examine
whether that specific type of claim is supported by relevant and sufficient
evidence
thank you for your time