The Reinhart/Rogoff Mistake and Economic Epistemology

April 18th, 2013 at 2:48 pm

By now, it is well known, at least in wonkish circles, that the economists Reinhart&Rogoff (R&R) made a number of errors in deriving their highly influential finding that debt-to-GDP ratios above 90% lead to slower growth.  Kudos to the various academics and bloggers who dug into this—particularly Herndan et al, who most recently found and documented the spreadsheet error and other questionable practices in R&R’s work, Mike Konczal who neatly summarized the critique, and Bivens and Irons, who back in 2010 showed the R&R thesis to be deeply flawed (and here’s a smart oped by Herndon’s co-authors).

Yet, as I noted in my brief review of this the other day, neither the older nor the more recent revelations are likely to have much impact on policy (or even, it would seem, on R&R, whose response has pretty been much been, “yeah, ok…but if we did it right, we still woulda got the same answer”).  In the first place, the fatal error in their work was not the spreadsheet error.  It was ignoring context (the unique, country, and time-specific reasons why is debt/GDP is rising) and thus conflating correlation with causality, specifically, periods when it’s slow growth leading to higher debt.

Why wouldn’t we expect a reaction from policymakers?  Because they’re using research findings the way a drunk uses a lamppost: for support, not for illumination.  If the R&R lamppost turns out to be wobbly, the austerions (or climate-change deniers, or supply-siders) will find another one.  In this town, I’m sorry to say, you can pretty much go think-tank shopping to buy the result you seek.

All of which brings me to a question of economic epistemology: in such a world as ours, how do we establish knowledge and facts in economics?  How do we know what we think we know?  Is the epistemological chain, if one exists, reliable—is our society getting what we need to make the best decisions?  Doesn’t really seem like it, does it?

There’s a lot more here than I can squeeze into one lil’ ol’ blog, but let me explain a bit.  By “chain,” I mean the creation and delivery mechanisms for the information/knowledge, the ones that initially failed in the R&R case: the research community and the media/dissemination function.

In quantitative economics, which this was, there are statistical rules regarding significance and establishing causation, and the peer review process does a decent job at enforcing those rules.  But R&R’s paper, like almost everything from think tanks, was not peer reviewed.

So the answer is to only accept peer-reviewed work as economic knowledge, right?  Nope.  That would be a) too limiting, and b) wouldn’t advance the epistemological cause as much as you think.  Peers have their own sets of biases, particularly as gate keepers.

So how can we establish better information?  Journalists need better BS meters, for one, but I don’t blame a reporter for getting a finding like “when debt to GDP exceeds 90%, growth slows because of it” from a prestigious economist and running with it.  I do blame them for not checking with others first–there are many equally prominent economists who immediately wondered about reverse causality driving R&R’s findings.  Checking with skeptics is especially important when the intellectual climate is one that provides knee-jerk support for anything that explains why we should fear high debt levels.

In fact, an understanding of the political motivations and climate may be a lot more important in establishing knowledge than we think, a lesson I hope we learn from this R&R episode.

There is a powerful, deep-pocketed movement, driven by ideology and personal financial interests, to shrink government and deregulate industry, especially financial markets.  This movement is greatly facilitated by findings that support debt reduction (as long as it comes from spending cuts, not tax increases, a formulation also congenial to most economists), regardless of the dynamics of the moment (i.e., the business cycle, zero-lower bound on interest rates, large output gaps, etc.).  Again, it is not a coincidence that Rep Ryan cites R&R as support for his safety-net-killing budget.

Facts, studies, reports that support that movement should be closely scrutinized, especially at a time like this, when austerity measures are demonstrably working very differently than their advocates claim.  That doesn’t mean they will always be wrong, of course.  But those who report and amplify such information want to be especially careful.  They may be turning on a highly unenlightening streetlamp.

Much more to say about this in coming posts…for now, I’m seeing a fair bit of soul-searching generated by the R&R revelation, and I think that’s very healthy.

Print Friendly, PDF & Email

20 comments in reply to "The Reinhart/Rogoff Mistake and Economic Epistemology"

  1. fyzzics says:

    Good points in this post, and the journalist’s dilemma, added to deadline pressure. It’s hard.

    I think that this paper was more important than most think-tank papers, though, because (1) it was not easily or obviously debunked (witness how long it took, even given its high profile), and (2) it came from the AER, a reputable journal – how’s the poor reader to know that the May issue isn’t peer reviewed at the same level as the rest of the year? I think this was a real hit. That’s not to say the drunk won’t get up again, but it’s like he had the support suddenly kicked out from under, and probably did a face plant. As we sometimes say when watching football: “Ooh, that’s gotta hurt!”

  2. tyler says:

    By now, it is well suspected, at least by ordinary people, that the economists Reinhart&Rogoff (R&R) made a number of intentional errors in deriving their highly influential finding that debt-to-GDP ratios above 90% lead to slower growth.

    • Barry says:

      Yes, it looks like they (a) selectively deleted data, (b) weighted the data in an extreme manner, (c) did not check to see if the weighting affected the results (which would have been trivially easy to do), (d) did not disclose the above, (e) made a number of public statements asserting proof of causality despite not having any, (f) refused to release the calculations for three years, and (g) were in the pay of Peter Peterson.

      At this point they should be charged with academic misconduct, stripped of tenure, and fired. Of course, that would probably be a new thing for both Harvard and academic economics (you know, holding people accountable).

  3. D. C. Sessions says:

    I find it interesting to compare the degree of credence lent to R&R on the one hand (two academics who did not, among other things, disclose their full methods) for one unreplicated paper, and to the entire body of work by climate scientists over the course of several decades.

    I’m not innocent enough to wonder at the relative receptions.

  4. Sandwichman says:

    Aside from the deep-pocketed movement, economics is founded on the epistemology of double-counting — if you can call double-counting an “epistemology”. Double counting arises from confusion between stocks and flows. It involves counting some things once as a stock and then a second time as a flow.

    The “stock” in question in economics is a representation of the flow. It is as if I were to think I had “two dogs”, one of which was an animal and the other the word “dog”. Ceci n’est pas un chien!

    Money is the culprit in this double-dealing, double-counting illusion. Money can represent the value of an asset. It also misrepresents it by virtue of the confusion between the abundance characteristics of the asset owned and the restrictive characteristics of ownership of that asset (John R. Commons).

    We pay the owner for access to the asset because he or she can otherwise restrict our access, not because the owner’s power to restrict “creates more” of the asset. But the epistemology of economics views the restriction as creation. Interest on money lent is taken as the model for return on capital invested and we mentally adhere to the metaphor even when it becomes utterly divorced from common sense.

    According to this metaphorical thinking money invested in a machine is imagined to continue to be entitled to earn a return on investment by virtue of the corporeality of the machine itself. The machine is supposedly a factor of production. Actually the causality works the other way. The machine earns a return on investment only because of the owner’s power to restrict production. Ownership is actually a factor of restriction. Thorstein Veblen understood this.

    • Jared Bernstein says:

      That’s freakin’ deep, dude. And I’ll need to think about it more, but I don’t fully get it. Imagine I work, create some value to someone, and get a paycheck. I then take my paycheck to the amusement park where I use it to gain access to the devices owned by that establishment. I rent them, as it were, in ways that bring my family some joy. Explain that transaction in your model.

      • Sandwichman says:

        Yep, it blows my mind! Crudely speaking, what I’m proposing is a synthesis of some remarks by Commons on “property”, Thomas Hodgskin’s co-existing labor critique of James Mill’s wages-fund doctrine (or “circulating capital”) and Irving Fisher’s definition of income. For now I’ll just tantalize with a few tidbits:

        Commons: “Going back over the economists from John Locke to the orthodox school of the present day, I found they always had a conflicting meaning of wealth, namely a material thing and the ownership of that thing. But ownership, at least in its modern meaning of intangible property, means power to restrict production on account of abundance while the material things arise from power to increase the abundance of things by production, even overproduction.”

        Hodgskin: “If we duly consider the number and importance of those wealth-producing operations which are not completed within the year, and the numberless products of daily labour, necessary to subsistence, which are consumed as soon as produced, we shall, I think, be sensible that the success and productive power of every different species of labour is at all times more dependent on the co-existing productive labour of other men than on any accumulation of circulating capital. The labourer, having no stock of commodities, undertakes to bring up his children, and teach them a useful art, always relying on his own labour; and various classes of persons undertake tasks the produce of which is not completed for a long period, relying on the labour of other men to procure them, in the meantime, what they require for subsistence. All classes of men carry on their daily toils in the full confidence that while each is engaged in his particular occupation some others will prepare whatever he requires, both for his immediate and future consumption and use.”

        Fisher: “A good definition should always conform to two tests: it must be useful for scientific analysis; and it must harmonize with popular and instinctive usage. We shall see that the usual definitions of income fail in one or both of these requisites. Many fail to lend themselves to scientific analysis by committing the fallacy of double counting, others by confusing income and capital, while almost all fail to harmonize with popular usage by making out income larger or smaller than common sense would dictate. Like most familiar notions, the notion of income seems to the uninitiated clear enough without definition. But pitfalls which are unseen are for that very reason all the more dangerous.”

  5. Richy says:

    Well said. Thank you.

  6. jonathan says:

    It’s the dishonesty that matters. Politicians have no credibility outside their true believers. But professions depend on credibility. R&R have shaken that. An Excel error is nothing.

    They not only excluded data from countries but they included one year from New Zealand when excluding the prior years. Their excuse? They say that data was new to them. That was 2010. This is 2013. They’ve had 3 years to look at that data, which means 3 years to note somewhere and somehow that a big chunk of their findings were absolutely wrong. They’ve been big speakers about their findings when they’ve now specifically admitted they knew about this unincluded data. That they chose to proceed with 1 year from NZ is rotten work. That they’ve never disclosed the truth is dishonest.

  7. Kevin Rica says:

    Something published in the AER is wrong and the referees didn’t catch it for ideological reasons (or conformity or because you can’t question Harvaard)? I’m shaken to my shoes.

    By pure coincidence, I happen to be reading a 2005 Rogoff and Obsfelf paper on the U.S. current account. Totally unperceptive and, in retrospect — zero predictive power.

    Oh, by the way. Isn’t the AER where David Card published his paper on immigration and wages? Maybe we shouldn’t take it is gospel either.

  8. Allen says:

    Thank you for this thoughtful post on the Reinhart/Rogoff paper. I work in the physical sciences, and seeing a graphic image of the spreadsheet they used (although perhaps a mock-up) made me laugh. I wouldn’t have touched that data with a ten foot poll. I wouldn’t even attempt to draw a conclusion based on such a lousy dataset with all the ‘n.a.’ entries. Should have waited for more data before publishing! And clipping off 3 data points in a simple Excel formula? Sounds like amateur night! Or am I missing something?

  9. Sue says:

    “So the answer is to only accept peer-reviewed work as economic knowledge, right? Nope. That would be a) too limiting, and b) wouldn’t advance the epistemological cause…Peers have their own sets of biases, particularly as gate keepers.”

    As a scientist, I find this assessment of the peer review process quite troubling. The peer review process has a long history of being the best system we’ve got for assessing scientific validity (or maybe, similar to democracy, the worst system except for all the others).

    In what way would it be limiting? Only in the urgency of getting papers to publication quickly? But this paper has been out for 3 years. Surely a peer review could have happened well before now, presumably discovering the problematic aspects of the paper.

    And as far as peers having their own biases, yes of course they do. But as long as an honest attempt is made to recruit peer reviewers from a range of the political spectrum, the system should work.

    Counting on the journalists to be the gatekeepers of the accuracy of a paper when they don’t have the expertise to analyze its mathematical or scientific results seems ludicrous. On the other hand, counting on them to call out an organization that is padding their “peer reviewers” with right-wing (or left-wing) demagogues seems more reasonable.

    And, by the way, I was ASTOUNDED that this paper was not peer-reviewed–I don’t care how “esteemed” one is amongst one’s colleagues; one is never above being judged by one’s peers!

    • Kevin Rica says:

      Even in the hard sciences, there are plenty of cases of scientific fraud.

      And there are also certain research topics that can’t be openly discussed because of political pressure — from both sides of the spectrum (e.g. geoengineering).

    • Jared Bernstein says:

      I started to answer this in a comment but I think it’s more of a post–coming soon.

  10. R. Nemo says:

    Economics is not science. Not even a ‘Dismal science.” It is the rationalization of perceived class interests.

    Only the ignorant think that econometrics tells us anything about the real world. They are just abstract models that signify nothing. Tools to be used for crass class and political interests…

    Societies are complex organic systems that cannot be quantified or understood. they are qualitative. As I like to say: We are too imbedded in reality to perceive reality.

    As for statistics! Just poetry with numbers. The fundamental problem is that you can never prove causation. You can ask the question: Is growth causing high debt or is it the opposite. Perhaps it is actually a third or fourth factor you did not account for. The paradoxes are infinite.

    We want to live in a predictable world so we make up these stories–but they are nonsense. Reality is qualitative. Numbers tell us nothing about the real world. They are mere poetic metaphors.

    Enjoy the R/R debate over which myth is right or wrong.

    The real question is who’s greed is served by who’s policy. That’s human motivation and that’s easy to figure out–for some…

  11. Pablo says:

    In the interest of better information can we focus on “multipliers” on Government spending? I have read opinions that the multipliers on the Stimulus program were everywhere from 0.5 to 2.5. This is a really important factor in the debate over stimulus versus austerity. If we can’t be sure that the stimulus was significantly above 1.0 it makes no sense to barrow to spend.


  12. Fred Donaldson says:

    The discovery of the 90% error was not by a team, although a student and his teachers collaborated to prepare the paper on the error. The student deserves all the credit, and it is an amazing story of how someone, studying on their own, can find obvious facts that the “experts” missed for years.

  13. Robert Goodman says:


    Kindly point to the writings of folks who are searching their souls over this. I confess to being inclined to read those with whom I feel in sympathy. And they are tending to wonder why they’ve been so polite in this debate until now. I hope that’s not what you mean by soul searching.