When the Results Look Weird, Check the Methods…Carefully!

June 29th, 2013 at 10:46 am

Some folks have asked me to weigh in on the seemingly implausible findings in this recent paper by Richard Burkhauser.  Thomas Edsall usefully raised some objections in this NYT piece, but at least as I read his critique, he failed to make the critical point: though the research asks a legitimate question, the results make no sense.

By finding that between 1989 and 2007, the real incomes of the wealthiest households have fallen—quite sharply by some of their measures—while those of the poorest households have gone up, their results fly in the face of the vast majority of inequality research based on a wide variety of data sources.

That doesn’t mean they’re wrong, of course, but it does mean you’d want your work to be as solid as possible, methodologically, before concluding that the trajectory of inequality is the opposite of what we thought (“when you shoot the king, don’t miss”).   As I (and other inequality experts) read what they’ve done, they’ve proved not that the trends go the opposite way but that their methods and data are flawed.

The paper, by Armour, Burkhauser (the spokesperson for the authors in the Edsall piece), and Larrimore, attempts to create a comprehensive measure of income using information on wages, capital income, business income, taxes, and transfers from a wide variety of sources, combining these sources in ways that ultimately appears to have gone wrong.

The figure below plots their findings (first two bars in each set) against the highly regarded Congressional Budget Office Household Income series (third bar).  Both sets of data are post-tax, post government transfers, like unemployment benefits or the value of food stamps.

The main difference between ABL and CBO is how they treat capital gains.  The CBO uses high quality tax data with the amount of realized capital gains reported by tax filers.  ABL try to do assign annual changes in unrealized gains—the extent to which the value of assets fluctuates each year—to households that hold such assets.

There are good arguments for both approaches but my point here is that the ABL approach demands more and better data than they had, as revealed by their peculiar results.  Yet instead of the skepticism those results should have generated, they argue that their approach “…dramatically reduces the observed growth in income inequality across the distribution.”

The CBO findings over these years reveal far faster growth at the top of the income scale than at the middle or low end.  In fact, in the CBO data, the real income of the top 5% of households grew three times the rate of the middle (90% vs. 30%), and the top 1% (not shown), more than doubled, up 128%.  Conversely, the ABL findings show real incomes growing more quickly at the bottom end of the income scale, flat in the middle, and falling in real terms at the top.



Source: ABL (tbl 3, col 6-7), CBO 

As EPIs State of Working America shows, this unequal pattern of growth shows up in wages from labor market surveys, earnings from Social Security records (high-quality administrative data), and net worth (assets-liabilities)—each of these is from a different data source.  CBO data reveal that taxes and transfers, while still progressive, have gotten less so over the last few decades—i.e., they reduce market-driven inequality, but they do so a bit less than they used to.  Put all of those trends together, and add in the fact that the macroeconomic data on “factor shares”—the share of national income going to compensation (the source of most income for middle and low-income families) is now at a 50-year low and that going to profits at an all-time high—are pointing in the same direction, and it should be clear that something is wrong with ABL’s methods.

Where they seem to have jumped the analytic shark is in imputing the appreciation or losses of unrealized capital, business, and housing wealth to households.  Here they make two errors that significantly distort the findings (h/t’s to Gary Burtless and Dean Baker, both cited in the Edsall piece).

First, since asset prices jump around from year to year, their results are highly sensitive to the years they choose to compare.  Since 1989, their base year, was one of relatively high stock market returns (up over 20%) relative to 2007 (up less than 5%–see their figure 1), the imputations will boost the base year incomes relative to the final year.  This is even a bigger problem with housing—had their endpoint been a year or two later, after the bubble burst, their results would have been completely different.

When results are that sensitive to endpoint choice, it’s usually a sign that something’s wrong with your methods such that you’re not revealing a meaningful trend.   In this case, ABL purport to show readers that the extent of inequality is the opposite of what they thought—as in the figure above, the poor are getting richer and the rich, poorer.  But if that result changes with minor endpoint changes, it’s too fragile a finding upon which to reverse conventional wisdom backed by much more evidence.

The second problem, emphasized by Burtless, is that they assigned asset same percentage returns in stock and home value appreciation to everyone who owns stocks or a home.  In other words, there’s no distribution of returns—everyone with any holdings is an equally skilled investor.  This assumption wipes out a major source of inequality: some investors did a whole lot better than others.  How can we measure inequality when we assume away dispersion!?

There are other problems—see Dean and Edsall’s pieces—but at this point, if you’re running these numbers and your results are coming up the opposite of the almost every other study, you’d really want to stop and think about the impact of assumptions like those above regarding the dispersion of returns and the timing of volatile asset prices.

Once you see these results—and in one appendix table, real income falls for every income group, 1989-2007 (so…um…where did the economy’s growth go?)—your best move is to say, “we have proved that given data constraints, we are unable to reliably impute income including yearly changes in asset valuations.”  That’s actually a helpful finding.  To plough ahead without that introspection, and to claim your findings reveal “dramatic” reductions in inequality suggests either methodological carelessness or an ideological thumb on the scale.

Print Friendly, PDF & Email

8 comments in reply to "When the Results Look Weird, Check the Methods…Carefully!"

  1. Stephen W says:

    A third possibility for this sort of work is “determination to show a big, surprising result,” which has more intentionality than mere carelessness but less directionality than ideology.

  2. Kevin Rica says:

    I’m confused about the graph. Are the percentage increases for average households for each group? The aggregate incomes of each group? For the median household? Need better labeling.

  3. Fred Donaldson says:

    With the falling price of homes many elderly folks have cashed in substantial amounts of their IRAs to pay unexpected expenses, versus going for home equity loans.

    These funds, along with withdrawals for nursing homes (%75k a year), medical expenses, and living expenses, are all counted as income and taxable as such on their income tax returns.

    The worse the economy, the more elderly raid their IRAs and 401Ks, increasing their “income” reported to IRS. Rich folks with millions in the bank that are not IRAs can just withdraw and not count that as taxable income, just transfer of wealth.

  4. Robert Goodman says:

    But Stephen and Jared…by choosing the research subject in which to come up with a big surprise they sort of reveal their aim, don’t they?

  5. Alex Bollinger says:

    “when you shoot the king, don’t miss”

    Amen! I like that expression. Back in my online media days I made the same argument with a PR guy for a nonprofit that was arguing that Americans were becoming *less* accepting of same-sex marriage: even if he were right (he wasn’t), he was drowning in other surveys and data that said he’s wrong.

    Although I think ABL could persuasively argue (although they won’t) that they weren’t aiming at the king. Redefine “king” to be “the people with a disproportionate amount of power and wealth” and they’re decidedly shooting in the king’s defense.

  6. Kevin Rica says:

    Then by eyeballing the table, according to CBO, everybody’s average income went up, but the top 5% got a bigger share over an 18-year period and two business cycles.

    According to ABL, the average household income for all households must have gone down; although the the lowest two quartiles (with the smallest weights) saw positive growth and the biggest share of the pie (top 5%) shrank.

    But during the same period, average per capita income increased about 45%. So in the ABL study, where did it go? That is weird! The only way to reconcile that would be for average household size to shrink (possibly did happen) or for household income to shrink during the period.

    It would be nice if a total (average income for all households) could be squeezed into the graph. It would help clarify the data.

  7. urban legend says:

    Off your excellent point, but I don’t think you — like almost everyone else who uses it — know what “jump the shark” means. It does not mean the same thing as “go off the deep end.” A stupid metaphor that nobody understands should be retired.