The Failure to Forecast the Great Recession
Simon Potter
November 25, 2011

>

Experience shows that what happens is always the thing against which one has not made provision in advance.
– John Maynard Keynes1

Our best plan is to plan for constant change and the potential for instability, and to recognize that the threats will constantly be changing in ways we cannot predict or fully understand.
– Timothy Geithner2

>

The economics profession has been appropriately criticized for its failure to forecast the large fall in U.S. house prices and the subsequent propagation first into an unprecedented financial crisis and then into the Great Recession. In this post, I examine the performance of the forecasts produced by the economic research staff of the Federal Reserve Bank of New York (New York Fed) over the period 2007-10 and consider some of the reasons why we, like most private sector forecasters, failed to predict the Great Recession. This spreadsheet contains staff forecasts, the outcomes, and a standard measure of private sector forecasts—the Blue Chip consensus. In addition, staff material prepared for bi-annual meetings of the New York Fed Economic Advisory Panel provide some further insights into the evolution of the staff outlook.

The staff forecasts of real activity (unemployment and real GDP growth) for 2008-09 had unusually large forecast errors relative to the forecasts’ historical performance, while the forecasts for inflation were in line with past performance. Moreover, although the risks to the staff outlook were to the downside throughout this period, it wasn’t until fall 2008 that a recession as deep as the Great Recession was given more than 15 percent weight in the staff assessment.

How Bad Were the Forecasts for Real Activity?

Economic forecasters never expect to predict precisely. One way of measuring the accuracy of their forecasts is against previous forecast errors. When judged by forecast error performance metrics from the macroeconomic quiescent period that many economists have labeled the Great Moderation, the New York Fed research staff forecasts, as well as most private sector forecasts for real activity before the Great Recession, look unusually far off the mark.

One source for such metrics is a paper by Reifschneider and Tulip (2007). They analyzed the forecast error performance of a range of public and private forecasters over 1986 to 2006 (that is, roughly the period that most economists associate with the Great Moderation in the United States).

On the basis of their analysis, one could have expected that an October 2007 forecast of real GDP growth for 2008 would be within 1.3 percentage points of the actual outcome 70 percent of the time. The New York Fed staff forecast at that time was for growth of 2.6 percent in 2008. Based on the forecast of 2.6 percent and the size of forecast errors over the Great Moderation period, one would have expected that 70 percent of the time, actual growth would be within the 1.3 to 3.9 percent range. The current estimate of actual growth in 2008 is -3.3 percent, indicating that our forecast was off by 5.9 percentage points.

Using a similar approach to Reifschneider and Tulip but including forecast errors for 2007, one would have expected that 70 percent of the time the unemployment rate in the fourth quarter of 2009 should have been within 0.7 percentage point of a forecast made in April 2008. The actual forecast error was 4.4 percentage points, equivalent to an unexpected increase of over 6 million in the number of unemployed workers. Under the erroneous assumption that the 70 percent projection error band was based on a normal distribution, this would have been a 6 standard deviation error, a very unlikely occurrence indeed.

Did We Calibrate the Risks to the Forecast Appropriately?

Of course, there is much more to forecasting than the point forecasts reported in the spreadsheet. In particular, it is crucial to assess the uncertainty and risks around any point forecast.

Throughout this period, the uncertainty and downside risks assessed around our point forecast were substantial relative to economic fluctuations in the Great Moderation. One way of gauging the appropriate calibration of downside risk is to measure the depth of the implied recession if the risks were realized.

The chart below does this by considering the probability distribution of the four consecutive quarters with the lowest GDP growth in a recession. It presents results based on the staff outlook in April 2008 and November 2008. The depth of the mild recessions shown in the chart was typical of the type of recessions expected during the Great Moderation. The actual depth of the 2007-09 recession as gauged by this metric is currently estimated to be 5 percent. As the chart shows, it was only by November 2008 that the probability of the actual outcome was above 15 percent.

Depth-of-recession

Upon seeing this type of calculation, Robert Barro, a Harvard professor and member of the New York Fed Economic Advisory Panel, noted that the decline in real stock market values in the United States was similar to that observed in countries experiencing depressions. Taking this relationship into account in the calibration of the downside risks produced about a 50/50 chance of the currently observed depth of the Great Recession (see the March 2009 Wall Street Journal op-ed article by Barro for his assessment of the probability of a depression).

Recent Forecast Performance

In contrast, the New York Fed staff forecasts for 2010 made in 2009 and early 2010 are quite accurate (under the assumption of no major revisions to the estimates of GDP growth in 2010). This accuracy, however, has not been sustained through 2011. As widely discussed by a number of Federal Reserve officials, the level of real activity in 2011 has been disappointing relative to expectations. This shortfall is evident in the chart below, which compares forecasts for GDP growth in 2011 and 2012 produced in April and October 2011. However, this chart also depicts the uncertainty and risks around the staff forecast as of April 2011. Given the uncertainty around the April forecast, the subsequent changes to the outlook are not very surprising. On the other hand, near-term downside risks to this forecast were low compared to other forecasts produced in the last four years, so the direction of the change was more surprising.

Chart-2_potter_with-note

Why Did We Fail to Forecast the Great Recession?

The quotations from Keynes and Geithner at the start of this post capture the importance of constantly striving to ensure that policy is robust to unexpected events. As explained in much of the recent work of the 2011 Nobel Prize–winning economist Tom Sargent, the unexpected events for which policymakers need to make provision have the characteristic of being the most likely unlikely bad event. The collapse in housing prices and its propagation to the economy certainly fit this description.

A leading example of how effective a robust approach to policymaking can be is the 2009 Supervisory Capital Assessment Program. In this program, large U.S. banks were evaluated against a capital standard under the assumption of a longer and deeper recession than contemplated in the prevailing consensus estimate. The idea was that if banks had sufficient capital to continue performing their intermediation function under this more adverse scenario, the scenario was less likely to occur.

Indeed, in the period leading up to the financial crisis, analysts who were suspicious of the stability of the Great Moderation, such as Nouriel Roubini, offered assessments that proved to be significantly more accurate than the point forecasts of New York Fed research staff or most professional forecasters in gauging the potential for unlikely bad outcomes. On a more positive note, if one compares the downside risk in the New York Fed research staff outlook with that of the Survey of Professional Forecasters (see the chart below from April 2008), there is some evidence that we had a more sober assessment of the risk of a severe downturn than did private sector forecasters.

Growth-real-GDP-bar

Looking through our briefing materials and other sources such as New York Fed staff reports reveals that the Bank’s economic research staff, like most other economists, were behind the curve as the financial crisis developed, even though many of our economists made important contributions to the understanding of the crisis. Three main failures in our real-time forecasting stand out:

  1. Misunderstanding of the housing boom. Staff analysis of the increase in house prices did not find convincing evidence of overvaluation (see, for example, McCarthy and Peach [2004] and Himmelberg, Mayer, and Sinai [2005]). Thus, we downplayed the risk of a substantial fall in house prices. A robust approach would have put the bar much lower than convincing evidence.
  2. A lack of analysis of the rapid growth of new forms of mortgage finance. Here the reliance on the assumption of efficient markets appears to have dulled our awareness of many of the risks building in financial markets in 2005-07. However, a March 2008 New York Fed staff report by Ashcraft and Schuermann provided a detailed analysis of how incentives were misaligned throughout the securitization process of subprime mortgages—meaning that the market was not functioning efficiently.
  3. Insufficient weight given to the powerful adverse feedback loops between the financial system and the real economy. Despite a good understanding of the risk of a financial crisis from mid-2007 onward, we were unable to fully connect the dots to real activity until 2008. Eventually, by building on the insights of Adrian and Shin (2008), we gained a better grasp of the power of these feedback loops.

However, the biggest failure was the complacency resulting from the apparent ease of maintaining financial and economic stability during the Great Moderation. Perhaps most important, as noted by some analysts as early as the 1990s, these adverse consequences of the Great Moderation were most likely to arise from the actions, judgments, and decisions of financial market participants:

Longer stretches of economic growth imply greater leverage and complacency and thus, greater financial problems when recessions do occur.
–William Dudley and Edward McKelvey3

___________________________
1Letter to Jacob Viner, June 9, 1943, Collected Writings of John Maynard Keynes, ed. Donald Moggridge, vol. 25. London: Macmillan, 1980.

2Letter from the Chair, Financial Stability Oversight Council Annual Report.

3The Brave New Business Cycle: No Recession in Sight, Goldman Sachs Economic Research Group, January 1997.

Category: Analysts, Economy, Federal Reserve, Think Tank

Please use the comments to demonstrate your own ignorance, unfamiliarity with empirical data and lack of respect for scientific knowledge. Be sure to create straw men and argue against things I have neither said nor implied. If you could repeat previously discredited memes or steer the conversation into irrelevant, off topic discussions, it would be appreciated. Lastly, kindly forgo all civility in your discourse . . . you are, after all, anonymous.

12 Responses to “NY Fed: The Failure to Forecast the Great Recession”

  1. Rouleur says:

    Freakin’ moron…

  2. stevphel says:

    Hoo-coulda-knowed-it-ism at its finest. File under Big Lie.

  3. Asymptosis says:

    IOW, what Minsky said.

  4. [...] You may have missed Simon Potter’s publication on Friday “The Failure to Forecast the Great Recession.” [...]

  5. BobCarver says:

    Bottom line: economics is bunk. Nothing more than witch-doctorism.

  6. DeDude says:

    I always give credit to people who are willing to analyze their own failures. At least they have a chance of improving their future performance.

    My problem are with those who say: “I know everything and this is the future”. Then after their predictions fail spectacularly they go stone-faced for a few days before they again say: “I know everything and this [new thing] is the future”.

  7. RW says:

    What Rouleur, stevphel, and Asymptosis said.

    Potter’s mea culpa is insufficient, virtually a whitewash.

    The giveaway for me was the citation of a fresh water, Real Business Cycle (RBC) theorist such as Robert Barro — who, to this day, still insists the RBC models show the recession is a function of people being unwilling to work as much as anything else — instead of choosing any of the dozens of neo-Keynesians or Modern Monetarists whose models not only more accurately represent what happened in the real world but predicted it as probable.

    Instead of an alternative model we get this “mistakes were made” on the course the NY Fed had already chosen w/ no indication that course might change and no deep analysis or reflection on the possibility (I would say the high probability) that the mistakes made were fundamental, that given the assumptions the Fed economists were operating under and models they were following, gross errors were in fact highly probable.

    That would be an analysis worth reading and, from my POV at least, the only kind of mea culpa that might make me believe any of the NY Fed economists should keep his or her job.

  8. twsf says:

    “Even as he quaked with fear of such a catastrophe, he knew he was letting himself wallow in it for a superstitious reason. If you consciously envisioned something that dreadful, then it couldn’t possibly take place, could it… ? God or Fate would refuse to be anticipated by a mere mortal, wouldn’t He… ?” – Tom Wolfe

  9. RW says:

    What Rouleur, stevphel, and Asymptosis said.

    Potter’s mea culpa is insufficient, virtually a whitewash.

    The giveaway for me was the citation of a fresh water, Real Business Cycle (RBC) theorist such as Robert Barro — who, to this day, still insists the RBC models show the recession is a function of people being unwilling to work as much as anything else — instead of choosing any of the dozens of neo-Keynesians or Modern Monetarists whose models not only more accurately represent what happened in the real world but predicted it as probable.

    Instead of an alternative model we get this “mistakes were made” on the course the NY Fed had already chosen w/ no indication that course might change and no deep analysis or reflection on the possibility (I would say the high probability) that those mistakes were fundamental, that given the assumptions the Fed economists were operating under and models they were following, errors this gross were highly probable.

    That would be an analysis worth reading and, from my POV at least, the only kind of mea culpa that might make me believe any of the NY Fed economists should keep his or her job.

  10. Giovanni says:

    What twsf says.

    Essentially then an economist’s job is to imagine and write about the worst thing they can imagine so that it doesn’t come to pass: “the unexpected events for which policymakers need to make provision have the characteristic of being the most likely unlikely bad event.”

  11. XRayD says:

    Our best plan is to plan for constant change and the potential for instability, and to recognize that the threats will constantly be changing in ways we cannot predict or fully understand.
    – Timothy Geithner

    And this guy is our Sec. of the Treasury?

    I’m gonna pilot this plane, but I’m not sure what’s going to happen after we take off, but we’ve got plans and they will change!

    Be Afraid. Very Afraid.

  12. AHodge says:

    well thanks for sharing what a complete incompetent loser and embarassment to the economics profession this fellow is.
    Citing Barro was an immediate tipoff. the article March 2009 where barro forecasts a 20% chance
    not just of the 2008 disaster that happened
    but a full blown Great Depression.
    based on some intl data with completely fallaceous ties to stock markets,
    that was about a month before the US stock market turned up.
    otherwise perhaps not surprising they found US housing not overvalued, also because they were expecting efficient markets everywhere,
    he is still bumBling around looking for shocks and unexpected variances
    and managed to completely ignore
    except for his moronic feedback loops
    that securitization had shut down by mid 2008