In yesterday’s afternoon reads, I mentioned a Boston Globe story “That guy who called the big one? Don’t listen to him.”

While it ostensibly looks at the track record of Nouriel Roubini, it isn’t really about him –its really about outliers and mean reversion in forecasting.

“How can someone with the insight to be so right about a major event be so wrong about so many other ones? According to a recent study, it’s simple: The people who successfully predict extreme events, and are duly garlanded with accolades, big book sales, and lucrative speaking engagements, don’t do so because their judgment is so sharp. They do it because it’s so bad . . .

[Oxford economist Jerker Denrell and Christina Fang of New York University] are the latest in a long line of researchers dismantling the notion that predictions are really worth anything. The most notable work in the field is “Expert Political Judgment” by Philip Tetlock of the University of Pennsylvania. Tetlock analyzed more than 80,000 political predictions ventured by supposed experts over two decades to see how well they fared as a group.

The answer: badly. The experts did about as well as chance. And the more in-demand the expert, the bolder, and thus the less accurate, the predictions. Research by a handful of others, Denrell included, suggests the same goes for economic forecasters. An accurate prediction — of an extreme event or even a series of nonextreme ones — can beget overconfidence, which can lead to making bolder and bolder bets, and thus, more and more errors . . .

There’s no great, complex explanation for why people who get one big thing right get most everything else wrong, argues Denrell. It’s simple: Those who correctly predict extreme events tend to have a greater tendency to make extreme predictions; and those who make extreme predictions tend to spend most of the time being wrong — on account of most of their predictions being, well, pretty extreme. There are few occurrences so out of the ordinary that someone, somewhere won’t have seen them coming, even if that person has seldom been right about anything else.”

Fascinating stuff, worth reading in full . . .


The Folly of Forecasting (June 2005)

The Illusory World of Economic Forecasting (September 19th, 2006)

Guru Time of Year (January 2nd, 2011)

That guy who called the big one? Don’t listen to him.
Joe Keohane
Boston Globe, January 9, 2011

Category: Really, really bad calls

Please use the comments to demonstrate your own ignorance, unfamiliarity with empirical data and lack of respect for scientific knowledge. Be sure to create straw men and argue against things I have neither said nor implied. If you could repeat previously discredited memes or steer the conversation into irrelevant, off topic discussions, it would be appreciated. Lastly, kindly forgo all civility in your discourse . . . you are, after all, anonymous.

13 Responses to “Inside the Paradox of Forecasting”

  1. Ilya says:

    The article should have been titled “In Defence of Mediocrity.” No wonder that tenured professorial types make the big bucks… What a hoot!

  2. b_thunder says:

    yes, interesting date, but at least in case of Roubini (and perhaps a few others) i think a clarification is needed:

    Roubini said that his early recession predictions were based on his studies of the data from other countries. Especially Latin American countries. From the same case studies he could have predicted the chain of events: RE crisis, then banking insolvency, then the general economic recession, etc etc.
    What he could not have sen in the prior data and could not have predicted is the magnitude of “extend and pretend” that followed: TARP, abolition of the accounting standards, AIG/GSE backdoor bank bailouts, “Bernanke Put”, QE 1.0, QE 1.5, QE 2.0. And who outside of the little “circle of power” could have imagined that that both administrations will throw trillion of dollars of public money to “save the system” (or rather to save the Wall St) while jeopardizing the standard of living for the future generations?

    So, yes, for the predictions of once-in-a-generation events, pay attention to the Roubinis of the day. For the daily predictions – there’s Cramer, Tepper, and the “Buy the F*&^% dip” puppets on youtube.

  3. constantnormal says:

    It strikes me that there are (at least) a couple of alternative ways to spin this … one being the approach of John Bogle, that one should just buy-and-hold index funds, allowing the rising mean to carry one forward and ignoring all the noise … another being that the future does not follow any fixed pattern, that one should keep a light-on-ones-feet posture, reacting to every little pop and dip in the road …

    I’m sure that there are other ways to spin this as well — the holy sect of the Elliott Wave, the fundamentalists, the Way of An Infinity of Charts, etc, etc.

    The truth is that nothing works demonstrably better than anything else over a long enough time horizon. How long a time horizon is sufficient for your purposes? I guess it depends …

    But nothing, repeat NOTHING, beats being lucky.

  4. seneca says:

    Did this article come out before Roubini settled on his $5.5 million NYC condo?

    Laszlo Birinyi says the S&P500 will gain 125 percent (excluding dividends) by Sept 4, 2013. Now that I know that almost all extreme predictions are rubbish, I’m going to close out all those ridiculously far-out-of-the-money LEAPS I had loaded up on.

  5. FC says:

    I think this goes back to Taleb’s points on randomness and the unpredictability of things. Everyone is going to guess some things right some of the time, so purely due to coincidence someone is going to predict the next rare event or get the direction of this or that right. But does that make their current guesses any more accurate? I think not.

    The vast majority of what’s out there is a combination of people trying to sell their own products, fill up space, or get attention/viewers. Bogle’s approach applied across a variety of asset classes seems to be the most reliable way to do well, with the right allocation based on resources and age. “Reacting to every little pop and dip in the road” is a recipe for disaster, as you never know which dip will become a dive, etc. So not only will you be pulling your money in and out, missing out on gains, but you’ll also be incurring transaction fees that will kill you in the long run. Having a stop-loss point is a good idea, but other than that I see obsession with current events and forecasting as a good way to lose.

  6. Arequipa01 says:

    This post from 8/20/08 touched on an aspect of forecasting that bears keeping in mind:

    Additionally, here is a comment on orders of magnitude (range of possibilities) and imponderables that rambles around and hides its point:

    “MRegan Says:
    August 20th, 2008 at 8:56 pm
    I was struck by the last line in the quote from D. Kass, “Too many imponderables…”.
    The notion reminded me of an argument I had with a man named Gilbert Butler (he’s some kind of PE guy out of New York)- I met him through a fluke of fate in Arequipa, Peru in January. Since there is limited room here I will boil it down. He wanted me to give an estimate of how long it would take him and guide to ride on bicycles from the entrance of Colca Valley to Chivay where he wanted to meet up with a glorified site superintendent to view a church restoration in a town called Maca. He, having been educated by Benedictines in RI, and I, by Jesuits in Chicago, proceeded to engage in dueling dancing angels over imponderables and orders of magnitude. My wife, a hard-headed characata, found this pricklish and tossed a number at him to put the kibosh on the argument.
    This leads me back to Kass, convictions and healthy fear in the face of the unknown and what to do. True, there are too many imponderables but the notion of orders of magnitude can help orient onesself in this current fracas. In regard to the financials it seems that if one regards the proportions of leveraging engaged in and the current deleveraging going on then a smart person could map out a general course for many of these entities. Timing is extremely difficult but if you kind of know where you’re going you’ll make it ok. Butler later conceded to me that insisting on setting a time was a mistake. Yes, it was, but it doesn’t matter. That old, disagreeable coot got on a bicycle at about 4000 mts above sealevel, rode his bike to Chivay, and finally made it to Maca. I might add that he is extremely rich and I alas am not (I blame the Jesuits).
    We need both the courage of our convictions and the courage to question our convictions.”

  7. Long term says:

    I predict the world economy will recover and everyone will be happy and have a job.

    (Now if this happens, maybe I can turn it into a $5.5 million NYC condo)

  8. northendmatt says:

    Ha, probability – at last, something I know a thing or two about!

    Given the difficulty of predicting the future, it is not surprising that the experts don’t do much better than chance. And given enough experts, the odds are that, out of luck, some of them will get on a “hot streak” from time to time, and be seen as prophets.

    Still, it is worth noting that this is aggregated data. Are all experts, over the long run, no better than chance? (Even if they’re not, it is very unlikely the data reveals this.) Or are there some who are consistently right and some who are consistently wrong? Or are there some who are paid to say certain things regardless of whether they’re right or wrong?

  9. dsawy says:

    “Prediction is very hard, especially about the future.”

    - Yogi Berra

  10. sinomania says:

    This concept definitely applies to the predictions of nearly all so-called China “experts” particularly Jim Chanos, Marc Faber, Andy Xie, and the extremest of the extreme Gordan Chang…

  11. Bill W says:

    We need an economic system that is robust enough to survive the inevitable crisis that no one can predict, instead of one that relies on a Federal Reserve team of wise men who think they can predict the future.

  12. Mike in Nola says:

    Comment on the subject in the FT:

    My take is you first need to sample people who really have an ability to think. Most of these so-called experts are just hacks. Look at who’s risen to head the Federal Reserve! The man who believe it was all contained and still thinks he can solve everything by giving money to primary dealers to buy stocks. Roubini is ok, but don’t think he is very profound; he is good at getting publicity.

    Second, as someone pointed out, why 2002-2005? Not a very long time line.

    Finally, even actually smart people have seemed to underestimate the ability of the politicians and Federal Reserve and the EU (and the PBOC, sinomania) to kick the can down the road by printing trillions. These actions have only postponed much of what has been predicted, not prevented it. The operative thought seems to be to prop up the system til things “get better” without actually fixing the what prevents things from getting better.

  13. jad714 says:

    Speaking of forecasting, where do you think the DOW will be on Feb. 28, 2011? Tell us by Jan. 12 midnight and win “The Little Book of Sideways Markets.”