A colleague of mine recently directed my attention to this interesting article from the January 9 edition of the Boston Globe: “That guy who called the big one? Don’t listen to him.” In the article, journalist Joe Keohane describes the now well-known 2006 prophecies made by Nouriel Roubini of New York University about an upcoming major financial crisis (he is described as, at that time, a “somewhat obscure economist“, which is extremely inaccurate I should say!). Due to his precise forecast about the crisis, which indeed began the year after, and took on an unprecedented world-wide scale in 2008, Roubini became a man to be taken really seriously. He apparently outsmarted the majority of economists who did not see the crisis coming. And he even gained a nickname for his capability of predicting this recession: “Dr. Doom”.
However, as the article goes, even though “Dr. Doom” predicted the big crisis, several of his subsequent predictions were not spot-on. As Keohane writes:
“For a prophet, he’s wrong an awful lot of the time. In October 2008, he predicted that hundreds of hedge funds were on the verge of failure and that the government would have to close the markets for a week or two in the coming days to cope with the shock. That didn’t happen. In January 2009, he predicted that oil prices would stay below $40 for all of 2009, arguing that car companies should rev up production of gas-guzzling SUVs. By the end of the year, oil was a hair under $80, Hummer was on its way out, and automakers were tripping over themselves to develop electric cars. In March 2009, he predicted the S&P 500 would fall below 600 that year. It closed at over 1,115, up 23.5 percent year over year, the biggest single year gain since 2003.”
Well, it is perhaps a bit cheap to dwell on some mistakes, which appear minor to the success in predicting the major event? But there is apparently a pattern here. Jerker Denrell of Stanford Graduate School of Business and Christina Fang of New York University has a new paper out in Management Science, vol. 56 (2010), “Predicting the Next Big Thing: Success as a Signal of Poor Judgment“. In this study they present a theoretical model of forecasting which imply that being able to successfully predict an “extreme event” (think “Dr. Doom” and the financial crisis), is consistent with having an intrinsically poor forecasting ability. They furthermore present evidence to support the model’s predictions. For example, using data on Wall Street forecasters they find that forecasters getting the “big events” right are among the poorest on average. A quote from their paper explains this nicely:
“The explanation is that because extreme outcomes are very rare, managers who take into account all the available information are less likely to make such extreme predictions, whereas those who rely on heuristics and intuition are more likely to make extreme predictions. As such, if the outcome was in fact extreme, an individual who predicts accurately an extreme event is likely to be someone who relies on intuition, rather than someone who takes into account all available information. She is likely to be someone who raves about any new idea or product.”
I like this story. I always had the gut feeling that one could suspect that those few who got it right about the financial crisis perhaps were just lucky. I mean, if you keep rambling about the coming of some scenario, at some point you will be right. And the more potential scenarios you ramble about, the more of your wild guesses may turn into reality. That does not necessarily mean that you have outsmarted the others. As Denrell and Fang shows us, you may indeed be systematically less smarter than others.
So “Dr. Doom” was maybe just “Mr. Lucky”? Probably not “just”, as Roubini is a very clever economist with an outstanding academic track record. But Denrell and Fang’s study is interesting as it warns us to be cautious about hailing those successfully predicting wild and crazy events as prophets.
UPDATE, February 11, 2011: Morten has directed my attention towards the following disclaimer now appended to the Boston Globe article:
Correction: Because of a reporting error, this article misattributed a statement about SUV production to economist Nouriel Roubini, and implied that he made an incorrect forecast about the failure of hedge funds in late 2008. He did not offer any advice about SUVs, and his prediction that hundreds of hedge funds would fail was correct.