Bob Shiller tells his personal story of why many people knew the bubble coming but still yielded to the 'consensus' view.  A delightful reading from NYT.

Challenging the Crowd in Whispers, Not Shouts

ALAN GREENSPAN, the former Federal Reserve chairman, acknowledged in a Congressional hearing last month that he had made an “error” in assuming that the markets would properly regulate themselves, and added that he had no idea a financial disaster was in the making. What’s more, he said the Fed’s own computer models and economic experts simply “did not forecast” the current financial crisis.

Mr. Greenspan’s comments may have left the impression that no one in the world could have predicted the crisis. Yet it is clear that well before home prices started falling in 2006, lots of people were worried about the housing boom and its potential for creating economic disaster. It’s just that the Fed did not take them very seriously.

For example, I clearly remember a taxi driver in Miami explaining to me years ago that the housing bubble there was getting crazy. With all the construction under way, which he pointed out as we drove along, he said that there would surely be a glut in the market and, eventually, a disaster.

But why weren’t the experts at the Fed saying such things? And why didn’t a consensus of economists at universities and other institutions warn that a crisis was on the way?

The field of social psychology provides a possible answer. In his classic 1972 book, “Groupthink,” Irving L. Janis, the Yale psychologist, explained how panels of experts could make colossal mistakes. People on these panels, he said, are forever worrying about their personal relevance and effectiveness, and feel that if they deviate too far from the consensus, they will not be given a serious role. They self-censor personal doubts about the emerging group consensus if they cannot express these doubts in a formal way that conforms with apparent assumptions held by the group.

Members of the Fed staff were issuing some warnings. But Mr. Greenspan was right: the warnings were not predictions. They tended to be technical in nature, did not offer a scenario of crashing home prices and economic confidence, and tended to come late in the housing boom.

A search of the Federal Reserve Board’s working paper series reveals a few papers that touch on the bubble. For example, a 2004 paper by Joshua Gallin, a Fed economist, concluded: “Indeed, one might be tempted to cite the currently low level of the rent-price ratio as a sign that we are in a house-price ‘bubble.'” But the paper did not endorse this view, saying that “several important caveats argue against such a strong conclusion and in favor of further research.”

One of Mr. Greenspan’s fellow board members, Edward M. Gramlich, urgently warned about the inadequate regulation of subprime mortgages. But judging at least from his 2007 book, “Subprime Mortgages,” he did not warn about a housing bubble, let alone that its bursting would have any systemic consequences.

From my own experience on expert panels, I know firsthand the pressures that people — might I say mavericks? — may feel when questioning the group consensus.

I was connected with the Federal Reserve System as a member the economic advisory panel of the Federal Reserve Bank of New York from 1990 until 2004, when the New York bank’s new president, Timothy F. Geithner, arrived. That panel advises the president of the New York bank, who, in turn, is vice chairman of the Federal Open Market Committee, which sets interest rates. In my position on the panel, I felt the need to use restraint. While I warned about the bubbles I believed were developing in the stock and housing markets, I did so very gently, and felt vulnerable expressing such quirky views. Deviating too far from consensus leaves one feeling potentially ostracized from the group, with the risk that one may be terminated.

Reading some of Mr. Geithner’s speeches from around that time shows that he was concerned about systemic risks but concluded that the financial system was getting “stronger” and more “resilient.” He was worried about the unsustainability of a low savings rate, government deficit and current account deficit, none of which caused our current crisis.

In 2005, in the second edition of my book “Irrational Exuberance,” I stated clearly that a catastrophic collapse of the housing and stock markets could be on its way. I wrote that “significant further rises in these markets could lead, eventually, to even more significant declines,” and that this might “result in a substantial increase in the rate of personal bankruptcies, which could lead to a secondary string of bankruptcies of financial institutions as well,” and said that this could result in “another, possibly worldwide, recession.”

I distinctly remember that, while writing this, I feared criticism for gratuitous alarmism. And indeed, such criticism came.

I gave talks in 2005 at both the Office of the Comptroller of the Currency and at the Federal Deposit Insurance Corporation, in which I argued that we were in the middle of a dangerous housing bubble. I urged these mortgage regulators to impose suitability requirements on mortgage lenders, to assure that the loans were appropriate for the people taking them.

The reaction to this suggestion was roughly this: yes, some staff members had expressed such concerns, and yes, officials knew about the possibility that there was a bubble, but they weren’t taking any of us seriously.

I BASED my predictions largely on the recently developed field of behavioral economics, which posits that psychology matters for economic events. Behavioral economists are still regarded as a fringe group by many mainstream economists. Support from fellow behavioral economists was important in my daring to talk about speculative bubbles.

Speculative bubbles are caused by contagious excitement about investment prospects. I find that in casual conversation, many of my mainstream economist friends tell me that they are aware of such excitement, too. But very few will talk about it professionally.

Why do professional economists always seem to find that concerns with bubbles are overblown or unsubstantiated? I have wondered about this for years, and still do not quite have an answer. It must have something to do with the tool kit given to economists (as opposed to psychologists) and perhaps even with the self-selection of those attracted to the technical, mathematical field of economics. Economists aren’t generally trained in psychology, and so want to divert the subject of discussion to things they understand well. They pride themselves on being rational. The notion that people are making huge errors in judgment is not appealing.

In addition, it seems that concerns about professional stature may blind us to the possibility that we are witnessing a market bubble. We all want to associate ourselves with dignified people and dignified ideas. Speculative bubbles, and those who study them, have been deemed undignified.

In short, Mr. Janis’s insights seem right on the mark. People compete for stature, and the ideas often just tag along. Presidential campaigns are no different. Candidates cannot try interesting and controversial new ideas during a campaign whose main purpose is to establish that the candidate has the stature to be president. Unless Mr. Greenspan was exceptionally insightful about social psychology, he may not have perceived that experts around him could have been subject to the same traps.