I recently had the pleasure of reading Justin Fox’s new book The Myth of the Rational Market . It offers an engaging history of the research that has come to be called the “efficient market hypothesis”. It is similar in style to the classic by the late Peter Bernstein, Against the Gods. All the quotes in this column are taken from it. The book was mostly written before the financial crisis . However, it is natural to ask if the experiences over the last year should change our view of the EMH.

It helps to start with a quick review of rational finance. Modern finance began in the 1950s when many of the great economists of the second half of the 20th century began their careers. The previous generation of economists, such as John Maynard Keynes, were less formal in their writing and less tied to rationality as their underlying tool. This is no accident. As economics began to stress mathematical models, economists found that the simplest models to solve were those that assumed everyone in the economy was rational. This is similar to doing physics without bothering with the messy bits caused by friction. Modern finance followed this trend.

From the starting point of rational investors came the idea of the efficient market hypothesis, a theory first elucidated by my colleague and golfing buddy Gene Fama. The EMH has two components that I call “The Price is Right” and “No Free Lunch”. The price is right principle says asset prices will, to use Mr Fama’s words “fully reflect” available information, and thus “provide accurate signals for resource allocation”. The no free lunch principle is that market prices are impossible to predict and so it is hard for any investor to beat the market after taking risk into account.

For many years the EMH was “taken as a fact of life” by economists, as Michael Jensen, a Harvard professor, put it, but the evidence for the price is right component was always hard to assess. Some economists took the fact that prices were unpredictable to infer that prices were in fact “right”. However, as early as 1984 Robert Shiller, the economist, correctly and boldly called this “one of the most remarkable errors in the history of economic thought”. The reason this is an error is that prices can be unpredictable and still wrong; the difference between the random walk fluctuations of correct asset prices and the unpredictable wanderings of a drunk are not discernable.

Tests of this component of EMH are made difficult by what Mr Fama calls the “joint hypothesis problem”. Simply put, it is hard to reject the claim that prices are right unless you have a theory of how prices are supposed to behave. However, the joint hypothesis problem can be avoided in a few special cases. For example, stock market observers – as early as Benjamin Graham in the 1930s – noted the odd fact that the prices of closed-end mutual funds (whose funds are traded on stock exchanges rather than redeemed for cash) are often different from the value of the shares they own. This violates the basic building block of finance – the law of one price – and does not depend on any pricing model. During the technology bubble other violations of this law were observed. When 3Com, the technology company, spun off its Palm unit, only 5 per cent of the Palm shares were sold; the rest went to 3Com shareholders. Each shareholder got 1.5 shares of Palm. It does not take an economist to see that in a rational world the price of 3Com would have to be greater than 1.5 times the share of Palm, but for months this simple bit of arithmetic was violated. The stock market put a negative value on the shares of 3Com, less its interest in Palm. Really.

Compared to the price is right component, the no free lunch aspect of the EMH has fared better. Mr Jensen’s doctoral thesis published in 1968 set the right tone when he found that, as a group, mutual fund managers could not outperform the market. There have been dozens of studies since then, but the basic conclusion is the same. Although there are some anomalies, the market seems hard to beat. That does not prevent people from trying. For years people predicted fees paid to money managers would fall as investors switched to index funds or cheaper passive strategies, but instead assets were directed to hedge funds that charge very high fees.

Now, a year into the crisis, where has it left the advocates of the EMH? First, some good news. If anything, our respect for the no free lunch component should have risen. The reason is related to the joint hypothesis problem. Many investment strategies that seemed to be beating the market were not doing so once the true measure of risk was considered. Even Alan Greenspan, the former Federal Reserve chairman, has admitted that investors were fooled about the risks of mortgage-backed securities.

The bad news for EMH lovers is that the price is right component is in more trouble than ever. Fischer Black (of Black-Scholes fame) once defined a market as efficient if its prices were “within a factor of two of value” and he opined that by this (rather loose) definition “almost all markets are efficient almost all the time”. Sadly Black died in 1996 but had he lived to see the technology bubble and the bubbles in housing and mortgages he might have amended his standard to a factor of three. Of course, no one can prove that any of these markets were bubbles. But the price of real estate in places such as Phoenix and Las Vegas seemed like bubbles at the time. This does not mean it was possible to make money from this insight. Lunches are still not free. Shorting internet stocks or Las Vegas real estate two years before the peak was a good recipe for bankruptcy, and no one has yet found a way to predict the end of a bubble.

What lessons should we draw from this? On the free lunch component there are two. The first is that many investments have risks that are more correlated than they appear. The second is that high returns based on high leverage may be a mirage. One would think rational investors would have learnt this from the fall of Long Term Capital Management, when both problems were evident, but the lure of seemingly high returns is hard to resist. On the price is right, if we include the earlier bubble in Japanese real estate, we have now had three enormous price distortions in recent memory. They led to misallocations of resources measured in the trillions and, in the latest bubble, a global credit meltdown. If asset prices could be relied upon to always be “right”, then these bubbles would not occur. But they have, so what are we to do?

While imperfect, financial markets are still the best way to allocate capital. Even so, knowing that prices can be wrong suggests that governments could usefully adopt automatic stabilising activity, such as linking the down-payment for mortgages to a measure of real estate frothiness or ensuring that bank reserve requirements are set dynamically according to market conditions. After all, the market price is not always right.