Reading this report from Bloomberg, I was wondering about another possible explanation for the puzzle raised many times by Paul Krugman in his NY Times column: Krugman claims if speculation is the true culprit, you will have to see a large inventory buildup of oil. However, there is no evidence for the inventory accumulation.

The new possibility is that if oil suppliers realize or fear that the current fast run-up of oil price is unlikely to last, they will not increase production capacity. Therefore on one hand, commodity futures and other financial products (such as ETFs) keep bidding up oil price higher and higher, while on the other hand there is not much increase of supply (then no inventory buildup).

Just like farmers decide not to plant more corns because they expect corn price will fall next year. Oil producers are likely to do the same thing: They don’t want to get burned. Seems like rational expectations theory in application.

Oil at $200 a barrel: that was the warning from Goldman Sachs, published last week. The real price is already at an all-time high (see chart). At $200 it would be twice as high as it was in any previous spike. Even so, it would be a mistake to focus in shock only on the short-term jump in prices. The bigger issues are longer term.

Here are three facts about oil: it is a finite resource; it drives the global transport system; and if emerging economies consumed oil as Europeans do, world consumption would jump by 150 per cent. What is happening today is an early warning of this stark reality. It is tempting to blame the prices on speculators and big bad oil companies. The reality is different.

Demand for oil grows steadily, as the vehicle fleets of the world expand. Today, the US has 250m vehicles and China just 37m. It takes no imagination to see where the Chinese fleet is headed. Other emerging countries will follow China’s example.

Meanwhile, spare capacity in members of the Organisation of the Petroleum Exporting Countries is currently at exceptionally low levels, while non-Opec production has equally consistently disappointed expectations. (See charts.)

It looks increasingly hard to expand supply by the annual amount of about 1.4m barrels a day needed to meet demand. This means an extra Saudi Arabia every seven years. According to the International Energy Agency, almost two-thirds of additional capacity needed over the next eight years is required to replace declining output from existing fields. This makes the task even harder than it seems. As the latest World Economic Outlook from the International Monetary Fund adds, the fact that peak production is reached sooner, because of today’s efficient technologies, also means that subsequent declines are steeper.

This is not to argue that speculation has played no role in recent rises in prices. But it is hard to believe it has been a really big one. True, the dollar price has risen sharply, but that is partly the result of the decline in the dollar’s relative value (see chart). As I have argued before, if speculation were raising prices above the warranted level, one would expect to see inventories piling up rapidly, as supply exceeds the rate at which oil is burned. Yet there is no evidence of such a spike in inventories, as Goldman Sachs and the IMF point out.

Similarly, it is not even true that the investment needed to boost the constrained production capacity has been lagging. The WEO shows that nominal investment by national and international oil companies more than doubled between 2000 and 2006. But real investment hardly increased, because of a global scarcity of rigs and associated skilled labour services. Against this background, it seems far more likely that such speculation as there is has been stabilising, rather than destabilising: in other words, it is moving prices in the right direction, in order to reduce demand.

Will the high prices succeed in doing this? Certainly. Demand has to match supply for a simple reason: we cannot burn oil that does not exist.

The price spikes of the 1970s were followed by big absolute falls in demand and output (see chart). This was partly because of the recessions and partly because of rising efficiency. Both forces should work again this time, but to a much smaller extent. The slowdown in the US economy is indeed likely to be significant. Slowdowns will also occur in western Europe and Japan and even in the emerging world. But the latter will still grow rapidly. Overall, the world economy – and so world oil demand – is likely to continue to grow reasonably briskly. Similarly, the improved efficiency of use of petroleum, as people switch to more efficient vehicles, notably in north America (where the room for doing so is so large), will be offset by the rising tide of demand for motorised transport in the world’s fast-growing emerging countries.

On balance, it is quite unlikely that aggregate demand for oil will collapse, as it did after the two previous price spikes, just as it is unlikely that massive net new oil supplies will come on stream in the near future. This does not mean that prices will remain as high as they are today for the indefinite future: such stability is improbable. But it means we should expect a sustained period of relatively high prices even if “peak oil” theorists are proved wrong. If proved right, this would be true in spades.

So what should be the response to these simple realities? Here are some obvious “do nots” and “dos”.

First, do not blame conspiracies by speculators, oil companies or even Opec. These are the messengers. The message is one of fundamental shifts in demand and supply. If speculators push prices up in response, they are helping the adjustment. Even if Opec keeps output back, it is preserving a valuable resource for the future.

Second, do not blame the emerging countries for their growing demand. Citizens of rich countries must adjust to the higher prices of resources that the rise of the emerging countries entails. The only alternative is to attempt to destroy those hopes. That would be a blunder and a crime.

Third, understand that prices at these levels are now playing a big macroeconomic role. At $100 a barrel the annual value of world oil output would be close to $3,000bn. That is 5 per cent of world gross product. The only previous years in which it was higher than that were 1979 to 1982.

Fourth, adjust to high prices, which will play a big part in encouraging more efficient use of this finite resource and ameliorating climate change. The current shock offers a golden opportunity to set a floor on prices, by imposing taxes on oil, fossil fuels or carbon emissions.

Fifth, do try to reach global agreement on a pact on trade in oil based on the fundamental principle that producers will be allowed to sell their oil to the highest bidder. In other words, the global oil market needs to remain integrated. Nobody should use military muscle to secure a privileged position within it.

Finally, do become serious about investing in basic research into alternative technologies. Energy self-sufficiency is an implausible goal. Investing for a post-oil future is not.

We are no longer living in an age of abundant resources. It is possible that huge shifts in supply and demand will reverse this situation, as happened in the 1980s and 1990s. We can certainly hope for that happy outcome. But hope is not a policy.

The great event of our era is the spread of industrialisation to billions of people. The high prices of resources are the market’s response to this transforming event. The market is saying that we must use more wisely resources that have now become more valuable. The market is right.