Why is Manhattan housing cost so high? Answer lies in restrictive regulations

The average rent in Manhattan has reached a new record, according to this Market Watch article.
The average rent in Manhattan is now $3,429 per month, well above pre-recession levels.  A new one bedroom unit in a doorman building fetches $3982 per month.  Even a one bedroom in a walk-up costs $2580 per month.  The article attributes this ongoing surge in rent to supply constraint, citing a vacancy rate of 1.16% - well below the national average of about 5%.

Surely, prices are ultimately set by supply and demand.  According to Wikipedia, the residential population of Manhattan is 1.58 million.  Every work day additional 1.61 million people pour into Manhattan - there is no shortage of potential buyers.

But having lived in NYC area briefly, I've long suspected that there's more than just market forces inflating housing costs.  It always appeared unusual to me that large tracts of Manhattan - one of the most valuable pieces of real estate in the world - are covered by run down buildings from over a century ago.  Large areas on the west side of Manhattan are covered by warehouses and low rise buildings.  If left to their own devices, land owners will surely demolish these existing structures and build high rise condominiums.  Housing stock in Manhattan would grow rapidly, and prices would come down.  I've always thought that there must be some political reasons behind this paradox.

This Harvard university article confirms my suspicions.  It identifies land use regulations as a primary cause of inflated prices in Manhattan.  Back in 2004, they calculate that the market price of housing is more than twice the marginal cost of constructing housing.  Increase in the number of people wishing to live in Manhattan has not resulted in increased supply, but merely higher prices.  Prices today are much higher than it was in 2004, so the article is more true today than ever.

What political forces might drive these land use regulations?  Basically, everyone who owns residential real estate in Manhattan has a huge interest in keeping supply down.  They form neighborhood associations, historical preservation societies, and also lobby city councilmen.  They sue to block new developments in their areas, often citing reasons like "environment" and "overcrowding."  Zoning laws fundamentally constrict what areas may be used for residential development - which is great if you happen to already own a residential property in Manhattan.  What ticks me off is that large number of renters buy into the arguments of those who already own.  Think if it make you that much happy that your block contains century old townhouses and not modern condominiums.  It might be charming, but the policies that preserve those townhouses are costing you hundreds if not thousands of dollars per month.